867155 - mozmill --restart clears profile between restart tests

Status: RESOLVED INVALID

(Testing Graveyard :: Mozmill, defect)

Description

[Andrei Eftimie]

While investigating bug 860670 I've found that running the tests with mozmill --reset would not install the addon correctly.
It resets the profile between tests.
Code responsible is: https://github.com/mozilla/mozmill/blob/master/mozmill/mozmill/__init__.py#L414

That line was added part of refactoring for 2.0 here: https://github.com/mozilla/mozmill/pull/59/files

We've had that at some point in 1.5, but latest version of mozmill 1.5 does not do any profile reset between tests: https://github.com/mozilla/mozmill/blob/hotfix-1.5/mozmill/mozmill/__init__.py#L621

Comment 1

[Andrei Eftimie]

Attachment: Patch v1

Removing the profile reset between tests fixes the problem.
(eg. on bug 860670 the extension installed in test1 is still installed in the following tests)

I would link a testrun but this code is not executed while running one.

Comment 2

[Andrei Eftimie]

Found bug 641956 which implemented this.

The problem we are facing now is that we want backwards compatibility.
Mozmill 2.0 should be able to run our present test suite.

How should we move forward?
If we leave this as is, we would need to refactor *all* restart tests to do internal restarts.

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Andrei Eftimie from comment #2)
> How should we move forward?
> If we leave this as is, we would need to refactor *all* restart tests to do
> internal restarts.

Yes, I believe this is the plan.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Andrei Eftimie from comment #0)
> That line was added part of refactoring for 2.0 here:
> https://github.com/mozilla/mozmill/pull/59/files

This assessment is wrong. It didn't came in with this pull or bug. It was a simple move of code from one place to another. So please check when the responsible code really came in. 

> We've had that at some point in 1.5, but latest version of mozmill 1.5 does
> not do any profile reset between tests:
> https://github.com/mozilla/mozmill/blob/hotfix-1.5/mozmill/mozmill/__init__.
> py#L621

I don't think we had this in 1.5 at all. The --reset feature is new for Mozmill 2.0 and exists so we can reset the profile between each test. But within a restart test we shouldn't do this. So i don't think we can go with the attached patch which will kill the feature. Reset will still have to be called but before the next tests starts.

Comment 5

[Andrei Eftimie]

As discussed in our Ask an Expert meeting yesterday, this is actually intended behaviour.
Mozmill 2.0 does not have/need a specific "restart" mode. This will be fixed by refactoring our tests to manually restart Firefox in bug 867217

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I actually checked the mentioned option '--reset' but it is not part of Mozmill. So I'm not sure what you mean here. I would imagine it's '--restart' given that I can reproduce this problem easily.

 (In reply to Andrei Eftimie from comment #5)
> As discussed in our Ask an Expert meeting yesterday, this is actually
> intended behaviour.

No, it's not and that's not what I have mentioned in the meeting. What I said was that in-between a restart test we never should reset the profile. This should only happen between different tests. So this bug is perfectly valid.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

If a test is correctly implemented as a restart test this will not happen. Means Firefox has to be restarted at the end of each test. Once that's done --restart will no longer reset the profile. Tested with the problem on bug 860670 and it works fine for me. Marking as invalid.