Closed Bug 811685 Opened 12 years ago Closed 11 years ago

Reset to a clean profile between tests

Categories

(Remote Protocol :: Marionette, defect)

Product:
Remote Protocol
Component:
Marionette
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: davehunt, Unassigned)

References

Details

We need a way to reset to a clean profile between tests. This is so that anything that isn't (or can't) be reset in a test's tearDown method will not affect the following test. One such example of this is a test that logs into Marketplace, which will cause all following testruns to remember that the user is logged in.
Why can't this test logout the user in the tearDown method if a login was successful during the test?
(In reply to Henrik Skupin (:whimboo) from comment #1)
> Why can't this test logout the user in the tearDown method if a login was
> successful during the test?

What we're suggesting here is basically the same as WebDriver tearing down the browser and creating a fresh unpolluted profile.

If the marionette session is lost somewhere along the way then the tearDown can't be completed.

IMO it's too fragile to use the UI to perform setup and teardown.
(In reply to Zac C from comment #2)
> What we're suggesting here is basically the same as WebDriver tearing down
> the browser and creating a fresh unpolluted profile.

This is not something we want to do in Mochitests, but I don't know the exact details of Marionette. It takes a long time to restart the browser and it will certainly add a certain amount of time to the testrun duration. It will probably work with about 100 or so tests but having ten thousands of tests it would kill us.
Every effort should be made to teardown appropriately. This can include logging out, clearing localStorage, removing cookies, etc. If any of these teardown items fail, however then we may be in an unknown state. Selenium starts Firefox with a fresh profile, and Web QA start a browser for each test. There is an additional startup cost but imo the benefit far outweighs that cost, and we make up for this by running tests in parallel using Selenium Grid.
To be clear, I think what we're talking about here is resetting the profile before each test run, not before each test.

This can be done via Jenkins; it doesn't (and shouldn't) involve Marionette.  I can describe the jobs that need to be added.
Can you state the reasoning why it shouldn't involve Marionette?

In WebDriver starting a session always involves a new profile. Although it's not in W3C spec as far as I can see.

Marionette could just replicate the state of a new profile rather than actually create a new profile.
(In reply to Jonathan Griffin (:jgriffin) from comment #5)
> To be clear, I think what we're talking about here is resetting the profile
> before each test run, not before each test.

As discussed on IRC, I was originally meaning this to be between tests. This would ensure all tests start from the same fresh state. I understand though that this would incur considerable overhead. I will look into strengthening test teardown and cleanup so we can use this as a best practice.

> This can be done via Jenkins; it doesn't (and shouldn't) involve Marionette.
> I can describe the jobs that need to be added.

Please do describe the necessary steps. To clarify for my own understanding, does each build of B2G or flash of Gaia not also reset the profile?
(In reply to Zac C from comment #6)
> Can you state the reasoning why it shouldn't involve Marionette?
> 
> In WebDriver starting a session always involves a new profile. Although it's
> not in W3C spec as far as I can see.
> 
> Marionette could just replicate the state of a new profile rather than
> actually create a new profile.

Selenium contains browser launching code, which in the case of Firefox manipulates the profile in use. When running the Gaia UI tests we simply connect to an already running instance of B2G, so as I understand it there would be no way for Marionette to influence the profile at this point.
> 
> > This can be done via Jenkins; it doesn't (and shouldn't) involve Marionette.
> > I can describe the jobs that need to be added.
> 
> Please do describe the necessary steps. To clarify for my own understanding,
> does each build of B2G or flash of Gaia not also reset the profile?

Flashing the build does reset the profile, yes.  Installing Gaia modifies part of the profile, but not everything (it doesn't change /data/b2g/mozilla/*.default).
David Burns and I chatted about this in real life.

We both agreed that the person writing the test should not have to consider resetting the profile between tests. 

That means (and we were in agreement here, too) is that some piece of software, whether Marionette or otherwise should do the profile tidy up.

The other thing is that this piece of software can do it authoritatively whereas a test cleanup is not particularly reliable either human error or dependency upon the marionette session coming in to play.
I should also mention that I agree that this shouldn't be in Marionette since Marionette is worried about browser automation and not profile management like MozProfile is.

Perhaps we can add all the things to clean up, e.g. deleting cookies, to a state that we are happy with (within reason). This can be possibly be in the GaiaTest teardown https://github.com/mozilla/gaia-ui-tests/blob/master/gaiatest/gaia_test.py#L187
We have a similar situation with the desktop automation using Mozmill. We can reset the profile and relaunch Firefox between tests, which would ensure test isolation, however the overhead this would add to the testrun would be substantial.

In order to resolve this, all Mozmill tests are responsible for cleaning up any changes they make. For example, a test for the addons manager will close the addons manager at the end of the test, or a test that changes a preference will clear the user set preference. Wherever possible, cleanup uses backend calls rather than the user interface. In addition, most of the Mozmill tests include setup to ensure Firefox is in a good state. This consists mostly of closing any open tabs.

I think we could take a similar approach with the Gaia tests. Anything that is specifically called out in a test should be the test's responsibility to clean up (if you login you should log out, if you install an app you should remove it, etc). We can also add more cleanup in the common setUp methods, such as killing any running apps, etc.

In bug 812670 I introduce the addCleanup feature from unittest (requires Python 2.7), and just adding this has seriously improved test stability for me locally. I would appreciate any feedback there.

All this said, if there's a way for us to reset the profile between tests that does not incur an unacceptable overhead then I am enthusiastic to see this *in addition to responsible test cleanup*
B2G was not designed to be used with multiple profiles; Gaia depends on a lot of things in the one-and-only B2G profile.  Thus, resetting the profile between tests would be a very costly operation (much more so than doing the same thing in Firefox; it would increase the testrun time by a factor of 10 or more, I'd guess); and it wouldn't be very resilient to future B2G/Gaia changes.

I agree that tests should cleanup whatever changes they've made (this is no different than mochitest or any of the other test frameworks we use in Mozilla); plus we can build in some basic cleanup functions in GaiaTest's setUp and/or tearDown, like kill_all().

I'll comment separately on bug 812670.
We have implemented a cleanup method that's called during test setup. Tests should endeavour to remove an artifacts they create, but we should also be able to rely on the cleanup method to perform a general reset.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Product: Testing → Remote Protocol
You need to log in before you can comment on or make changes to this bug.