672165 - Add feature to use a subfolder with a pre-configured profile for a Mozmill testrun

Status: RESOLVED INVALID

(Mozilla QA Graveyard :: Mozmill Automation, enhancement)

Description

[Marc-Aurèle DARCHE]

Add the possibility to select a subfolder in the add-ons test directory which contains a pre-configured profile with sample credentials our automation script can use to somehow inject it into the newly created profile. That way we wouldn't be dependent on the Mozmill integration.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

My patch on bug 704068 implements the basic support of handling different profiles. Given that most of our tests are wiping away session data we need a more robust version of injecting existent profile data in-between tests.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I would like to get more discussion on this topic. I think it's kinda important for our future, especially when we have migration tests.

Right now we reset the application state after each test and most of the time even before the test. Those steps mean that anything which was part of the example profile gets wiped out and will not be available for the next test anymore. So there are different paths we could follow:

1. Before a test gets started we have to ensure that all pre-configured data gets added and will be available for Firefox immediately. I see a problem here because at least some preferences are requiring a restart.

2. We have to find a better strategy in resetting the application state in setup/teardown. This can become kinda difficult, and we should probably consider to have a specific testrun for migration tests.

3. We limit the pre-configured data, which will be available across all testruns, to specific types which we will not delete in setup/teardown. At the same step we should think about if tests should only reset the data, which has been set by themselves.

In case of the original request we need a way to have a preconfigured signons.sqlite database, we can inject for the test-run. As long as it's a test user and the password can be distributed, it has to land sideways the tests. That's the only good solution for Mozmill Crowd yet, and I don't think that we want to offer rich test profiles for any type of test-run/test-case.

What do others think?

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

If I'm understanding correctly, then I think I'd vote for option 2. With the API refactoring we're coming close to abstracting most of the restoring state out of the test, and we should be able to include some logic in here to preset values other than the fresh install defaults.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

So I think for tests with this kind of requirement we would need a separate testrun, and each of those tests would have to be a restart test. That way an existing profile could be re-used via the --profile option. We would also have to work on a profile clone, so the original data is not getting lost.

Nothing in here looks like where we need action on the automation repository now. If we need something specific lets file an issue at https://github.com/mozilla/mozmill-automation