1294403 - Allow tests to control life cycle of Firefox

Status: RESOLVED DUPLICATE of bug 1301781

(Remote Protocol :: Marionette, defect)

Description

[André Reinald]

A test should be able to control the lifecycle of Firefox:
- take actions (including tweaking the profile),
- start Firefox, do things inside Firefox, quit Firefox,
- do things outside Firefox,
- start Firefox, do things inside Firefox, quit Firefox,
- do things outside Firefox.

Pseudo-code:

class ExampleSessionTestClass(...):

  def setUp(self):
    TC.setUp(self)
    self.pingServer=PingServer(...) #just an example
    self.pingServer.start()
    self.preferences.set("pingServerAddress", self.pingServer.address)

    self.profile.do_something_1(...)

  def tearDown(self):
    self.pingServer.stop()
    TC.tearDown(self)

  test_example_test(self):
    self.profile.do_something_2(...)

    marionette=Marionette(..., self.profile, ...)
    marionette.do_something_in_firefox_1(...)
    session=marionette.quit(...)

    self.profile.do_something_3(...)

    marionette.restart(session) #run with same profile
    marionette.do_something_in_firefox_2(...)
    session=marionette.quit(...)

    self.profile.do_something_4(...)

    marionette.cleanup(...)

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to André Reinald from bug 1293990 comment #18)
> (In reply to André Reinald from comment #12)
> > To be more specific about the changes from "marionette" to "session":
> > 
> > * In the "marionette" branch (harness.marionette) the marionette driver is
> > instantiated (and Firefox launched) from
> > marionette.runner.base.BaseMarionetteTestRunner.run_tests()
> > 
> > * In the "session" branch (harness.session) the marionette driver is
> > instantiated from session.session_test.CommonTestCase.setUp()
> > 
> > That brought a lot of changes on the way, as the marionette driver instance
> > was used through the "marionette" branch much before tests were started.
> 
> Bringing that together with Maja's tip that it's possible to instantiate a
> Marionette driver without launching Firefox (by passing the constructor
> bin=Null), I wonder if we can address my use case inside the "marionette"
> branch (and dispose of the "session" branch). That would mean adding an
> option to "mach marionette-test" telling the framework to skip launching
> Firefox (which my tests would do themselves).

Personally I would call this a hack, and we should better fix it correctly. Please keep in mind that for now my patch on bug 1279243 landed on mozilla-central. It means your test is able to control the life-cycle of Firefox - not from the very beginning but after it has been launched the first time. With that you can have the following workaround:

1. In setUp() quit Firefox
2. Do any profile related change including creating a new one if necessary
3. Start Firefox again
4. Quit Firefox to check for changes in the profile
5. Start Firefox again so telemetry data gets send
6. Quit Firefox to check the profile for telemetry data sent

I don't know which specific code you added for telemetry, but maybe having a new TestCase class or a mixin for Telemetry might make more sense, as duplicating everything under session.

You mentioned loss of time when doing it with the workaround. Well, for those 28 tests we talk about ~5s max for each of these. I believe your tests run definitely longer, so this little delay should not hurt. Also I see another benefit of it... Once we got the feature implemented to not directly start Firefox, it will be a simple change in setUp() to let your tests benefit from the new behavior.

Comment 2

[John Dorlus [:Silne30]]

(In reply to Henrik Skupin (:whimboo) from comment #1)
> (In reply to André Reinald from bug 1293990 comment #18)
> > (In reply to André Reinald from comment #12)
> > > To be more specific about the changes from "marionette" to "session":
> > > 
> > > * In the "marionette" branch (harness.marionette) the marionette driver is
> > > instantiated (and Firefox launched) from
> > > marionette.runner.base.BaseMarionetteTestRunner.run_tests()
> > > 
> > > * In the "session" branch (harness.session) the marionette driver is
> > > instantiated from session.session_test.CommonTestCase.setUp()
> > > 
> > > That brought a lot of changes on the way, as the marionette driver instance
> > > was used through the "marionette" branch much before tests were started.
> > 
> > Bringing that together with Maja's tip that it's possible to instantiate a
> > Marionette driver without launching Firefox (by passing the constructor
> > bin=Null), I wonder if we can address my use case inside the "marionette"
> > branch (and dispose of the "session" branch). That would mean adding an
> > option to "mach marionette-test" telling the framework to skip launching
> > Firefox (which my tests would do themselves).
> 
> Personally I would call this a hack, and we should better fix it correctly.
> Please keep in mind that for now my patch on bug 1279243 landed on
> mozilla-central. It means your test is able to control the life-cycle of
> Firefox - not from the very beginning but after it has been launched the
> first time. With that you can have the following workaround:
> 
> 1. In setUp() quit Firefox
> 2. Do any profile related change including creating a new one if necessary
> 3. Start Firefox again
> 4. Quit Firefox to check for changes in the profile
> 5. Start Firefox again so telemetry data gets send
> 6. Quit Firefox to check the profile for telemetry data sent
> 
> I don't know which specific code you added for telemetry, but maybe having a
> new TestCase class or a mixin for Telemetry might make more sense, as
> duplicating everything under session.
> 
> You mentioned loss of time when doing it with the workaround. Well, for
> those 28 tests we talk about ~5s max for each of these. I believe your tests
> run definitely longer, so this little delay should not hurt. Also I see
> another benefit of it... Once we got the feature implemented to not directly
> start Firefox, it will be a simple change in setUp() to let your tests
> benefit from the new behavior.

I think the workaround should work so I want to give this a shot. We an create a new test class like (SessionTestCase or TelemetryTestCase) to use the workaround and implement a basic telemetry test.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Is this bug still necessary? It's definitely not a P1, which is for webdriver only at the moment.

Comment 4

[John Dorlus [:Silne30]]

With the implementation of the TelemetryTestCase class, I would think that this has been resolved. Seems like it was a duplicate anyways.