Add the --restart command line option for the Gaia UI tests

RESOLVED FIXED

Status

Release Engineering
General Automation
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: davehunt, Assigned: jgriffin)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

4 years ago
We need to add the --restart option to make sure we use a fresh clone of the profile for every test.
(Assignee)

Updated

4 years ago
Assignee: nobody → jgriffin
(Assignee)

Comment 1

4 years ago
Created attachment 806340 [details] [diff] [review]
Add --restart to gaia-ui-tests,
Attachment #806340 - Flags: review?(dave.hunt)
(Reporter)

Comment 2

4 years ago
Comment on attachment 806340 [details] [diff] [review]
Add --restart to gaia-ui-tests,

Review of attachment 806340 [details] [diff] [review]:
-----------------------------------------------------------------

::: scripts/marionette.py
@@ +362,2 @@
>              cmd.extend(self._build_arg('--binary', os.path.join(dirs['abs_work_dir'],
>                                                                  'b2g', 'b2g')))

Unrelated to this patch, but should this be 'b2g-bin' in order to use the specified profile?
Attachment #806340 - Flags: review?(dave.hunt) → review+
(Assignee)

Comment 3

4 years ago
(In reply to Dave Hunt (:davehunt) from comment #2)
> Comment on attachment 806340 [details] [diff] [review]
> Add --restart to gaia-ui-tests,
> 
> Review of attachment 806340 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: scripts/marionette.py
> @@ +362,2 @@
> >              cmd.extend(self._build_arg('--binary', os.path.join(dirs['abs_work_dir'],
> >                                                                  'b2g', 'b2g')))
> 
> Unrelated to this patch, but should this be 'b2g-bin' in order to use the
> specified profile?

Ah, yes, you're quite right, thanks!
(Assignee)

Comment 4

4 years ago
So one disadvantage of this is that it will mean the tests are running a little differently between TBPL and Travis.  The speed hit that this will cause on TBPL isn't important, but the Gaia team wants Travis runs to be as fast as possible.

Do we understand what kind of state isn't being reset between tests that we're using --restart to work around?  Are those things Gaia bugs, or faults in test case tearDown, or some combination of the two?

I know that, on a device at least, if a test causes an app to crash, that it will likely cause a chain reaction of failures without --restart.  But that doesn't seem to be the case with the clock tests in bug 910430.
(Reporter)

Comment 5

4 years ago
(In reply to Jonathan Griffin (:jgriffin) from comment #4)
> So one disadvantage of this is that it will mean the tests are running a
> little differently between TBPL and Travis.  The speed hit that this will
> cause on TBPL isn't important, but the Gaia team wants Travis runs to be as
> fast as possible.

Travis is also running with the --restart argument: https://github.com/mozilla-b2g/gaia/blob/master/tests/travis_ci/gaia_ui_tests/script#L14

> Do we understand what kind of state isn't being reset between tests that
> we're using --restart to work around?  Are those things Gaia bugs, or faults
> in test case tearDown, or some combination of the two?

As far as I know there are no outstanding Gaia/Marionette bugs regarding the need to restart. Originally a lot of the tests included teardown steps, however these will not be executed if setup fails. Also, probably because of the success of the --restart argument, a lot of the test teardown has been removed, making it much more necessary.

> I know that, on a device at least, if a test causes an app to crash, that it
> will likely cause a chain reaction of failures without --restart.  But that
> doesn't seem to be the case with the clock tests in bug 910430.

The app is not crashing in bug 910430, and I don't think I've seen a crash when running against the desktop build, so I'm not sure if the same applies here.
(Assignee)

Comment 6

4 years ago
Ok, if Travis is running with --restart too, then I'm less worried.
(Assignee)

Comment 8

4 years ago
in production
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.