872364 - gaia-ui-tests is green when it runs zero tests and evaluates that as T-FAIL

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Phil Ringnalda (:philor)]

We pointlessly run gaia-ui-tests on Panda builds on b2g18.* and Birch (hidden) and Cedar (unhidden, though by its new rules it should be hidden) despite nobody having any interest in it, but that's not what I want to talk about.

Sometime in late April, the gaia-ui-tests suite was modified to prevent people from wiping data on their phone by running it, by requiring that a locally-modified file saying that the person running the suite has read a devmo page about it be passed on the commandline. We don't pass that file on the commandline, so we don't run the suite, but that's not what I want to talk about.

Since the locally modified file isn't passed on the commandline, we run zero tests, and as

https://tbpl.mozilla.org/php/getParsedLog.php?id=22948971&tree=Cedar

10:16:11     INFO -  ***** ACKNOWLEDGE RISKS ********************************************************
10:16:11     INFO -  These tests are destructive and will remove data from the target Firefox OS
10:16:11     INFO -  instance as well as using services that may incur costs! Before you can run
10:16:11     INFO -  these tests you must follow the steps to indicate you have acknowledged the
10:16:11     INFO -  risks detailed at the following address:
10:16:11     INFO -  https://developer.mozilla.org/en-US/docs/Gaia_Test_Runner#Risks
10:16:11     INFO -  ********************************************************************************
10:16:11     INFO - Return code: 0
10:16:11     INFO - TinderboxPrint: gaia-ui-tests: <em class="testfail">T-FAIL</em>

says, that's a T-FAIL. But it goes on to say

10:16:11     INFO - # TBPL SUCCESS #
10:16:11     INFO - #####
10:16:11     INFO - ##### Running close-request step.
10:16:11     INFO - #####
10:16:11     INFO - Request 'http://mobile-imaging-001.p1.releng.scl1.mozilla.com/api/request/207488/' deleted on cleanup
10:16:11     INFO - Copying logs to upload dir...
10:16:11     INFO - mkdir: /builds/panda-0158/test/build/upload/logs
program finished with exit code 0

which is patently false - T-FAIL and zero tests does not equal TBPL SUCCESS or an exit code 0.

Comment 1

[Jonathan Griffin (:jgriffin)]

So there are several things wrong here:

1 - we shouldn't be exiting with return code 0 in this case (gaia-ui-tests bug)
2 - even if we do, we shouldn't T-FAIL and TBPL SUCCESS at the same time (mozharness script bug)
3 - we really shouldn't force the user to acknowledge risks on the panda or b2g desktop builds (gaia-ui-tests bug)

I'll own 1 and 3, I'm inclined not to fix 2 until and unless we start caring about these tests on pandas again.  But, that code was likely copied and pasted from somewhere we do care about, so I'll check the master Marionette script and see if it suffers from the same problem.

Comment 2

[Jonathan Griffin (:jgriffin)]

pull request for 1 and part of 3: https://github.com/mozilla/gaia-ui-tests/pull/839

Comment 3

[Jonathan Griffin (:jgriffin)]

Attachment: Don't TBPL_SUCCESS when no tests have been run,

Comment 4

[Jonathan Griffin (:jgriffin)]

Comment on attachment 751858 [details] [diff] [review]
Don't TBPL_SUCCESS when no tests have been run,

https://hg.mozilla.org/build/mozharness/rev/2b54c036787f

Comment 5

[Aki Sasaki (not active)]

Merged to production.

Comment 6

[Jonathan Griffin (:jgriffin)]

Attachment: Make gaia-ui-tests skip the warning in buildbot,

This is the last piece of this bug.

Comment 7

[Jonathan Griffin (:jgriffin)]

Comment on attachment 762291 [details] [diff] [review]
Make gaia-ui-tests skip the warning in buildbot,

https://hg.mozilla.org/build/mozharness/rev/448071030542

Comment 8

[Aki Sasaki (not active)]

In production.