Closed Bug 1515599 Opened 11 months ago Closed 11 months ago

Prevent android hardware tests from running when using try option syntax

Categories

(Firefox for Android :: Testing, defect)

defect
Not set

Tracking

()

RESOLVED FIXED
Firefox 66
Tracking Status
firefox66 --- fixed

People

(Reporter: bc, Assigned: bc)

References

Details

Attachments

(1 file)

Many people continue to use try: -b o -p all -u all -t none when running tests on try which results in what is most likely unwanted tests running on real android hardware at Bitbar. This is a particular issue with the jit tests but is potentially a problem for any tests.

Unnecessary tests on android hardware consume the capacity of our limited number of devices and reduce their lifetimes.

To make the selection of android hardware tests more conscious, I would like to disable any try testing of android hardware using try option syntax and instead just support mach try fuzzy.
<https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&author=bclary%40mozilla.com&fromchange=e57d51cca4e93e9efadd1b5a18c26f7c9795a899&tochange=637a44216e783dbf684382fbc9c881b175b8d69e>

shows 3 runs:

1. try: -b o -p android-api-16,android-aarch64 -u jittests -t none
   without the patch. jit tests are run.

2. try: -b o -p android-api-16,android-aarch64 -u jittests -t none
   with the patch. jit tess are not run.

3. Fuzzy query=android-hw jit
   jit tests are run.
Attachment #9032638 - Flags: review?(ahal)
Comment on attachment 9032638 [details] [diff] [review]
restrict-try-syntax-android-hw.patch

Review of attachment 9032638 [details] [diff] [review]:
-----------------------------------------------------------------

Lgtm, though I think this will disable android-hw from all try syntax in general, not just when using -p all. I assume that was your intent.
Attachment #9032638 - Flags: review?(ahal) → review+
Yes, if they aren't using try fuzzy they don't get to run android-hw tests.
Pushed by bclary@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/527fcd973d9f
Prevent android hardware tests from running when using try option syntax, r=ahal.
https://hg.mozilla.org/mozilla-central/rev/527fcd973d9f
Status: ASSIGNED → RESOLVED
Closed: 11 months ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 66
You need to log in before you can comment on or make changes to this bug.