1515599 - Prevent android hardware tests from running when using try option syntax

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: Testing, defect)

Description

[Bob Clary [:bc] (inactive)]

Many people continue to use try: -b o -p all -u all -t none when running tests on try which results in what is most likely unwanted tests running on real android hardware at Bitbar. This is a particular issue with the jit tests but is potentially a problem for any tests.

Unnecessary tests on android hardware consume the capacity of our limited number of devices and reduce their lifetimes.

To make the selection of android hardware tests more conscious, I would like to disable any try testing of android hardware using try option syntax and instead just support mach try fuzzy.

Comment 1

[Bob Clary [:bc] (inactive)]

Attachment: restrict-try-syntax-android-hw.patch

<https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&author=bclary%40mozilla.com&fromchange=e57d51cca4e93e9efadd1b5a18c26f7c9795a899&tochange=637a44216e783dbf684382fbc9c881b175b8d69e>

shows 3 runs:

1. try: -b o -p android-api-16,android-aarch64 -u jittests -t none
   without the patch. jit tests are run.

2. try: -b o -p android-api-16,android-aarch64 -u jittests -t none
   with the patch. jit tess are not run.

3. Fuzzy query=android-hw jit
   jit tests are run.

Comment 2

[Andrew Halberstadt [:ahal]]

Comment on attachment 9032638 [details] [diff] [review]
restrict-try-syntax-android-hw.patch

Review of attachment 9032638 [details] [diff] [review]:
-----------------------------------------------------------------

Lgtm, though I think this will disable android-hw from all try syntax in general, not just when using -p all. I assume that was your intent.

Comment 3

[Bob Clary [:bc] (inactive)]

Yes, if they aren't using try fuzzy they don't get to run android-hw tests.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/527fcd973d9f
Prevent android hardware tests from running when using try option syntax, r=ahal.

Comment 5

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/527fcd973d9f