1246538 - mochitest-o should trigger mochitest-a11y and mochitest-chrome-*

Status: RESOLVED WONTFIX

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Phil Ringnalda (:philor)]

If you request -u mochitest-o and build on linux32 debug, even if we did have mochitest-a11y and mochitest-chrome-1 and mochitest-chrome-2 and mochitest-chrome-3 on trychooser (a separate bug), you shouldn't have to choose them along with mochitest-o to to get the same tests to run on every platform you built on.

Comment 1

[Alin Selagea [:aselagea]]

If my understanding is correct, in bug 1211889 we had mochitest-oth splitted into mochitest-a11y(a) and mochitest-chrome(c1,c2,c3) for linux32 (debug only). 

At the moment, we don't have this implemented in TryChooser, but my guess is that "mochitest-o" will be replaced with the corresponding "mochitest-a11y" and "mochitest-chrome-*" entries once the change is made available for more platforms. I may be wrong here, but I think it would be way more useful to be able to choose if you want to run only mochitest-a11y(which is faster) or only mochitest-chrome (or chunks of it) instead of triggering all those tests when selecting "mochitest-o". Try-Parser will also need changes to reflect that.

I'm am a bit confused about the bug description, so my question would be: do we want to find a workaround so that selecting "mochitest-o" in TryChooser to trigger mochitest-a11y and mochitest-chrome-* tests (which would be available for linux32 debug only atm)? Or should we wait for more progress in bug 1211889? (we can help on that if you want us) :-)

Thanks.

Comment 2

[Phil Ringnalda (:philor)]

Yes, we want a workaround for the period while it's only one platform which is split apart (or one and a half, depending on how you count the taskcluster linux64 debug tests and whether or not they will pay any attention to what you do), because there's no reason to believe that the rest will follow in just a matter of days or weeks, it's far far more likely to be months at the shortest.

It took four months to get to the point of having only one platform split; finding enough free builders to split opt linux32 and opt and debug and asan linux64 will mean something closer to finding an entire branch to decommission, and we've already done that to every branch we could find that we could get away with getting rid of. Short of that, you're either looking at when e10s becomes the only thing supported and we drop non-e10s testing (I'd bet on... call it "year" since I'd take less than two years), or when taskcluster jobs replace all the linux tests and we shut off the buildbot ones (I'd bet on less than a year, but not significantly less), or when we shut off the b2g emulator-ics tests (no bet, totally random).

Comment 3

[Alin Selagea [:aselagea]]

Attachment: bug_1246538.patch

Did some local tests using "test_try_parser.py script": added "mochitest-a11y" and "mochitest-chrome" to the list of UNITTEST_SUITES entries, also added some test names to VALID_TESTER_NAMES list and then updated Tryparser as seen in the patch attached. 

Things seem to be working fine, meaning that submitting a command like:
   "tm = 'try: -b od -p linux -u mochitest-o'"
will output the following tests: 
   "['Rev3 Fedora 12 try opt test mochitest-chrome', 'Rev3 Fedora 12 try debug test mochitest-other', 'Rev3 Fedora 12 try opt test mochitest-other', 'Rev3 Fedora 12 try opt test mochitest-a11y']".

Comment 4

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbotcustom/rev/926a0351a0d7

Comment 5

[Xidorn Quan [:xidorn] UTC+10]

It doesn't seem to work. In my try push: https://treeherder.mozilla.org/#/jobs?repo=try&revision=2923bf1cc720 there is mochitest-o specified, but c1/c2/c3 are not triggered.

Probably this patch hasn't been deployed?

Comment 6

[Alin Selagea [:aselagea]]

Sorry for the late response. 

The patch has been deployed, but for some reason it omits c1/c2/c3 tests. I will need to investigate the issue in more depth, hopefully time will allow me to do so.

Comment 7

[Phil Ringnalda (:philor)]

Comment on attachment 8720277 [details] [diff] [review]
bug_1246538.patch

>+    elif u in ('mochitest-o'):
>+        return v in ['mochitest-other', 'mochitest-a11y', 'mochitest-chrome']


mochitest-chrome-1 isn't *in* [mochitest-chrome], it only startswith.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

It looks like this also omitted to update trychooser to indicate that `-u mochitest-o` doesn't mean "mochitest-other" anymore.

Also, per bug 1335072, the in-tree taskgraph generation will need to be updated to support this syntax as well.

Comment 10

[Phil Ringnalda (:philor)]

It certainly wasn't any intentional part of the patch here to make mochitest-o not trigger mochitest-other, so if in fact it doesn't that's more bustage.

But I wouldn't treat bug 1335072 as meaning that it doesn't work, since all smaug's mochitest-o pushes this morning were -p linux,linux64, and on Linux there is no mochitest-other, only mochitest-a11y and mochitest-chrome-1/2/3. Having -p linux,linux64 -u mochitest-o work by triggering mochitest-chrome-1, mochitest-chrome-2, mochitest-chrome-3 and mochitest-a11y so that people don't have to know that getting chrome to run involves mochitest-o if you are talking about Windows, and mochitest-chrome if you are talking about Linux or Mac, and mochitest-chrome,mochitest-o if your are talking about both, is exactly what this bug is about. Of course, now it's two bugs rather than one, since for Mac it's this one about try_parser.py and for Linux it's a different one about try_option_syntax.py.

The bug which failed to update trychooser would be bug 1211889 and whatever bug split mochitest-other apart on Mac, though I can't quite picture how you would go about having trychooser explain

[ ] mochitest-chrome (only on Linux and Mac, on Windows this is mochitest-o and for both it's both)
    [ ] mochitest-chrome-1 (only on Linux and Mac, on Windows this is mochitest-o and for both it's both)
    [ ] mochitest-chrome-2 (only on Linux and Mac, on Windows this is mochitest-o and for both it's both)
    [ ] mochitest-chrome-3 (only on Linux and Mac, on Windows this is mochitest-o and for both it's both)
[ ] mochitest-a11y (only on Linux and Mac, on Windows this is mochitest-o and for both it's both)

in a non-horrible way.

Comment 11

[Phil Ringnalda (:philor)]

That went well, I'd say.

Now there is no mochitest-other on any platform, but it's still in trychooser and this still doesn't work.

Please back out this broken patch and then wontfix this bug.