664857 - Create a 'reftest' alias for 'reftest-1' and 'reftest-2' linux-android unittest suites

Status: RESOLVED FIXED

(Release Engineering :: General, enhancement, P5)

Description

[Mats Palmgren (inactive)]

See URL, where I used "try: -b o -u reftest" expecting to get reftest
results for Opt builds on all platforms.  The reftests for "Android opt"
is missing though.

Here is an earlier push that shows the R1 and R2 results that I expected:
http://tbpl.mozilla.org/?tree=Try&rev=026132968d01

Comment 2

[John Ford [:jhford] CET/CEST Berlin Time]

bug 665331 covers the what looks like the same issue, but for jsreftests.

Comment 3

[Lukas Blakk [:lsblakk] use ?needinfo]

Perhaps someone will get to this while I am away.

Comment 4

[Mats Palmgren (inactive)]

Android also has M(b-c), same problem there?

Comment 5

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Very hearty "metoo" on getting this fixed.  I just wasted several hours because "-u jsreftest" didn't work (http://hg.mozilla.org/try/rev/ffb1eb679dfa).

Comment 6

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

(This also means that I have to request all unit tests even if I just want reftests or jsreftests, which places more strain on the tegra2 pool.)

Comment 9

[Lukas Blakk [:lsblakk] use ?needinfo]

updated http://people.mozilla.org/~lsblakk/trychooser/ to reflect the unique suite names for android tests

Comment 10

[Mats Palmgren (inactive)]

So if I specify "-u reftest" does that give me all the reftest* variations
on the platforms where they apply?

Comment 11

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to Mats Palmgren [:mats] from comment #10)
> So if I specify "-u reftest" does that give me all the reftest* variations
> on the platforms where they apply?

No, if you check http://people.mozilla.org/~lsblakk/trychooser/ Android-Only Unittest Suites section, you need to specify 'reftest-1' and 'reftest-2'. Since the other platforms still use reftests there's no non-trivial way to break it out into 1 & 2 for just linux-android variations.

Comment 12

[Mats Palmgren (inactive)]

Sorry, but I really feel that "-u reftest" should cover all the variations of
reftests on different platforms.  That's what this bug is about originally.

Comment 13

[Lukas Blakk [:lsblakk] use ?needinfo]

Adjusting this accordingly then.  As far as I'm concerned it works right now and the tools have been updated to help people figure out how to get the specific tests they need. Creating an alias and special-casing for linux-android only tests turns this into a 'nice to have' enhancement.

Comment 15

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

https://wiki.mozilla.org/Build:TryChooser is also wrong, which cost me a few hours this morning.

Comment 16

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to David Baron [:dbaron] from comment #15)
> https://wiki.mozilla.org/Build:TryChooser is also wrong, which cost me a few
> hours this morning.

Can you be more specific about what's wrong on the wiki? 
If there's something incorrect there please make the necessary edits.

Comment 17

[Andrew Halberstadt [:ahal]]

This caused bustage and a backout because I just skimmed my green try results and didn't notice that Android didn't get run. I think this is more than a nice-to-have enhancement because it is wasting slave cycles in backouts and faulty try runs.

'mochitests' is an alias for mochitest-1, mochitest-2 etc. on Android, so naturally people are going to assume reftest works the same way. Note that since this bug has been filed, Android now uses 'plain-reftest-1'

Comment 18

[Daniel Holbert [:dholbert]]

Also since this bug has been filed, the B2G-specific jobs "reftest-1" through "reftest-10" have been created. TryChooser syntax with "-u reftest" does correctly trigger these jobs -- it only misses the Android reftests.

(This inconsistency makes this all the more difficult to keep track of, even for folks who are aware of this bug, and it often ends up making me feel like I need to click each of the 20 mobile-specific reftest boxes when I start Try runs. That's clearly not a good use of developer time.)

Comment 19

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Taras, is there a "Developer Productivity" bugzilla category we can move this bug to and maybe get some movement on it?

Comment 20

[Daniel Holbert [:dholbert]]

I believe "Release Engineering:Tools" is the right category for this bug (as a Try Server / Try Chooser bug), according to:
  https://bugzilla.mozilla.org/describecomponents.cgi?product=Release%20Engineering
though it's possible there are some "developer productivity boost" keywords that would be applicable.

CC'ing catlee, in case he knows of any RelEng folks who could fix this in the near term.

Comment 21

[Chris AtLee [:catlee]]

I think the code we need to patch is here:
http://hg.mozilla.org/build/buildbotcustom/file/1ecb6b76a48b/try_parser.py#l22

Looks like 'reftest' should also match 'plain-reftest-*'

Comment 22

[(dormant account)]

(In reply to Chris Jones [:cjones] mostly inactive; ni?/f?/r? if you need me from comment #19)
> Taras, is there a "Developer Productivity" bugzilla category we can move
> this bug to and maybe get some movement on it?

catlee is working on this asap.

Comment 23

[Chris AtLee [:catlee]]

checked in https://hg.mozilla.org/build/buildbotcustom/rev/d9b2f8bcc44b

should be deployed tomorrow morning

Comment 24

[John Hopkins (:jhopkins)]

in production.