Closed Bug 664857 Opened 9 years ago Closed 6 years ago

Create a 'reftest' alias for 'reftest-1' and 'reftest-2' linux-android unittest suites

Categories

(Release Engineering :: General, enhancement, P5)

All
Android
enhancement

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: mats, Assigned: catlee)

References

()

Details

(Whiteboard: [trychooser][try])

See URL, where I used "try: -b o -u reftest" expecting to get reftest
results for Opt builds on all platforms.  The reftests for "Android opt"
is missing though.

Here is an earlier push that shows the R1 and R2 results that I expected:
http://tbpl.mozilla.org/?tree=Try&rev=026132968d01
Assignee: nobody → lsblakk
Duplicate of this bug: 665331
bug 665331 covers the what looks like the same issue, but for jsreftests.
Summary: "-u reftest" should run Android R1 and R2 tests → "-u reftest" should run Android R1 and R2 tests and "-u jsreftest" should run Android JS Reftests
Perhaps someone will get to this while I am away.
Assignee: lsblakk → nobody
Android also has M(b-c), same problem there?
Priority: -- → P3
Whiteboard: [trychooser][try]
Very hearty "metoo" on getting this fixed.  I just wasted several hours because "-u jsreftest" didn't work (http://hg.mozilla.org/try/rev/ffb1eb679dfa).
(This also means that I have to request all unit tests even if I just want reftests or jsreftests, which places more strain on the tegra2 pool.)
Duplicate of this bug: 693591
Duplicate of this bug: 680121
Assignee: nobody → lsblakk
Status: NEW → ASSIGNED
Summary: "-u reftest" should run Android R1 and R2 tests and "-u jsreftest" should run Android JS Reftests → Try syntax not working for Android unittest requests
updated http://people.mozilla.org/~lsblakk/trychooser/ to reflect the unique suite names for android tests
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
So if I specify "-u reftest" does that give me all the reftest* variations
on the platforms where they apply?
(In reply to Mats Palmgren [:mats] from comment #10)
> So if I specify "-u reftest" does that give me all the reftest* variations
> on the platforms where they apply?

No, if you check http://people.mozilla.org/~lsblakk/trychooser/ Android-Only Unittest Suites section, you need to specify 'reftest-1' and 'reftest-2'. Since the other platforms still use reftests there's no non-trivial way to break it out into 1 & 2 for just linux-android variations.
Sorry, but I really feel that "-u reftest" should cover all the variations of
reftests on different platforms.  That's what this bug is about originally.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Adjusting this accordingly then.  As far as I'm concerned it works right now and the tools have been updated to help people figure out how to get the specific tests they need. Creating an alias and special-casing for linux-android only tests turns this into a 'nice to have' enhancement.
Assignee: lsblakk → nobody
Severity: normal → enhancement
Priority: P3 → P5
Summary: Try syntax not working for Android unittest requests → Create a 'reftest' alias for 'reftest-1' and 'reftest-2' linux-android unittest suites
Duplicate of this bug: 695030
https://wiki.mozilla.org/Build:TryChooser is also wrong, which cost me a few hours this morning.
(In reply to David Baron [:dbaron] from comment #15)
> https://wiki.mozilla.org/Build:TryChooser is also wrong, which cost me a few
> hours this morning.

Can you be more specific about what's wrong on the wiki? 
If there's something incorrect there please make the necessary edits.
Status: REOPENED → RESOLVED
Closed: 9 years ago8 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
This caused bustage and a backout because I just skimmed my green try results and didn't notice that Android didn't get run. I think this is more than a nice-to-have enhancement because it is wasting slave cycles in backouts and faulty try runs.

'mochitests' is an alias for mochitest-1, mochitest-2 etc. on Android, so naturally people are going to assume reftest works the same way. Note that since this bug has been filed, Android now uses 'plain-reftest-1'
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Component: Other → Tools
QA Contact: hwine
Also since this bug has been filed, the B2G-specific jobs "reftest-1" through "reftest-10" have been created. TryChooser syntax with "-u reftest" does correctly trigger these jobs -- it only misses the Android reftests.

(This inconsistency makes this all the more difficult to keep track of, even for folks who are aware of this bug, and it often ends up making me feel like I need to click each of the 20 mobile-specific reftest boxes when I start Try runs. That's clearly not a good use of developer time.)
Taras, is there a "Developer Productivity" bugzilla category we can move this bug to and maybe get some movement on it?
Flags: needinfo?(taras.mozilla)
I believe "Release Engineering:Tools" is the right category for this bug (as a Try Server / Try Chooser bug), according to:
  https://bugzilla.mozilla.org/describecomponents.cgi?product=Release%20Engineering
though it's possible there are some "developer productivity boost" keywords that would be applicable.

CC'ing catlee, in case he knows of any RelEng folks who could fix this in the near term.
I think the code we need to patch is here:
http://hg.mozilla.org/build/buildbotcustom/file/1ecb6b76a48b/try_parser.py#l22

Looks like 'reftest' should also match 'plain-reftest-*'
(In reply to Chris Jones [:cjones] mostly inactive; ni?/f?/r? if you need me from comment #19)
> Taras, is there a "Developer Productivity" bugzilla category we can move
> this bug to and maybe get some movement on it?

catlee is working on this asap.
Flags: needinfo?(taras.mozilla)
checked in https://hg.mozilla.org/build/buildbotcustom/rev/d9b2f8bcc44b

should be deployed tomorrow morning
Assignee: nobody → catlee
in production.
Status: REOPENED → RESOLVED
Closed: 8 years ago6 years ago
Resolution: --- → FIXED
Depends on: 1009598
Component: Tools → General
You need to log in before you can comment on or make changes to this bug.