If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

Unhide B2G emulator tests once they are more reliable and/or output most failures in a format that appears in the annotated summary

RESOLVED FIXED

Status

Tree Management
Visibility Requests
--
major
RESOLVED FIXED
5 years ago
3 years ago

People

(Reporter: emorley, Assigned: emorley)

Tracking

(Depends on: 1 bug)

Dependency tree / graph

Details

(Assignee)

Description

5 years ago
At the moment the B2G emulator tests frequently fail & in most of the common failure modes, do not correctly output the message as ERROR, so TBPL's parser cannot display it in the summary, so virtually every time the full log has to be opened.

This leads to:
1) Extra sheriff load (bearing in mind the majority of the people who sheriff are volunteers)
2) People ignoring B2G tests on Try and elsewhere.

#2 is the one that concerns me the most; after what happened with Android, we cannot afford B2G to become the next "oh it's probably just an infra failure let's ignore the failure on my Try run and/or my mozilla-central landing".

As such, I have hidden emulator test runs on {mozilla-central, mozilla-inbound, try, fx-team} until such a point where they are more reliable and/or don't just display in TBPL as "hh:mm:ss ERROR - Return code: 1" regardless of the actual failure.

In the meantime, results can be seen by using "&noignore=1". eg:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&noignore=1&jobname=emulator

Once a sizeable proportion of the dependant bugs are fixed, I will unhide.
(Assignee)

Comment 1

5 years ago
Just to clarify:

If bug 809437, bug 809529 & ideally bug 809529 comment 6 were fixed, we could likely unhide before the others are done, since they are the worst offenders.
I have noticed at least 3 (possibly more) occasions where a checkin has completely broken Mn or other tests and gotten backed out immediately as a result. The set of tests we have running at the moment is less about having test coverage and more about making sure checkins don't break the automation while we are in the middle of trying to improve it (which has caused us many many delays dating back to June).

I know it's been really annoying for the sheriffs, but would it be possible to at least have Mn tests (or mochitests) unhidden? I'll be landing another attempt at a fix for bug 809437 today
(Assignee)

Comment 3

5 years ago
We're still running the tests, so breakage-causing landings can still be identified, by the sheriffs (or people working on the emulator) using &noignore=1. The only difference is that:

a) We don't have to star every failure if they are hidden for everyone else, just keep an eye out for permaorange.
b) Emulator tests don't become the next "platform that cried wolf", alongside Android (which to this day people still ignore, even though our infra-failure & TBPL starring story is so much better than it was).

I'm keeping a tab open with ...&noignore=1&jobname=emulator if you and other interested parties can too, then I don't think we lose much by hiding (given that this is going to just be for a couple of days).
Ok, that works for me.  Thanks :)
(Assignee)

Comment 5

5 years ago
I've unhidden on inbound, in the hope that the various fixes have helped. Is definitely looking much greener than it was :-)

Once the fixes have had a chance to propagate, I'll unhide elsewhere (presuming no surprises).
(Assignee)

Comment 6

5 years ago
Now unhidden on all trees & I've starred things until a few days back, so people's Try runs etc don't look broken.

Thank you to everyone who worked on improving both the reliability and also the harness output for TBPL :-)
Assignee: nobody → bmo
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Product: Webtools → Tree Management
(Assignee)

Updated

3 years ago
Component: TBPL → Visibility Requests
You need to log in before you can comment on or make changes to this bug.