Closed Bug 803408 Opened 12 years ago Closed 12 years ago

Ascertain which Android test suites have >30% failure rate and hide them

Categories

(Tree Management Graveyard :: TBPL, defect)

Product:
Tree Management Graveyard
Component:
TBPL
Platform:
ARM
Android
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: philor, Unassigned)

References

Details

(Whiteboard: [sheriff-want]) 

+++ This bug was initially created as a clone of Bug #777436 +++

Bug 797942, which was the fix for bug 784278, was backed out. Since then, we've had a 44% failure rate for Android reftests on mozilla-inbound (44% of the runs, it's actually something more like 10-20% of the tests within the run which fail, when we run reftests where everything is all white).

Bug 790102 doesn't really know why, maybe a code push, maybe a talos update, maybe an env var, but robocheck, robocheck2, robocheck3 and robopan now mostly fail. I haven't bothered to calculate the percentage, because it's obviously over 30%, but of the last four runs on inbound, robopan was green twice, and the other three were green zero times, so for that little sample it's 87.5% orange.

All Android reftests on Armv6 and Armv7, and those four robo* talos suites are now hidden on mozilla-inbound, mozilla-central, try, fx-team, services-central and profiling.
For future reference: Armv6 R2 and R4 are the ones that were already hidden for permaorange.
Every Android test suite on all of those trees is now unhidden. Good luck to whoever has to deal with them.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → WONTFIX
Without unhiding anything until I get through a meeting tomorrow, I'll note that, at least on inbound, Android [Armv7 and Amrv6] reftests 2 and 3 look green now. However Reftest-1 looks to have picked up a *new* perma-orange.
(In reply to Phil Ringnalda (:philor) from comment #2)
> Every Android test suite on all of those trees is now unhidden. Good luck to
> whoever has to deal with them.

Advised over IRC to re-hide what was previously hidden, so we can actually evaluate if it needs to stay hidden or can be unhidden. An unhidden perma-orange doesn't really help anyone, sheriffs, devs, or us.

/me gets back to work.
Product: Webtools → Tree Management
Product: Tree Management → Tree Management Graveyard
You need to log in before you can comment on or make changes to this bug.