Closed Bug 853024 Opened 7 years ago Closed 7 years ago

Un-hide B2G emulator reftests, crashtests, and mochitests when failure rate is no longer unacceptably high

Categories

(Tree Management Graveyard :: Visibility Requests, defect)

ARM
Gonk (Firefox OS)
defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: RyanVM, Unassigned)

References

(Depends on 1 open bug)

Details

The failure rate of the B2G emulator reftests/crashtests/mochitests is currently 25+%, which is unacceptably high. Until the situation is resolved, I have hidden these tests on mozilla-central and all branches that feed directly into it.
(For reference) 

To view _just_ the B2G jobs, use the TBPL job name filter, eg:

mozilla-central:
https://tbpl.mozilla.org/?showall=1&jobname=b2g

mozilla-inbound:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&showall=1&jobname=b2g
Depends on: 851666
Depends on: 853949
Depends on: 853747
Depends on: 854514
Depends on: 854996
Depends on: 839787
Depends on: 855352
No longer depends on: 839787
Had this via email:
{
quick question : i've been trying to land something and it's been held up due to oranges on try with the b2g mochitests,
which seem unrelated to my changes 

i just looked at quite a few recent patches to m-c and m-i and it seems like we aren't running the b2g
mochitests on the emulator on them ? 
}

-> Hidden on Try as well to avoid confusion.
Depends on: 856127
Depends on: 856186
With a few recent fixes, mochitests are nearly always green on inbound.  On the last 20 runs, there was only 1 intermittent failure (a socket.timeout), excepting some failures that were solved by a backout.  Therefore, I plan on unhiding mochitests only today unless I hear some objections; reftests will remain hidden as their failure rate continues to be very high.
Sounds great, thanks Johathan!
I've unhidden B2G mochitests on m-i, m-c, and fx-team (they were already visible on try and services-central), and starred the last couple of days worth of failures (only a handful of these existed).
(Moving to TBPL's component, since that's where the unhiding takes place, and is the component sheriffs watch. Fixing the failure rate to get us to the point where we can unhide is covered by the dependant bugs, which are in B2G/testing components)
Component: General → Tinderboxpushlog
Product: Boot2Gecko → Webtools
Version: unspecified → Trunk
Blocks: 861024
Depends on: 861186
Depends on: 852821
No longer blocks: 861024
Duplicate of this bug: 861024
I have un-hidden all B2G reftests and crashtests on m-c/inbound/birch except for:
* crashtest-1 due to bug 852821
* reftest-1/6 for various failures that appear to be unfiled currently
R6 is filed (twice, since I didn't find bug 855352 before I filed bug 856186).
Depends on: 861928
Depends on: 862787
(In reply to Ryan VanderMeulen [:RyanVM] from comment #8)
> I have un-hidden all B2G reftests and crashtests on m-c/inbound/birch except
> for:
> * crashtest-1 due to bug 852821
> * reftest-1/6 for various failures that appear to be unfiled currently

I filed the failures in R1 and part of R6 in bug 862787. The input-file failures in R6 are bug 855352
Depends on: 869011
This was done a while ago :-)
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Product: Webtools → Tree Management
Component: TBPL → Visibility Requests
Product: Tree Management → Tree Management Graveyard
You need to log in before you can comment on or make changes to this bug.