853024 - Un-hide B2G emulator reftests, crashtests, and mochitests when failure rate is no longer unacceptably high

Status: RESOLVED FIXED

(Tree Management Graveyard :: Visibility Requests, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

The failure rate of the B2G emulator reftests/crashtests/mochitests is currently 25+%, which is unacceptably high. Until the situation is resolved, I have hidden these tests on mozilla-central and all branches that feed directly into it.

Comment 1

[Ed Morley [:emorley]]

(For reference) 

To view _just_ the B2G jobs, use the TBPL job name filter, eg:

mozilla-central:
https://tbpl.mozilla.org/?showall=1&jobname=b2g

mozilla-inbound:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&showall=1&jobname=b2g

Comment 2

[Ed Morley [:emorley]]

Had this via email:
{
quick question : i've been trying to land something and it's been held up due to oranges on try with the b2g mochitests,
which seem unrelated to my changes 

i just looked at quite a few recent patches to m-c and m-i and it seems like we aren't running the b2g
mochitests on the emulator on them ? 
}

-> Hidden on Try as well to avoid confusion.

Comment 3

[Jonathan Griffin (:jgriffin)]

With a few recent fixes, mochitests are nearly always green on inbound.  On the last 20 runs, there was only 1 intermittent failure (a socket.timeout), excepting some failures that were solved by a backout.  Therefore, I plan on unhiding mochitests only today unless I hear some objections; reftests will remain hidden as their failure rate continues to be very high.

Comment 4

[Ryan VanderMeulen [:RyanVM]]

Sounds great, thanks Johathan!

Comment 5

[Jonathan Griffin (:jgriffin)]

I've unhidden B2G mochitests on m-i, m-c, and fx-team (they were already visible on try and services-central), and starred the last couple of days worth of failures (only a handful of these existed).

Comment 6

[Ed Morley [:emorley]]

(Moving to TBPL's component, since that's where the unhiding takes place, and is the component sheriffs watch. Fixing the failure rate to get us to the point where we can unhide is covered by the dependant bugs, which are in B2G/testing components)

Comment 8

[Ryan VanderMeulen [:RyanVM]]

I have un-hidden all B2G reftests and crashtests on m-c/inbound/birch except for:
* crashtest-1 due to bug 852821
* reftest-1/6 for various failures that appear to be unfiled currently

Comment 9

[Phil Ringnalda (:philor)]

R6 is filed (twice, since I didn't find bug 855352 before I filed bug 856186).

Comment 10

[Andrew Halberstadt [:ahal]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #8)
> I have un-hidden all B2G reftests and crashtests on m-c/inbound/birch except
> for:
> * crashtest-1 due to bug 852821
> * reftest-1/6 for various failures that appear to be unfiled currently

I filed the failures in R1 and part of R6 in bug 862787. The input-file failures in R6 are bug 855352

Comment 11

[Ed Morley [:emorley]]

This was done a while ago :-)