778954 - Unhide native Android mochitest-3 once its failure rate <20%

Status: RESOLVED FIXED

(Tree Management Graveyard :: Visibility Requests, defect)

Description

[Ed Morley [:emorley]]

Bug 777436 hid a number of test suites on Android that had a >30% failure rate (figures included all method of failure apart from blue retries). To unhide each suite, I would really like the failure rate to be <20%, given that even 30% was a bit excessive.

Rates from bug 777436 comment 3:

> Native:
> * mochitest-3          39%

Which after some test disabling in bug 775227, lowered to:

> Native:
> * mochitest-3: 76 failing, 78 total (97%)

We'll need to unhide on:
 * mozilla-central
 * mozilla-inbound
 * fx-team
 * services-central
 * try

Comment 1

[Ed Morley [:emorley]]

(In reply to Ed Morley [:edmorley] from comment #0)
> Which after some test disabling in bug 775227, lowered to:
> 
> > Native:
> > * mochitest-3: 76 failing, 78 total (97%)

Forgot to edit this after cloning the bug.

Clearly didn't lower!

All of the new failures are of type:
https://tbpl.mozilla.org/php/getParsedLog.php?id=13834506&tree=Mozilla-Inbound

ie: bug 772632 and friends.

Comment 2

[Ed Morley [:emorley]]

Last 24 hours:
28 failing, 29 total (97%)

Comment 3

[Ed Morley [:emorley]]

Now unhidden on inbound & m-c. Will unhide on rest once https://hg.mozilla.org/mozilla-central/rev/905bbf61dbe6 merges to branches.

Comment 4

[Ryan VanderMeulen [:RyanVM]]

This is currently 100% orange in /tests/dom/imptests/webapps/WebStorage/tests/submissions/Ms2ger/test_storage_session_in_js.html. I'm re-hiding it for the time being for sanity's sake.

Comment 5

[Ed Morley [:emorley]]

(In reply to Ryan VanderMeulen from comment #4)
> This is currently 100% orange in
> /tests/dom/imptests/webapps/WebStorage/tests/submissions/Ms2ger/
> test_storage_session_in_js.html. I'm re-hiding it for the time being for
> sanity's sake.

Appears to have been regressed by:
https://hg.mozilla.org/mozilla-central/rev/99fb6977d46c

Comment 6

[Ed Morley [:emorley]]

Looking at the last 10 inbound m3 runs:
* One instance of bug 780863 (just filed).
* One instance of bug 763166 (frequent on all the mochitests).
* Permaorange bug 781789 - but I've disabled the test for now.
* Permaorange bug 781837 - going to disable in a sec.

Comment 7

[Ed Morley [:emorley]]

Even after disabling all of:
dom/imptests/webapps/WebStorage/tests/submissions/*
dom/imptests/webapps/DOMCore/tests/submissions/*
dom/imptests/webapps/DOMCore/tests/approved/*
dom/imptests/editing/selecttest/*

...we're still getting timeouts - this time in dom/file/test/test_truncate.html, eg:
https://tbpl.mozilla.org/php/getParsedLog.php?id=14368637&tree=Mozilla-Inbound

To try and see if something different helps - I've pushed to try with test_ipc.html disabled (given comment 5 here and bug 770046 comment 17. Wait times are high, so might be a while before we get useful results:
https://tbpl.mozilla.org/?tree=Try&rev=f02ca4688136&noignore=1

Comment 8

[Ed Morley [:emorley]]

Right, ok disabling test_ipc.html turned the run green...

Comment 9

[Ed Morley [:emorley]]

I've tried again, but this time also reverting the other recent android.json changes, to see if we don't need to disable all the other tests, if we instead disable just test_ipc.html:
https://tbpl.mozilla.org/?tree=Try&rev=7394e04d70ca&noignore=1

Comment 10

[Ed Morley [:emorley]]

(In reply to Ed Morley [:edmorley] from comment #9)
> I've tried again, but this time also reverting the other recent android.json
> changes, to see if we don't need to disable all the other tests, if we
> instead disable just test_ipc.html:
> https://tbpl.mozilla.org/?tree=Try&rev=7394e04d70ca&noignore=1

No OOMs/failed runs, just a single test timed out in the newly landed test_browserElement_inproc_AppWindowNamespace.html

Have pushed another run with that disabled:
https://tbpl.mozilla.org/?tree=Try&rev=8834b189866b

Joel, presuming this comes back green - would you like me to disable test_ipc.html (I'll file an appropriate bug) and revert the other recent disablings as well (per the try patch) - or not do the latter?

Comment 11

[Ed Morley [:emorley]]

\o/ green, let's ship it!

Comment 12

[Ed Morley [:emorley]]

Unhidden on mozilla-central and inbound (and still green :-D).

Will do the others as and when the fixes propagate to them.

Comment 13

[Ed Morley [:emorley]]

Unhidden on all trees! \o/ :-D