908306 - Web Audio crashtests are very timeout-prone on B2G

Status: RESOLVED WORKSFORME

(Core :: Web Audio, defect, P3)

Description

[Ryan VanderMeulen [:RyanVM]]

We have been fighting extremely frequent (near perma-fail) timeouts in the Web Audio crashtests on B2G. To date, we've already had to disable a few tests in the past, but that only bought us some time. I will soon be pushing another set of disablings. If that doesn't work, I'm afraid we're at a point where we're going to have to consider disabling the entire set of tests. Whack-A-Mole is not an effective strategy for dealing with test failures.

This bug is filed for finding the root cause of these failures, fixing it, and re-enabling the tests.

Comment 1

[Ryan VanderMeulen [:RyanVM]]

Consider this our last attempt at not disabling everything.
https://hg.mozilla.org/integration/b2g-inbound/rev/2e7089db4e86

Comment 2

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/2e7089db4e86

Comment 3

[(no longer active)]

jwwang, do you mind investigating this, please?  It sucks that we're skipping all of these tests on Firefox OS... :/

Comment 4

[Karl Tomlinson (:karlt)]

Perhaps related to bug 908462.

Comment 5

[Ryan VanderMeulen [:RyanVM]]

These have been flaky for awhile, but it seems that it started getting out of hand around Wednesday.

https://tbpl.mozilla.org/?jobname=b2g_emulator_vm%20mozilla-central%20opt%20test%20crashtest-1

You can hit the green arrow at the bottom to show more jobs going back further.

Comment 6

[JW Wang [:jwwang] [:jw_wang]]

In the middle of other major issues. I will come back at this later.

Comment 7

[Karl Tomlinson (:karlt)]

30 or so runs with 2e7089db4e86 reverted and CC debugging enabled are not reproducing:

https://tbpl.mozilla.org/?tree=Try&rev=bf6bdabf1476

Comment 8

[Karl Tomlinson (:karlt)]

40 or so runs with 2e7089db4e86 reverted and no CC debugging  didn't reproduce either https://tbpl.mozilla.org/?tree=Try&rev=be5f7bcdf2f5

This and comment 7 based on revision 8c5a94ba1096 plus some local patches.

Comment 9

[Karl Tomlinson (:karlt)]

Not certain this is related but crashtest 907986.html that I added on try is intermittently timing out only on B2G emulator.  CC timing shows that CC is happening but not taking more than a fraction of a second => the problem is not CC.
https://hg.mozilla.org/try/rev/d7318bb842af
https://tbpl.mozilla.org/php/getParsedLog.php?id=27309980&tree=Try&full=1#error2

Comment 10

[Ed Morley [:emorley]]

Thank you for your persistence with this! :-)

Comment 11

[Karl Tomlinson (:karlt)]

When 907986.html times out, the ScriptProcessorNode never gets the "audioprocess" event.

The audioprocess count logging in the logs of successful runs was to observe the MSG thread spamming the main thread with up to 100 more audioprocess events than the main thread wanted.

https://tbpl.mozilla.org/?tree=Try&rev=6bedbed18eaf

Comment 12

[Karl Tomlinson (:karlt)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=27398230&tree=Mozilla-Inbound#error2
b2g_emulator_vm mozilla-inbound opt test crashtest-1 on 2013-09-04 15:41:47 PDT for push 4a5e5f94a4f7

TEST-UNEXPECTED-FAIL | http://10.0.2.2:8888/tests/content/media/test/crashtests/876249.html | application timed out after 330 seconds with no output

Comment 13

[Karl Tomlinson (:karlt)]

(In reply to Karl Tomlinson (:karlt) from comment #11)
> When 907986.html times out, the ScriptProcessorNode never gets the
> "audioprocess" event.

That could be bug 916773.

Comment 14

[Karl Tomlinson (:karlt)]

Repeated the experiments of comment 7 and 8, and still no timeouts:
https://tbpl.mozilla.org/?tree=Try&rev=2b9ca7dcbbf6

So reenabled most tests.  880202.html is still disabled, not because I have reason to suspect that test, but because I missed that test in my experiments.  If there are no reports in the next few days, I'll look at enabling 880202.html.  At least crashtests should have stack traces on timeouts now.

https://hg.mozilla.org/integration/mozilla-inbound/rev/70cf74d59b55

Comment 15

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/70cf74d59b55

Comment 16

[JW Wang [:jwwang] [:jw_wang]]

Release the bug so that someone can take it. I will continue working on media tests.

Comment 17

[Ryan VanderMeulen [:RyanVM]]

These tests are mostly enabled on B2G today, and the few disabled tests have their own tracking bugs annotated in the manifest. Closing this out.