1755075 - split mochitest-media webrtc tests into smaller manifests

Status: RESOLVED FIXED

(Testing :: Mochitest, task)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

there are concerns that we get too many failure on the webrtc tests in both mochitest and web-platform-tests. One thing we can do is help isolate the failures to prevent bleedthrough. In doing that we can easily split a large mochitest.ini into multiple manifests.

I did a --rebuild 10 and saw very few failures (13 total out of ~500 jobs - 7 failures were timeout/shutdown hangs):
https://treeherder.mozilla.org/jobs?repo=try&revision=11da11dc96132c5ae4db10319b285df9545f564d

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

:bwc, does this help the problem much? ignoring the 7 failures, that leaves 6 test failures which is just a hair >1% failure rate. That is pretty good for a large volume of tests.

I am happy to submit my patch if you like this and I don't see any drawbacks to maintenance going forward.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

3 shutdown hangs on linux/debug were waiting for shutdown after finishing: dom/media/test/mochitest.ini - maybe split this up into 5-8 manifests?
4 shutdown hangs on osx/debug were waiting for shutdown after finishing: dom/media/webaudio/test/ - maybe split this into 3 manifests?

Comment 3

[Byron Campen [:bwc]]

Most of our tests aren't running here; I only see the identity webrtc tests in these logs. I don't think the new manifests are being picked up?

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1755075 - split up webrtc tests and dom tests for smaller manifest chunks. r=bwc!,alwu!

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1755075 - split up webrtc tests and dom tests for smaller manifest chunks. r=bwc!,alwu!

https://phabricator.services.mozilla.com/D139153

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/887c82bbb32d
split up webrtc tests and dom tests for smaller manifest chunks. r=bwc,media-playback-reviewers,alwu

Comment 8

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/887c82bbb32d

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

based off of mozilla-central commits (runs all tests) it seems like the week prior we had ~3 failures/push and the 5 days after landing we are at ~1.5 failures/push. A slight win. We run 109 mochitest-media jobs (tier-1 + 2). 1% total failure rate for intermittents is pretty good for a large suite, almost there- but other big fish to fry