Closed Bug 1264772 Opened 8 years ago Closed 8 years ago

Schedule android mochitest-media as --chunk-by-runtime

Categories

(Testing :: General, defect)

Version 3
defect
Not set
normal

Tracking

(firefox48 fixed)

RESOLVED FIXED
mozilla48
Tracking Status
firefox48 --- fixed

People

(Reporter: dminor, Assigned: dminor)

Details

Attachments

(5 files)

The mochitest-media tests on Android are currently quite unbalanced. The first chunk takes around ~55 minutes to run, the second ~12 minutes. Using --chunk-by-runtime should help with this.
Initial try run here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=1cef2c4a4a23e6048156b522c721e8690e471cfd&selectedJob=19501270

It's not as balanced as I had hoped, but it has the nice side effect of putting all of the WebRTC tests in one (slow) chunk so I think it's worthwhile going forward with this change.
you could consider splitting up the webrtc chunks into 2 manifests, then chunk-by-runtime would calculate the weight for each manifest independently and do a better job balancing :)
Yes chunk-by-runtime will keep manifests of tests as one atomic unit (to avoid test dependency issues). So if the mda tests live in one or two very large manifests, then it won't help very much.
This fixes a few typos in the script and updates the README with
instructions on how to run it.

Review commit: https://reviewboard.mozilla.org/r/46515/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/46515/
Attachment #8741484 - Flags: review?(ahalberstadt)
Attachment #8741485 - Flags: review?(ahalberstadt)
Attachment #8741486 - Flags: review?(ahalberstadt)
Attachment #8741487 - Flags: review?(ahalberstadt)
We currently hard code the runtime name to be mochitest-plain. This
changes things to use the subsuite name if available.

Review commit: https://reviewboard.mozilla.org/r/46519/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/46519/
(In reply to Joel Maher (:jmaher) from comment #2)
> you could consider splitting up the webrtc chunks into 2 manifests, then
> chunk-by-runtime would calculate the weight for each manifest independently
> and do a better job balancing :)

I'm hoping we'll be able to eventually run these tests on Autophone rather than the emulator. They only take 5 - 10 minutes to run there.
Comment on attachment 8741484 [details]
MozReview Request: Bug 1264772 - Fixup writeruntimes.py; r=ahal

https://reviewboard.mozilla.org/r/46515/#review43325
Attachment #8741484 - Flags: review?(ahalberstadt) → review+
Attachment #8741485 - Flags: review?(ahalberstadt) → review+
Comment on attachment 8741485 [details]
MozReview Request: Bug 1264772 - Add runtimes file for mochitest-media; r=ahal

https://reviewboard.mozilla.org/r/46517/#review43327
Comment on attachment 8741486 [details]
MozReview Request: Bug 1264772 - Use subsuite name for chunk-by-runtime; r=ahal

https://reviewboard.mozilla.org/r/46519/#review43331
Attachment #8741486 - Flags: review?(ahalberstadt) → review+
Comment on attachment 8741487 [details]
MozReview Request: Bug 1264772 - Set chunk-by-runtime option for mochitest-media on android; r=ahal

https://reviewboard.mozilla.org/r/46521/#review43333
Attachment #8741487 - Flags: review?(ahalberstadt) → review+
Thanks for the review!

I'm going to hold off landing this until I figure out whether this made the WebRTC mochitests on debug builds more intermittent or if they've always been horrible.
Interestingly, it's the first test that is failing pretty consistently [1]. I wonder if there is some kind of required start up time that is no longer elapsing now that we're running a WebRTC test right off the bat.

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=d42a554f893dfe404566c13f1f82d00532b28aab
sounds very likely- or a previous test which ran and set the state of firefox/prefs to something needed.
I added a no-op test and things seem much stabler [1].

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=886a309ae28cb8e5cc7f0978cf92a56c3c9b738d
The first test in the group seems to have a disproportionate number of
intermittents. Adding a no-op test to the beginning of the group helps
things.

Review commit: https://reviewboard.mozilla.org/r/47197/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/47197/
Attachment #8741484 - Attachment description: MozReview Request: Bug 1264772 - Fixup writeruntimes.py; r?ahal → MozReview Request: Bug 1264772 - Fixup writeruntimes.py; r=ahal
Attachment #8741485 - Attachment description: MozReview Request: Bug 1264772 - Add runtimes file for mochitest-media; r?ahal → MozReview Request: Bug 1264772 - Add runtimes file for mochitest-media; r=ahal
Attachment #8741486 - Attachment description: MozReview Request: Bug 1264772 - Use subsuite name for chunk-by-runtime; r?ahal → MozReview Request: Bug 1264772 - Use subsuite name for chunk-by-runtime; r=ahal
Attachment #8741487 - Attachment description: MozReview Request: Bug 1264772 - Set chunk-by-runtime option for mochitest-media on android; r?ahal → MozReview Request: Bug 1264772 - Set chunk-by-runtime option for mochitest-media on android; r=ahal
Attachment #8742407 - Flags: review?(drno)
Comment on attachment 8741484 [details]
MozReview Request: Bug 1264772 - Fixup writeruntimes.py; r=ahal

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/46515/diff/1-2/
Comment on attachment 8741485 [details]
MozReview Request: Bug 1264772 - Add runtimes file for mochitest-media; r=ahal

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/46517/diff/1-2/
Comment on attachment 8741486 [details]
MozReview Request: Bug 1264772 - Use subsuite name for chunk-by-runtime; r=ahal

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/46519/diff/1-2/
Comment on attachment 8741487 [details]
MozReview Request: Bug 1264772 - Set chunk-by-runtime option for mochitest-media on android; r=ahal

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/46521/diff/1-2/
Comment on attachment 8742407 [details]
MozReview Request: Bug 1264772 - Add no-op WebRTC Mochitest; r=drno

https://reviewboard.mozilla.org/r/47197/#review43981

::: dom/media/tests/mochitest/test_a_noOp.html:16
(Diff revision 1)
> +</head>
> +<body>
> +<a target="_blank" href="https://bugzilla.mozilla.org/show_bug.cgi?id=1264772">Mozilla Bug 1264772</a>
> +<p id="display"></p>
> +<div id="content" style="display: none">
> +  

Nit: WS
Attachment #8742407 - Flags: review?(drno) → review+
Comment on attachment 8742407 [details]
MozReview Request: Bug 1264772 - Add no-op WebRTC Mochitest; r=drno

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/47197/diff/1-2/
Attachment #8742407 - Attachment description: MozReview Request: Bug 1264772 - Add no-op WebRTC Mochitest; r?drno → MozReview Request: Bug 1264772 - Add no-op WebRTC Mochitest; r=drno
You need to log in before you can comment on or make changes to this bug.