Closed Bug 1329819 Opened 5 years ago Closed 5 years ago

Create 3 chunks for mochitest-media on TC linux64 opt/debug/pgo

Categories

(Testing :: General, defect)

defect
Not set
normal

Tracking

(firefox54 fixed)

RESOLVED FIXED
mozilla54
Tracking Status
firefox54 --- fixed

People

(Reporter: jgriffin, Assigned: pyang)

References

(Blocks 1 open bug)

Details

Attachments

(2 files)

According to the End to End times dashboard, https://people-mozilla.org/~klahnakoski/MoBuildbotTimings/End-to-End.html, mochitest-media is by far the longest-running suite on TaskCluster linux64 opt/debug, and the one we need to chunk in order to bring down our E2E times.

Let's initially split this into 3 chunks.
I ran 3 chunks of mochitest-media in several platforms
opt https://treeherder.mozilla.org/#/jobs?repo=try&revision=400fcce8889a7bbc4f44e732dbafe16fd86cc329
debug https://treeherder.mozilla.org/#/jobs?repo=try&revision=9cf4be8ddeb152103b7391a61ea196aa6d9d6490

linux64 debug reduced from 1 hour to max 40 minutes, avg. 33 minutes
linux64 opt reduced from 50 minutes to max 35 minutes, avg. 27 minutes
I would run each of the chunks 10 times each as each run is more variable than one would like to admit.

In addition we have linux64-asan (which is different from opt/debug).
Thanks!
Here're summary and each of chunks ran 10 time
opt https://treeherder.mozilla.org/#/jobs?repo=try&revision=400fcce8889a7bbc4f44e732dbafe16fd86cc329
debug https://treeherder.mozilla.org/#/jobs?repo=try&revision=9cf4be8ddeb152103b7391a61ea196aa6d9d6490
linux64 asan https://treeherder.mozilla.org/#/jobs?repo=try&revision=6dc2684aace7a6ba84f4a7c56f17ce2d02eb5218


For statistics I remove those extremely fast, since majority of each data set were close.
linux64 debug reduced from 1 hour to max 42 minutes (chunk1 avg. 32 min., chunk2 avg. 42 min., chunk3 avg. 22 min.)
linux64 opt reduced from 50 minutes to max 34 minutes (chunk1 avg. 25 min., chunk2 avg. 34 min., chunk3 avg. 19 min.)

Result of other platforms was included in the link.
Thanks Kyle for rich dashboard! 
I can see about 15 to 20 minutes reduced by this change.  However if there's a way to split them by running minutes would be nicer.
Here is another result using chunks=5
https://treeherder.mozilla.org/#/jobs?repo=try&author=pyang@mozilla.com&selectedJob=74236498
about 10 more minutes eliminated. Considering longest running task after media suite split is 82 minutes (windows2012 64bit pgo build), we might not gain a lot to have 5 chunks.

jgriffin, do we need more observation or simply move to next target?
Flags: needinfo?(jgriffin)
We're explicitly excluding PGO from our efforts to reduce end-to-end times, because there isn't a lot we can do to improve them.

I think the averages you gave are pretty reasonable, and I think we can green up the tests and finish this.
Flags: needinfo?(jgriffin)
Attachment #8827377 - Flags: review?(jgriffin)
on the try runs in comment #7, there seem to be perma-fails in chunks 3 and 4 on debug. I don't seem them with just three chunks from try runs in comment #3. Have you investigated these yet?
I retry by 5 chunks with latest m-c.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=8866c2f74dcebd36d9b6ab9ea775c28c86785c3c

We can't see same issue reproduced so possibly a bug fixed.

However in mda3 it showed "INFO - [Child 2098] WARNING: Late NotifyFinished after MediaManager shutdown: file /home/worker/workspace/build/src/dom/media/MediaManager.cpp,"
then test process froze and kept complaining message dropped.

Might be a teardown issue but need more investigating.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=8ff374e9906144befe59d2d262730bdc32648162
While 3 chunks version kept being green.
Comment on attachment 8827377 [details]
Bug 1329819 - Create 3 chunks for mochitest-media on TC linux64 opt/debug/pgo

https://reviewboard.mozilla.org/r/105074/#review113052
Attachment #8827377 - Flags: review?(jgriffin) → review+
Great, thanks!
Is this good to land now?
Comment on attachment 8827377 [details]
Bug 1329819 - Create 3 chunks for mochitest-media on TC linux64 opt/debug/pgo

I think it's good to go.
Attachment #8827377 - Flags: checked-in?
Pushed by cbook@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/e5b189f0363b
Create 3 chunks for mochitest-media on TC linux64 opt/debug/pgo r=jgriffin
Keywords: checkin-needed
https://hg.mozilla.org/mozilla-central/rev/e5b189f0363b
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla54
You need to log in before you can comment on or make changes to this bug.