1329819 - Create 3 chunks for mochitest-media on TC linux64 opt/debug/pgo

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Jonathan Griffin (:jgriffin)]

According to the End to End times dashboard, https://people-mozilla.org/~klahnakoski/MoBuildbotTimings/End-to-End.html, mochitest-media is by far the longest-running suite on TaskCluster linux64 opt/debug, and the one we need to chunk in order to bring down our E2E times.

Let's initially split this into 3 chunks.

Comment 1

[Paul Yang [:pyang] (away)]

I ran 3 chunks of mochitest-media in several platforms
opt https://treeherder.mozilla.org/#/jobs?repo=try&revision=400fcce8889a7bbc4f44e732dbafe16fd86cc329
debug https://treeherder.mozilla.org/#/jobs?repo=try&revision=9cf4be8ddeb152103b7391a61ea196aa6d9d6490

linux64 debug reduced from 1 hour to max 40 minutes, avg. 33 minutes
linux64 opt reduced from 50 minutes to max 35 minutes, avg. 27 minutes

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

I would run each of the chunks 10 times each as each run is more variable than one would like to admit.

In addition we have linux64-asan (which is different from opt/debug).

Comment 3

[Paul Yang [:pyang] (away)]

Thanks!
Here're summary and each of chunks ran 10 time
opt https://treeherder.mozilla.org/#/jobs?repo=try&revision=400fcce8889a7bbc4f44e732dbafe16fd86cc329
debug https://treeherder.mozilla.org/#/jobs?repo=try&revision=9cf4be8ddeb152103b7391a61ea196aa6d9d6490
linux64 asan https://treeherder.mozilla.org/#/jobs?repo=try&revision=6dc2684aace7a6ba84f4a7c56f17ce2d02eb5218


For statistics I remove those extremely fast, since majority of each data set were close.
linux64 debug reduced from 1 hour to max 42 minutes (chunk1 avg. 32 min., chunk2 avg. 42 min., chunk3 avg. 22 min.)
linux64 opt reduced from 50 minutes to max 34 minutes (chunk1 avg. 25 min., chunk2 avg. 34 min., chunk3 avg. 19 min.)

Result of other platforms was included in the link.

Comment 4

[Paul Yang [:pyang] (away)]

Attachment: Bug 1329819 - Create 3 chunks for mochitest-media on TC linux64 opt/debug/pgo

Review commit: https://reviewboard.mozilla.org/r/105074/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/105074/

Comment 5

[Paul Yang [:pyang] (away)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=251fc5022721a78f44c98daf97d3719888627019&selectedJob=69509104&filter-searchStr=pgo
pgo results are close to opt build

Comment 6

[Kyle Lahnakoski [:ekyle]]

Attachment: Try Runs

You can see the sample runs as orange, blue (and green) dots in the test durations chart.

https://people-mozilla.org/~klahnakoski/MoBuildbotTimings/TaskCluster-Tests-Detail.html#sampleMax=2017-01-18&sampleMin=2017-01-11&showWaitTime=false&test=mochitest-media&product=Firefox&type=OPT&platform=Linux32&branch=Try

I attached an image, just in case.

Comment 7

[Paul Yang [:pyang] (away)]

Thanks Kyle for rich dashboard! 
I can see about 15 to 20 minutes reduced by this change.  However if there's a way to split them by running minutes would be nicer.
Here is another result using chunks=5
https://treeherder.mozilla.org/#/jobs?repo=try&author=[e-mail]&selectedJob=74236498
about 10 more minutes eliminated. Considering longest running task after media suite split is 82 minutes (windows2012 64bit pgo build), we might not gain a lot to have 5 chunks.

jgriffin, do we need more observation or simply move to next target?

Comment 8

[Jonathan Griffin (:jgriffin)]

We're explicitly excluding PGO from our efforts to reduce end-to-end times, because there isn't a lot we can do to improve them.

I think the averages you gave are pretty reasonable, and I think we can green up the tests and finish this.

Comment 9

[Jonathan Griffin (:jgriffin)]

on the try runs in comment #7, there seem to be perma-fails in chunks 3 and 4 on debug. I don't seem them with just three chunks from try runs in comment #3. Have you investigated these yet?

Comment 10

[Paul Yang [:pyang] (away)]

I retry by 5 chunks with latest m-c.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=8866c2f74dcebd36d9b6ab9ea775c28c86785c3c

We can't see same issue reproduced so possibly a bug fixed.

However in mda3 it showed "INFO - [Child 2098] WARNING: Late NotifyFinished after MediaManager shutdown: file /home/worker/workspace/build/src/dom/media/MediaManager.cpp,"
then test process froze and kept complaining message dropped.

Might be a teardown issue but need more investigating.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=8ff374e9906144befe59d2d262730bdc32648162
While 3 chunks version kept being green.

Comment 11

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8827377 [details]
Bug 1329819 - Create 3 chunks for mochitest-media on TC linux64 opt/debug/pgo

https://reviewboard.mozilla.org/r/105074/#review113052

Comment 12

[Jonathan Griffin (:jgriffin)]

Great, thanks!

Comment 13

[Jonathan Griffin (:jgriffin)]

Is this good to land now?

Comment 14

[Paul Yang [:pyang] (away)]

Comment on attachment 8827377 [details]
Bug 1329819 - Create 3 chunks for mochitest-media on TC linux64 opt/debug/pgo

I think it's good to go.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e5b189f0363b
Create 3 chunks for mochitest-media on TC linux64 opt/debug/pgo r=jgriffin

Comment 16

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/e5b189f0363b