1411342 - consider reducing chunks for mochitest-headless now that we run on c3.large instance type

Status: RESOLVED FIXED

(Testing :: Mochitest, enhancement)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

now that bug 1411334 is landed, we should revisit the number of chunks so we are not spending more time setting up vs testing.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

before on m1.medium we had:
opt: 11, 18, 18, 13, 13, 19, 18, 20, 20, 18
debug: 29, 49, 49, 22, 35, 58, 33, 42, 65, 51

after the change, we see:
opt: 7, 12, 9, 6, 8, 11, 12, 11, 11, 10
debug: 14, 25, 24, 11, 19, 30, 20, 22, 31, 26


I don't think we need to adjust debug, while 2 of the chunks are low (<15 minutes) the rest are still in the 20-30 minute range.

As for opt I think it would make sense to reduce to 7 or 8 chunks.

:gbrown, does this look reasonable to you?  also should we check out mochitest-plain, and browser-chrome?

Comment 2

[Geoff Brown [:gbrown]]

Yes, 7 or 8 opt chunks looks about right. I think opt mochitest-plain could stand a similar reduction, if you like.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

I noticed for osx and windows we run in 5 chunks- it would make sense to keep parity where possible.

In doing a try run and comparing one set of runtimes vs a single set of runtimes from m-c, I have a good bit of data:
https://docs.google.com/a/mozilla.com/spreadsheets/d/1FVLkCG35LFF-fMJ6FFF1rCZAZezaN6YYq0e3KtjVlZU/edit?usp=sharing

Ideally I should compare across many runs and exclude all failing jobs, etc.


In summary above:
mochitest-plain: opt is good for 5 chunks
mochitest-bc: opt/debug is good for 7 chunks (not ccov/jsdcov as well)
mochitest-dt: linux64/opt is all that looks good

:gbrown- could you look through the data and see if my summary is good?

Comment 4

[Geoff Brown [:gbrown]]

I didn't understand/trust the duration numbers in your spreadsheet, so I looked at the treeherder "Duration:"s in the links instead.

I agree with your summary, except I have a little concern about mochitest-bc on linux32/debug -- some of those times seem a little too long.

Generally, I think opt looks good and debug is marginal and probably needs more chunks.

Also, I don't understand and am a little suspicious: for mochitest-bc on linux32/debug how did you reduce the number of chunks from 16 to 7 with such a minimal effect on run-time?

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

I am really confused by the bc stuff to be honest, I just loaded up the two urls:

try:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=5aba5223f064e77c586ce5b5adfcf6a20977d094&filter-searchStr=linux%20debug%20browser-chrome%20stylo&selectedJob=139980986

m-c:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=9056f2ee492f&filter-searchStr=linux%20debug%20browser-chrome%20stylo&selectedJob=139018464

you can see the runtimes on m-c are really large (~53 minutes/chunk for all 16 chunks- in fact that is the runtime on the 7 chunks in my try push), and now I see why- this m-c revision is *before* the change from m1.medium -> c3.large (from bug 1408506)

So we are not comparing apples to apples.

A more modern push:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=ae49d4a5762264ded3aae4006baddc2203b79b94&filter-searchStr=linux%20stylo%20debug%20browser-chrome

this recent push has ~23 minutes/chunk, which basically translates to a runtime of 53 minutes for 7 chunks.  I am inclined to leave debug alone and just modify opt everywhere.

Comment 6

[Geoff Brown [:gbrown]]

(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #5)
> this recent push has ~23 minutes/chunk, which basically translates to a
> runtime of 53 minutes for 7 chunks.  I am inclined to leave debug alone and
> just modify opt everywhere.

That works for me!

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: reduce chunks and cleanup some redundancies

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/801369e5614a
reduce chunks for mochitests on linux now that we run on faster hardware. r=gbrown

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/2a8eb883a279
fix mochitest-devtools-chrome chunks definition to not break decision task. r=gbrown CLOSED TREE

Comment 10

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/801369e5614a
https://hg.mozilla.org/mozilla-central/rev/2a8eb883a279