Closed Bug 1411342 Opened 5 years ago Closed 5 years ago

consider reducing chunks for mochitest-headless now that we run on c3.large instance type

Categories

(Testing :: Mochitest, enhancement)

enhancement
Not set
normal

Tracking

(firefox58 fixed)

RESOLVED FIXED
mozilla58
Tracking Status
firefox58 --- fixed

People

(Reporter: jmaher, Assigned: jmaher)

References

Details

Attachments

(1 file)

now that bug 1411334 is landed, we should revisit the number of chunks so we are not spending more time setting up vs testing.
before on m1.medium we had:
opt: 11, 18, 18, 13, 13, 19, 18, 20, 20, 18
debug: 29, 49, 49, 22, 35, 58, 33, 42, 65, 51

after the change, we see:
opt: 7, 12, 9, 6, 8, 11, 12, 11, 11, 10
debug: 14, 25, 24, 11, 19, 30, 20, 22, 31, 26


I don't think we need to adjust debug, while 2 of the chunks are low (<15 minutes) the rest are still in the 20-30 minute range.

As for opt I think it would make sense to reduce to 7 or 8 chunks.

:gbrown, does this look reasonable to you?  also should we check out mochitest-plain, and browser-chrome?
Flags: needinfo?(gbrown)
Yes, 7 or 8 opt chunks looks about right. I think opt mochitest-plain could stand a similar reduction, if you like.
Flags: needinfo?(gbrown)
I noticed for osx and windows we run in 5 chunks- it would make sense to keep parity where possible.

In doing a try run and comparing one set of runtimes vs a single set of runtimes from m-c, I have a good bit of data:
https://docs.google.com/a/mozilla.com/spreadsheets/d/1FVLkCG35LFF-fMJ6FFF1rCZAZezaN6YYq0e3KtjVlZU/edit?usp=sharing

Ideally I should compare across many runs and exclude all failing jobs, etc.


In summary above:
mochitest-plain: opt is good for 5 chunks
mochitest-bc: opt/debug is good for 7 chunks (not ccov/jsdcov as well)
mochitest-dt: linux64/opt is all that looks good

:gbrown- could you look through the data and see if my summary is good?
Flags: needinfo?(gbrown)
I didn't understand/trust the duration numbers in your spreadsheet, so I looked at the treeherder "Duration:"s in the links instead.

I agree with your summary, except I have a little concern about mochitest-bc on linux32/debug -- some of those times seem a little too long.

Generally, I think opt looks good and debug is marginal and probably needs more chunks.

Also, I don't understand and am a little suspicious: for mochitest-bc on linux32/debug how did you reduce the number of chunks from 16 to 7 with such a minimal effect on run-time?
Flags: needinfo?(gbrown)
I am really confused by the bc stuff to be honest, I just loaded up the two urls:

try:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=5aba5223f064e77c586ce5b5adfcf6a20977d094&filter-searchStr=linux%20debug%20browser-chrome%20stylo&selectedJob=139980986

m-c:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=9056f2ee492f&filter-searchStr=linux%20debug%20browser-chrome%20stylo&selectedJob=139018464

you can see the runtimes on m-c are really large (~53 minutes/chunk for all 16 chunks- in fact that is the runtime on the 7 chunks in my try push), and now I see why- this m-c revision is *before* the change from m1.medium -> c3.large (from bug 1408506)

So we are not comparing apples to apples.

A more modern push:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=ae49d4a5762264ded3aae4006baddc2203b79b94&filter-searchStr=linux%20stylo%20debug%20browser-chrome

this recent push has ~23 minutes/chunk, which basically translates to a runtime of 53 minutes for 7 chunks.  I am inclined to leave debug alone and just modify opt everywhere.
(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #5)
> this recent push has ~23 minutes/chunk, which basically translates to a
> runtime of 53 minutes for 7 chunks.  I am inclined to leave debug alone and
> just modify opt everywhere.

That works for me!
Assignee: nobody → jmaher
Status: NEW → ASSIGNED
Attachment #8922905 - Flags: review?(gbrown)
Attachment #8922905 - Flags: review?(gbrown) → review+
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/801369e5614a
reduce chunks for mochitests on linux now that we run on faster hardware. r=gbrown
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/2a8eb883a279
fix mochitest-devtools-chrome chunks definition to not break decision task. r=gbrown CLOSED TREE
https://hg.mozilla.org/mozilla-central/rev/801369e5614a
https://hg.mozilla.org/mozilla-central/rev/2a8eb883a279
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla58
You need to log in before you can comment on or make changes to this bug.