Closed Bug 1411342 Opened 5 years ago Closed 5 years ago

consider reducing chunks for mochitest-headless now that we run on c3.large instance type


(Testing :: Mochitest, enhancement)

Not set


(firefox58 fixed)

Tracking Status
firefox58 --- fixed


(Reporter: jmaher, Assigned: jmaher)




(1 file)

now that bug 1411334 is landed, we should revisit the number of chunks so we are not spending more time setting up vs testing.
before on m1.medium we had:
opt: 11, 18, 18, 13, 13, 19, 18, 20, 20, 18
debug: 29, 49, 49, 22, 35, 58, 33, 42, 65, 51

after the change, we see:
opt: 7, 12, 9, 6, 8, 11, 12, 11, 11, 10
debug: 14, 25, 24, 11, 19, 30, 20, 22, 31, 26

I don't think we need to adjust debug, while 2 of the chunks are low (<15 minutes) the rest are still in the 20-30 minute range.

As for opt I think it would make sense to reduce to 7 or 8 chunks.

:gbrown, does this look reasonable to you?  also should we check out mochitest-plain, and browser-chrome?
Flags: needinfo?(gbrown)
Yes, 7 or 8 opt chunks looks about right. I think opt mochitest-plain could stand a similar reduction, if you like.
Flags: needinfo?(gbrown)
I noticed for osx and windows we run in 5 chunks- it would make sense to keep parity where possible.

In doing a try run and comparing one set of runtimes vs a single set of runtimes from m-c, I have a good bit of data:

Ideally I should compare across many runs and exclude all failing jobs, etc.

In summary above:
mochitest-plain: opt is good for 5 chunks
mochitest-bc: opt/debug is good for 7 chunks (not ccov/jsdcov as well)
mochitest-dt: linux64/opt is all that looks good

:gbrown- could you look through the data and see if my summary is good?
Flags: needinfo?(gbrown)
I didn't understand/trust the duration numbers in your spreadsheet, so I looked at the treeherder "Duration:"s in the links instead.

I agree with your summary, except I have a little concern about mochitest-bc on linux32/debug -- some of those times seem a little too long.

Generally, I think opt looks good and debug is marginal and probably needs more chunks.

Also, I don't understand and am a little suspicious: for mochitest-bc on linux32/debug how did you reduce the number of chunks from 16 to 7 with such a minimal effect on run-time?
Flags: needinfo?(gbrown)
I am really confused by the bc stuff to be honest, I just loaded up the two urls:



you can see the runtimes on m-c are really large (~53 minutes/chunk for all 16 chunks- in fact that is the runtime on the 7 chunks in my try push), and now I see why- this m-c revision is *before* the change from m1.medium -> c3.large (from bug 1408506)

So we are not comparing apples to apples.

A more modern push:

this recent push has ~23 minutes/chunk, which basically translates to a runtime of 53 minutes for 7 chunks.  I am inclined to leave debug alone and just modify opt everywhere.
(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #5)
> this recent push has ~23 minutes/chunk, which basically translates to a
> runtime of 53 minutes for 7 chunks.  I am inclined to leave debug alone and
> just modify opt everywhere.

That works for me!
Assignee: nobody → jmaher
Attachment #8922905 - Flags: review?(gbrown)
Attachment #8922905 - Flags: review?(gbrown) → review+
Pushed by
reduce chunks for mochitests on linux now that we run on faster hardware. r=gbrown
Pushed by
fix mochitest-devtools-chrome chunks definition to not break decision task. r=gbrown CLOSED TREE
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla58
You need to log in before you can comment on or make changes to this bug.