840236 - Increase the number of mochitest chunks run for B2G

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Jonathan Griffin (:jgriffin)]

We're enabling more B2G mochitests; with the patch in bug 817638 the longest-running chunk is 86 minutes.

I believe we want to keep chunks to less than an hour, so we should increase the number of chunks from the current 6 to more, probably 9 to give us some more room to add additional tests in the near future.

Comment 1

[Jonathan Griffin (:jgriffin)]

Attachment: Increase number of B2G mochitest chunks to 9,

Comment 2

[Aki Sasaki (not active)]

Comment on attachment 712604 [details] [diff] [review]
Increase number of B2G mochitest chunks to 9,

This looks good, but we're missing ubuntu32-b2g here:
https://bugzilla.mozilla.org/show_bug.cgi?id=793022#c31
r=me with that section updated as well.

I'd like to get this easier to edit, but not sure how to do that atm.

Comment 3

[Jonathan Griffin (:jgriffin)]

Ah, thanks, I'd totally overlooked that.

Comment 4

[Jonathan Griffin (:jgriffin)]

Comment on attachment 712604 [details] [diff] [review]
Increase number of B2G mochitest chunks to 9,

http://hg.mozilla.org/build/buildbot-configs/rev/f402d5ef5513

Comment 5

[Jonathan Griffin (:jgriffin)]

Leaving open until this goes live during the next buildbot reconfig.

Comment 6

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to Jonathan Griffin (:jgriffin) from comment #0)
> We're enabling more B2G mochitests; with the patch in bug 817638 the
> longest-running chunk is 86 minutes.
Thanks for noticing that!

> I believe we want to keep chunks to less than an hour, so we should increase
> the number of chunks from the current 6 to more, probably 9 to give us some
> more room to add additional tests in the near future.
We actually want chunks in the 25-35min range. Lets watch timings after this "change to 9 chunks" rolls live in production, and see if we need to increase further.

Comment 7

[Phil Ringnalda (:philor)]

I'm pretty sure you meant "see if we need to move heaven and earth to find a way to reduce the overhead time, which is between a crazy 10 and 15 minutes for these test runs, because just blindly increasing the number of hunks to get down to 30 minute runtimes would mean approaching 50% setup/teardown time, and 30 or 40 hunks, and 7 or 8 hours of wasted time on our most critically understocked slaves if we ever managed to run all the hunks on one push which we wouldn't because we'd be constantly out of slaves and coalescing away all but maybe one hunk per push, unless we switch to the VMs, which are so much slower that it would be more like 50 or 60 hunks."

Comment 8

[Phil Ringnalda (:philor)]

Actually, I'd really like it if you would reconsider even this increase to 9, and back it out.

Trying to get to 30 minute run times is nice because it lets a person off the hook, they can stop watching their push, 30 minutes after their builds finish.

Gavin pushed to mozilla-beta at 10am today. He's still on the hook, and will be for a minimum of 11 hours, because his linux32 debug xpcshell test still hasn't started.

Once we move something, b2g or desktop linux, off the incredibly limited and completely insufficient number of hardware linux32 slaves, sure, worry about test runtime then. Until then, let's worry about not adding 30-45 minutes of wasted setup and teardown time to every push (in the increasingly unlike case where a push actually runs every test), and worry about not making it even easier to get into the situation we're in on inbound right now, where b2g mochitest-2 hasn't run since 8 hours ago, because every time a slave comes free, each job has an even smaller chance of being the one that gets it every time we add more hunks.

Comment 9

[Phil Ringnalda (:philor)]

Reftest, of which we're already running far too many hunks, is in even worse shape: we haven't run a b2g reftest-7 on inbound since 8:30 this morning.

Comment 10

[Jonathan Griffin (:jgriffin)]

Yes, that's a good point.  I'm fine with backing this out until we are running them on Amazon AWS VM's, if joduinn is.

Comment 11

[Aki Sasaki (not active)]

This is live.

Comment 12

[Jonathan Griffin (:jgriffin)]

I'm going to close this since this is live.  But, I agree this is causing our wait times to be worse due to the increased setup time.  Maybe we should move the discussion about whether to reduce the number of (mochitest/reftest) chunks for B2G to another bug?