945351 - Increase the number of chunks for B2G debug mochitests

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Jonathan Griffin (:jgriffin)]

We're currently running 9 chunks, but these may be too long (at any rate, they commonly exceed 60 minutes).  We should expand this to at least 12 chunks.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

jgriffin: thanks for filing this. Ideally we'd like each specific job to run 25-30 mins... If we can do that with "only" 12 chunks great... if we need more chunks, go for it!

Comment 2

[Jonathan Griffin (:jgriffin)]

It's very likely we'll need more chunks then...we can try with 15 initially.

Comment 3

[Jonathan Griffin (:jgriffin)]

AFAICT, there's no way to change just the number of debug chunks without introducing a new test suite name, e.g., mochitest-debug-1, instead of mochitest-1.  I think we are forced to choose either introducing a new suite name, or increasing the number of opt chunks as well.

We'd have to update TBPL to understand the new suite name, if we went that route, as well as downstream consumers like OrangeFactor.  On the other hand, increasing the number of opt chunks would increase our compute hours per commit, due to the overhead of the additional setup/teardown times.

releng, do you have a preference here?

Comment 4

[Aki Sasaki (not active)]

Hm, I don't have strong opinions, but:

assuming debug tests will take longer forever (or for the long term, at least), that would probably mean we want a different number of chunks for the long term, and the mochitest-debug-# would make sense.  I think this is the case, but I'm not sure.

If debug tests are going to converge with opt tests in terms of runtime in the short- or medium-term (not sure if this is possible, but if), increasing the total number of opt+debug chunks (or having slightly fewer debug chunks that take longer) might make sense.

Comment 5

[Jonathan Griffin (:jgriffin)]

I think debug tests will run much slower for the foreseeable future.  I'll go with the different suite name.

Comment 6

[Jonathan Griffin (:jgriffin)]

Attachment: Use 15 chunks for debug B2G mochitests,

Comment 7

[Aki Sasaki (not active)]

Attachment: builderdiff

This looks good.

Comment 8

[Aki Sasaki (not active)]

Comment on attachment 8341497 [details] [diff] [review]
Use 15 chunks for debug B2G mochitests,

Thanks Jonathan!

Comment 9

[Chris AtLee [:catlee]]

agree with Aki, just note that we'll see possibly different test behaviour because of different chunking between opt and debug.

Comment 10

[Phil Ringnalda (:philor)]

And different developer behavior, since most people think that if they look in one log and find that the test they are working on runs in (opt) m-5, then they can use try: -b do -p emulator -u mochitest-5 to run their test on opt and debug.

Comment 11

[Gregor Wagner [:gwagner]]

We can fix those issues if they become a real problem. Right now we have to get debug emulator tests running.

Comment 12

[Jonathan Griffin (:jgriffin)]

https://hg.mozilla.org/build/buildbot-configs/rev/7fd217b1cee3

Note that when this goes live, we'll need to update TBPL to handle the new suite name.

Comment 13

[John Hopkins (:jhopkins)]

in production