1142743 - Increase chunk count for Android 2.3 mochitest-gl jobs

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Kelsey Gilbert [:jgilbert]]

I'm adding a bunch of tests, just about doubling the number of test files, and tripling the number of test assertions.

mochitest-gl2 is currently running 60-70 minutes so far, but we hit a couple tests that time out and kill the run. (4 timeouts = abort?)[1]
I tried skipping those tests, but it seems like tests almost immediately after that are timing out too.[2]
I'm disabling those and trying again, but I'm concerned we're at the machine's limit.

Further, it looks like there are plenty of tests (over 280 test files) still to run.[3] We probably don't want tests that take this long to run anyways.

I propose we double our chunk count to 4 and see where that gets us.

[1]: https://treeherder.mozilla.org/#/jobs?repo=try&revision=cff3f51fb695
[2]: https://treeherder.mozilla.org/#/jobs?repo=try&revision=c96128a4b34b
[3]:
> 548 INFO TEST-UNEXPECTED-FAIL | (SimpleTest/TestRunner.js) | 4 test timeouts, giving up. - expected PASS
> 549 INFO TEST-UNEXPECTED-FAIL | (SimpleTest/TestRunner.js) | Skipping 285 remaining tests. - expected PASS

Comment 1

[Kim Moir [:kmoir] ET]

Attachment: bug1142743bbconfigs.patch

Comment 2

[Kim Moir [:kmoir] ET]

Attachment: bug1142743mh.patch

Comment 3

[Kim Moir [:kmoir] ET]

Attachment: bug1142743builder.diff

builder diff

Comment 4

[Jordan Lund (:jlund)]

Comment on attachment 8577303 [details] [diff] [review]
bug1142743mh.patch

Review of attachment 8577303 [details] [diff] [review]:
-----------------------------------------------------------------

shweet, should achieve what we want. couple sanity checks on the rollout:

- is orange expected here for all for chunks: https://treeherder.mozilla.org/#/jobs?repo=try&revision=cff3f51fb695 ?

- should we be riding trains on this instead of changing across all our builders? e.g.
   - does anything in tree have to land in-tree on any of the release branches? robocop chunking just got bit by this: https://bugzilla.mozilla.org/show_bug.cgi?id=1142553#c11
   - mozilla-release is still on pre-apk-split: "Android 2.3 Emulator mozilla-release opt test mochitest-gl-3" (from your builderlist). I doubt this will affect things but it's worth pointing out in case we have new tests that assume the split.

- also, judging by treeherder try link in first comment, this will need a treeherder patch too

Comment 5

[Jordan Lund (:jlund)]

Comment on attachment 8577301 [details] [diff] [review]
bug1142743bbconfigs.patch

Review of attachment 8577301 [details] [diff] [review]:
-----------------------------------------------------------------

see https://bugzilla.mozilla.org/show_bug.cgi?id=1142743#c4 for sanity check questions on the pair of these patches.

also, just thinking, the mh patch will need to have a mozharness.json pin bump across all branches

Comment 6

[Kelsey Gilbert [:jgilbert]]

(In reply to Jordan Lund (:jlund) from comment #4)
> Comment on attachment 8577303 [details] [diff] [review]
> bug1142743mh.patch
> 
> Review of attachment 8577303 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> shweet, should achieve what we want. couple sanity checks on the rollout:
> 
> - is orange expected here for all for chunks:
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=cff3f51fb695 ?
Yeah, there's hundreds of new test files and our infra drivers are pretty bad. A number of test failures, crashes, and timeouts still need to be marked.

> - should we be riding trains on this instead of changing across all our
> builders? e.g.
It might help with existing intermittents on the platform, but otherwise we've been OK without this bug's increase so far. (We just need a bigger split moving forward)
>    - does anything in tree have to land in-tree on any of the release
> branches? robocop chunking just got bit by this:
> https://bugzilla.mozilla.org/show_bug.cgi?id=1142553#c11
>    - mozilla-release is still on pre-apk-split: "Android 2.3 Emulator
> mozilla-release opt test mochitest-gl-3" (from your builderlist). I doubt
> this will affect things but it's worth pointing out in case we have new
> tests that assume the split.
> 
> - also, judging by treeherder try link in first comment, this will need a
> treeherder patch too

Comment 7

[Kim Moir [:kmoir] ET]

Thanks for the reviews Jordan.

I don't think the mozilla-release opt test mochitest-gl-3 etc from the builderlist will affect things

I don't see how a treeherder patch is required for more chunking

Comment 8

[Jordan Lund (:jlund)]

> I don't see how a treeherder patch is required for more chunking

derp, I saw:  (gl1, gl2, gl2, gl2) and thought treeherder wasn't able for higher chunking splits but I'm guessing this was just a hack to split in four while buildbot only knows about two builders (chunk 1 and 2)

Comment 9

[Kim Moir [:kmoir] ET]

I need commit level 3 to land the changes for the m-h revisions on the associated branches, which I currently lack.  I'll open a bug for this.  Jordan, thanks for offering to land the mh changes for me.

Comment 10

[Jordan Lund (:jlund)]

rather than confuse the matter by landing a mh version bump on m-c/m-i, I'm going to let the trees re-open first.

Comment 11

[Ryan VanderMeulen [:RyanVM]]

Mozharness patch merged to production and bumped on m-c.

Comment 12

[Jordan Lund (:jlund)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/756f369e434a

Comment 13

[Jordan Lund (:jlund)]

Attachment: bump total chunk count to 4 for mochitest-gl

r+ from kmoir earlier today

landed directly on m-c: https://hg.mozilla.org/mozilla-central/rev/0b0a46d3ccf2

Comment 14

[Kim Moir [:kmoir] ET]

Verified on treeherder

Thanks Jordan, I didn't realize the chunking had to be set in tree and I have my level 3 commit rights now so I'm set for next time.

Comment 15

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/dc63b8a5854d
https://hg.mozilla.org/releases/mozilla-beta/rev/2d55d8220616