Closed Bug 1142743 Opened 5 years ago Closed 5 years ago

Increase chunk count for Android 2.3 mochitest-gl jobs

Categories

(Infrastructure & Operations :: CIDuty, task)

x86_64
Windows 8.1
task
Not set

Tracking

(firefox37 fixed, firefox38 fixed, firefox39 fixed)

RESOLVED FIXED
Tracking Status
firefox37 --- fixed
firefox38 --- fixed
firefox39 --- fixed

People

(Reporter: jgilbert, Assigned: kmoir)

References

Details

Attachments

(4 files)

I'm adding a bunch of tests, just about doubling the number of test files, and tripling the number of test assertions.

mochitest-gl2 is currently running 60-70 minutes so far, but we hit a couple tests that time out and kill the run. (4 timeouts = abort?)[1]
I tried skipping those tests, but it seems like tests almost immediately after that are timing out too.[2]
I'm disabling those and trying again, but I'm concerned we're at the machine's limit.

Further, it looks like there are plenty of tests (over 280 test files) still to run.[3] We probably don't want tests that take this long to run anyways.

I propose we double our chunk count to 4 and see where that gets us.

[1]: https://treeherder.mozilla.org/#/jobs?repo=try&revision=cff3f51fb695
[2]: https://treeherder.mozilla.org/#/jobs?repo=try&revision=c96128a4b34b
[3]:
> 548 INFO TEST-UNEXPECTED-FAIL | (SimpleTest/TestRunner.js) | 4 test timeouts, giving up. - expected PASS
> 549 INFO TEST-UNEXPECTED-FAIL | (SimpleTest/TestRunner.js) | Skipping 285 remaining tests. - expected PASS
Assignee: nobody → kmoir
Attachment #8577301 - Flags: review?(jlund)
Attachment #8577303 - Flags: review?(jlund)
Attached file bug1142743builder.diff
builder diff
Comment on attachment 8577303 [details] [diff] [review]
bug1142743mh.patch

Review of attachment 8577303 [details] [diff] [review]:
-----------------------------------------------------------------

shweet, should achieve what we want. couple sanity checks on the rollout:

- is orange expected here for all for chunks: https://treeherder.mozilla.org/#/jobs?repo=try&revision=cff3f51fb695 ?

- should we be riding trains on this instead of changing across all our builders? e.g.
   - does anything in tree have to land in-tree on any of the release branches? robocop chunking just got bit by this: https://bugzilla.mozilla.org/show_bug.cgi?id=1142553#c11
   - mozilla-release is still on pre-apk-split: "Android 2.3 Emulator mozilla-release opt test mochitest-gl-3" (from your builderlist). I doubt this will affect things but it's worth pointing out in case we have new tests that assume the split.

- also, judging by treeherder try link in first comment, this will need a treeherder patch too
Attachment #8577303 - Flags: review?(jlund) → review+
Comment on attachment 8577301 [details] [diff] [review]
bug1142743bbconfigs.patch

Review of attachment 8577301 [details] [diff] [review]:
-----------------------------------------------------------------

see https://bugzilla.mozilla.org/show_bug.cgi?id=1142743#c4 for sanity check questions on the pair of these patches.

also, just thinking, the mh patch will need to have a mozharness.json pin bump across all branches
Attachment #8577301 - Flags: review?(jlund) → review+
(In reply to Jordan Lund (:jlund) from comment #4)
> Comment on attachment 8577303 [details] [diff] [review]
> bug1142743mh.patch
> 
> Review of attachment 8577303 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> shweet, should achieve what we want. couple sanity checks on the rollout:
> 
> - is orange expected here for all for chunks:
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=cff3f51fb695 ?
Yeah, there's hundreds of new test files and our infra drivers are pretty bad. A number of test failures, crashes, and timeouts still need to be marked.

> - should we be riding trains on this instead of changing across all our
> builders? e.g.
It might help with existing intermittents on the platform, but otherwise we've been OK without this bug's increase so far. (We just need a bigger split moving forward)
>    - does anything in tree have to land in-tree on any of the release
> branches? robocop chunking just got bit by this:
> https://bugzilla.mozilla.org/show_bug.cgi?id=1142553#c11
>    - mozilla-release is still on pre-apk-split: "Android 2.3 Emulator
> mozilla-release opt test mochitest-gl-3" (from your builderlist). I doubt
> this will affect things but it's worth pointing out in case we have new
> tests that assume the split.
> 
> - also, judging by treeherder try link in first comment, this will need a
> treeherder patch too
Thanks for the reviews Jordan.

I don't think the mozilla-release opt test mochitest-gl-3 etc from the builderlist will affect things

I don't see how a treeherder patch is required for more chunking
> I don't see how a treeherder patch is required for more chunking

derp, I saw:  (gl1, gl2, gl2, gl2) and thought treeherder wasn't able for higher chunking splits but I'm guessing this was just a hack to split in four while buildbot only knows about two builders (chunk 1 and 2)
Attachment #8577303 - Flags: checked-in+
Attachment #8577301 - Flags: checked-in+
I need commit level 3 to land the changes for the m-h revisions on the associated branches, which I currently lack.  I'll open a bug for this.  Jordan, thanks for offering to land the mh changes for me.
rather than confuse the matter by landing a mh version bump on m-c/m-i, I'm going to let the trees re-open first.
Mozharness patch merged to production and bumped on m-c.
r+ from kmoir earlier today

landed directly on m-c: https://hg.mozilla.org/mozilla-central/rev/0b0a46d3ccf2
Attachment #8579068 - Flags: review+
Attachment #8579068 - Flags: checked-in+
Verified on treeherder

Thanks Jordan, I didn't realize the chunking had to be set in tree and I have my level 3 commit rights now so I'm set for next time.
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.