1156421 - Split Linux mochitest-dt into chunks

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

It's been hitting the 120min timeouts for awhile now, and it's basically permafail at this point. We should split it into 4 chunks like we do for debug builds. Setting this to critical severity given that it affects the continued visibility of the job.

Comment 1

[Chris AtLee [:catlee]]

Related at all to the spike in run-times in bug 1156426?

Comment 2

[Ryan VanderMeulen [:RyanVM]]

Graphite says we've been steadily getting worse since February.

Comment 3

[Ryan VanderMeulen [:RyanVM]]

Which of course turns out to be way easier said than done if I'm understanding config.py correctly. Unless we change all opt m-dt runs to 4 chunks, that is.

Comment 4

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #3)
> Unless we change all opt m-dt runs to 4 chunks, that is.

Any reason not to? m-dt on linux takes an hour or more to finish.

Comment 5

[Ryan VanderMeulen [:RyanVM]]

Wasting time on OSX/Windows slaves.

Comment 6

[Ryan VanderMeulen [:RyanVM]]

Hitting 120min timeouts pretty regularly on other Linux opt runs now as well.

Comment 7

[Ryan VanderMeulen [:RyanVM]]

I split it into 2 chunks recently and it's still timing out in dt1. Conveniently, jgriffin has some code in config.py for special-casing linux32 debug that I should be able to shamelessly steal from at this point.

Comment 8

[Ryan VanderMeulen [:RyanVM]]

Attachment: split ASAN m-dt into 4 chunks

Code shamelessly stolen from jgriffin's example above it \m/

Builders added:
+ Ubuntu ASAN VM 12.04 x64 alder opt test mochitest-devtools-chrome-3
+ Ubuntu ASAN VM 12.04 x64 alder opt test mochitest-devtools-chrome-4
+ Ubuntu ASAN VM 12.04 x64 ash opt test mochitest-devtools-chrome-3
+ Ubuntu ASAN VM 12.04 x64 ash opt test mochitest-devtools-chrome-4
+ Ubuntu ASAN VM 12.04 x64 b2g-inbound opt test mochitest-devtools-chrome-3
+ Ubuntu ASAN VM 12.04 x64 b2g-inbound opt test mochitest-devtools-chrome-4
+ Ubuntu ASAN VM 12.04 x64 cedar opt test mochitest-devtools-chrome-3
+ Ubuntu ASAN VM 12.04 x64 cedar opt test mochitest-devtools-chrome-4
+ Ubuntu ASAN VM 12.04 x64 fx-team opt test mochitest-devtools-chrome-3
+ Ubuntu ASAN VM 12.04 x64 fx-team opt test mochitest-devtools-chrome-4
+ Ubuntu ASAN VM 12.04 x64 gum opt test mochitest-devtools-chrome-3
+ Ubuntu ASAN VM 12.04 x64 gum opt test mochitest-devtools-chrome-4
+ Ubuntu ASAN VM 12.04 x64 jamun opt test mochitest-devtools-chrome-3
+ Ubuntu ASAN VM 12.04 x64 jamun opt test mochitest-devtools-chrome-4
+ Ubuntu ASAN VM 12.04 x64 maple opt test mochitest-devtools-chrome-3
+ Ubuntu ASAN VM 12.04 x64 maple opt test mochitest-devtools-chrome-4
+ Ubuntu ASAN VM 12.04 x64 mozilla-aurora opt test mochitest-devtools-chrome-3
+ Ubuntu ASAN VM 12.04 x64 mozilla-aurora opt test mochitest-devtools-chrome-4
+ Ubuntu ASAN VM 12.04 x64 mozilla-beta opt test mochitest-devtools-chrome-3
+ Ubuntu ASAN VM 12.04 x64 mozilla-beta opt test mochitest-devtools-chrome-4
+ Ubuntu ASAN VM 12.04 x64 mozilla-central opt test mochitest-devtools-chrome-3
+ Ubuntu ASAN VM 12.04 x64 mozilla-central opt test mochitest-devtools-chrome-4
+ Ubuntu ASAN VM 12.04 x64 mozilla-inbound opt test mochitest-devtools-chrome-3
+ Ubuntu ASAN VM 12.04 x64 mozilla-inbound opt test mochitest-devtools-chrome-4
+ Ubuntu ASAN VM 12.04 x64 mozilla-release opt test mochitest-devtools-chrome-3
+ Ubuntu ASAN VM 12.04 x64 mozilla-release opt test mochitest-devtools-chrome-4
+ Ubuntu ASAN VM 12.04 x64 oak opt test mochitest-devtools-chrome-3
+ Ubuntu ASAN VM 12.04 x64 oak opt test mochitest-devtools-chrome-4
+ Ubuntu ASAN VM 12.04 x64 try opt test mochitest-devtools-chrome-3
+ Ubuntu ASAN VM 12.04 x64 try opt test mochitest-devtools-chrome-4

Comment 9

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 8648949 [details] [diff] [review]
split ASAN m-dt into 4 chunks

Thanks for the quick review!

https://hg.mozilla.org/build/buildbot-configs/rev/9205b834c3d7

Comment 10

[Kim Moir [:kmoir] ET]

In production