1353016 - linux debug e10s M-bc jobs run for too long

Status: RESOLVED FIXED

(Testing :: General, enhancement)

Description

[Geoff Brown [:gbrown]]

All jobs in the series run > 60 minutes usually, with some running > 90 minutes. We're seeing a few 90 minute timeouts in bug 1204281.

Linux 32 seems more problematic, but Linux 64 seems to have similar job run times.

Comment 1

[Geoff Brown [:gbrown]]

Attachment: more chunks for linux mochitest-bc

https://treeherder.mozilla.org/#/jobs?repo=try&revision=da9f55d10f1dbb82b0196665e951b71972bcfa23

Comment 2

[(not currently active) Ted Mielczarek]

Do you have any idea if there are particular tests that take a long time, or is this just all the tests being slow enough in this configuration to push it over the limit?

Comment 3

[Geoff Brown [:gbrown]]

I don't have a clear idea. Reporting the slowest test is handy for that sort of query...but we don't seem to report that for mochitest-bc.

I don't see individual test timeouts in the jobs that timed out, and I don't notice huge gaps in timestamps when I scroll through the logs.

Consider one of our recent failures, https://treeherder.mozilla.org/logviewer.html#?job_id=87994648&repo=autoland, a 90 minute timeout in linux32 debug mochitest-bc-e10s-5. The previous run of that job was https://treeherder.mozilla.org/logviewer.html#?job_id=87988277&repo=autoland, which took 89 minutes. Before that was https://treeherder.mozilla.org/logviewer.html#?job_id=87961805&repo=autoland, 90 minutes, but green.

There is considerable variability in the run time of the linux debug mochitest-bc jobs, but almost all of them are usually 60+ minutes and 80+ minute runs are not uncommon.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/edfb3b0fc079
Add more test chunks for linux debug mochitest-bc; r=jmaher

Comment 5

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/edfb3b0fc079

Comment 6

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/8bf7e4280c94

Comment 7

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/fc66dce2192b

Comment 8

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr52/rev/154f1bb4b731