Closed Bug 1137757 Opened 9 years ago Closed 7 years ago

Intermittent Linux debug mochitest-dt1 command timed out: 4800 seconds elapsed running ['/tools/buildbot/bin/python', 'scripts/scripts/desktop_unittest.py', '--cfg', 'unittests/linux_unittest.py', '--mochitest-suite', 'mochitest-devtools-chrome-chunked',

Categories

(Release Engineering :: General, defect)

x86
Linux
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: RyanVM, Unassigned)

References

(Depends on 1 open bug)

Details

(Keywords: intermittent-failure)

Attachments

(1 file)

We've been bumping into this pretty regularly lately. I think it's time to bite the bullet and up the chunks from 3 to 4.
Attachment #8570583 - Flags: review?(catlee)
Attachment #8570583 - Flags: review?(catlee) → review+
Unfortunately, the chunking change barely made a dent in linux debug m-dt1 runtime AFAICT. Only reduced the number of jobs from just over 16000 to a bit under 15000. And the debugger tests continue to kill us within it. I'd push for 5 chunks, but who knows how much it'd actually help. It seems like this problem is really boiling down to the bigger issue of having saner ways to balance the chunk runtimes.
Depends on: 1162622
Hopefully bug 1162622 is going to fix a regression, but there may be also something to do with our tests.
I looked at execution time and there is this one
browser/devtools/debugger/test/browser_dbg_breakpoints-break-on-last-line-of-script-on-reload.js | took 44595ms

I think it is already reported as bug 981258.

But there is a bunch of other tests that are quite slow:
browser/devtools/debugger/test/browser_dbg_variables-view-edit-getset-01.js | took 37747ms
browser/devtools/debugger/test/browser_dbg_host-layout.js | took 33857ms (bug 936379)
browser/devtools/debugger/test/browser_dbg_variables-view-filter-02.js | took 32897ms
browser/devtools/commandline/test/browser_cmd_highlight_01.js | took 31706ms (bug 1091669)
browser/devtools/debugger/test/browser_dbg_variables-view-filter-01.js | took 30703ms
Ah. No. Bug 981258 was for something else than just a timeout. 
browser_dbg_breakpoints-break-on-last-line-of-script-on-reload.js already requests a longer timeout.
(In reply to Alexandre Poirot [:ochameau] from comment #424)
> Ah. No. Bug 981258 was for something else than just a timeout. 
> browser_dbg_breakpoints-break-on-last-line-of-script-on-reload.js already
> requests a longer timeout.

The problem here isn't any particular test; it's just that the total time of the testrun exceeds 4800s, which is the max time allowed for it by its buildbot config.

Optimally, we could increase the number of chunks here, but we can't in this case (at least for now) because buildbot doesn't have enough available builders.  We could also increase the timeout, but that's not great either, as that increases the total E2E time per push.

So we've been trying to figure out why the tests are slower than they used to be.  Bug 1162622 is one regression, but looking at the history of this bug there may be multiple regressions involved.
Blocks: 1167984
Linux32 debug dt1 and dt2 hidden on trunk and aurora, so it'll get a lot quieter in here.
On fx-team, dt2 is now hovering around 77-80 minutes on linux64 dbg when it succeeds, and the limit is 80 minutes. Time to increase the number of chunks again?

(I was worried this was caused by browser_parsable_css.js, which I just touched, but that seems to be in chunk 4 which is at a healthy 44 minutes on the same machines, so I don't think that's it)
Flags: needinfo?(ryanvm)
You're in luck, bug 1203227 is aiming to do just that!
Flags: needinfo?(ryanvm)
So, looking at https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1137757&startday=2015-09-01&endday=2015-10-13&tree=all ... Most of the recent hits are Linux64 debug or Linux32 opt.

As far as I can tell, only 32-bit Linux debug has 8 dt chunks, Linux64 asan and debug have four, and everything else has 2.

Bug 1203227 has patches to bump everything to 7 chunks, but it's stalled until we can free up enough builders.

Any chance we could at least make Linux64 debug use 8 chunks like it's 32-bit counterpart?


Alternatively, the script_maxtime's in https://dxr.mozilla.org/build_buildbot-configs/source/mozilla-tests/config.py#459 were reduced from 7200s to 4800s a few months ago to help act as a canary in the coalmine for runtime regressions like this, but it's pretty clear by now that the canary's been dead for a month. Could we at least bump them back up to 7200s if scraping together enough free builders is going to take too long?
Flags: needinfo?(jgriffin)
Yes, we can bump linux64-debug to 8 chunks; I've filed bug 1214853 to do that.
Flags: needinfo?(jgriffin)
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: