Frequent Windows build timeouts after "Leaving directory 'e:\builds\moz2_slave\m-in-w32\build\obj-firefox\modules\libjar\test'" (or, rarely, "\dom\workers")

RESOLVED WORKSFORME

Status

Firefox Build System
General
--
major
RESOLVED WORKSFORME
6 years ago
4 months ago

People

(Reporter: philor, Unassigned)

Tracking

({intermittent-failure})

Trunk
x86
Windows Server 2008
intermittent-failure
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [purple][capacity][fixed by bug 788954])

(Reporter)

Description

6 years ago
10800 seconds is a nasty long timeout, so we're losing Windows build capacity, with 60 minute jobs taking 180 minutes.

https://tbpl.mozilla.org/php/getParsedLog.php?id=14980802&tree=Firefox
WINNT 5.2 mozilla-central build on 2012-09-05 04:20:33 PDT for push e00f615159f1
slave: w64-ix-slave07
w64-ix-slave07
make.py[6]: Leaving directory 'e:\builds\moz2_slave\m-cen-w32\build\obj-firefox\dom\workers'
command timed out: 10800 seconds without output, attempting to kill

https://tbpl.mozilla.org/php/getParsedLog.php?id=14987823&tree=Mozilla-Inbound
slave: w64-ix-slave77
make.py[6]: Leaving directory 'e:\builds\moz2_slave\m-in-w32\build\obj-firefox\modules\libjar\test'
command timed out: 10800 seconds without output, attempting to kill

https://tbpl.mozilla.org/php/getParsedLog.php?id=14989826&tree=Mozilla-Inbound
w64-ix-slave17

https://tbpl.mozilla.org/php/getParsedLog.php?id=14992543&tree=Mozilla-Inbound
w64-ix-slave70

https://tbpl.mozilla.org/php/getParsedLog.php?id=14993214&tree=Mozilla-Inbound
w64-ix-slave77

https://tbpl.mozilla.org/php/getParsedLog.php?id=14995768&tree=Mozilla-Inbound
w64-ix-slave81

https://tbpl.mozilla.org/php/getParsedLog.php?id=14997901&tree=Mozilla-Inbound
w64-ix-slave70
Note the cause could very well partly be bug 788954, combined with a dead-lock in pymake.

Updated

6 years ago
Whiteboard: [capacity] → [orange][purple][capacity]

Updated

6 years ago
Blocks: 438871
Bug 788954 should have fixed this.
I've merged bug 788954 to mozilla-inbound.

Pymake shouldn't have stalled in the first place, though. That's being tracked in bug 788971.
Pretty sure this is fixed.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Whiteboard: [orange][purple][capacity] → [orange][purple][capacity][fixed by bug 788954]
The timeout is still too long for anything.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Mmm. So it's 10800 seconds right now, which is 3 hours. The longest taking thing we have by far is PGO-linking xul.dll, which typically takes 80-85 minutes. Should we make the timeout 2 hours long?
(Assignee)

Updated

6 years ago
Keywords: intermittent-failure
(Assignee)

Updated

6 years ago
Whiteboard: [orange][purple][capacity][fixed by bug 788954] → [purple][capacity][fixed by bug 788954]
Resolving WFM keyword:intermittent-failure bugs last modified >3 months ago, whose whiteboard contains none of:
{random,disabled,marked,fuzzy,todo,fails,failing,annotated,time-bomb,leave open}

There will inevitably be some false positives; for that (and the bugspam) I apologise. Filter on orangewfm.
Status: REOPENED → RESOLVED
Last Resolved: 6 years ago5 years ago
Resolution: --- → WORKSFORME

Updated

4 months ago
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.