Closed Bug 788917 Opened 12 years ago Closed 11 years ago

Frequent Windows build timeouts after "Leaving directory 'e:\builds\moz2_slave\m-in-w32\build\obj-firefox\modules\libjar\test'" (or, rarely, "\dom\workers")

Categories

(Firefox Build System :: General, defect)

Product:
Firefox Build System
Component:
General
Platform:
x86
Windows Server 2008
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: philor, Unassigned)

References

Details

(Keywords: intermittent-failure, Whiteboard: [purple][capacity][fixed by bug 788954]) 

10800 seconds is a nasty long timeout, so we're losing Windows build capacity, with 60 minute jobs taking 180 minutes.

https://tbpl.mozilla.org/php/getParsedLog.php?id=14980802&tree=Firefox
WINNT 5.2 mozilla-central build on 2012-09-05 04:20:33 PDT for push e00f615159f1
slave: w64-ix-slave07
w64-ix-slave07
make.py[6]: Leaving directory 'e:\builds\moz2_slave\m-cen-w32\build\obj-firefox\dom\workers'
command timed out: 10800 seconds without output, attempting to kill

https://tbpl.mozilla.org/php/getParsedLog.php?id=14987823&tree=Mozilla-Inbound
slave: w64-ix-slave77
make.py[6]: Leaving directory 'e:\builds\moz2_slave\m-in-w32\build\obj-firefox\modules\libjar\test'
command timed out: 10800 seconds without output, attempting to kill

https://tbpl.mozilla.org/php/getParsedLog.php?id=14989826&tree=Mozilla-Inbound
w64-ix-slave17

https://tbpl.mozilla.org/php/getParsedLog.php?id=14992543&tree=Mozilla-Inbound
w64-ix-slave70

https://tbpl.mozilla.org/php/getParsedLog.php?id=14993214&tree=Mozilla-Inbound
w64-ix-slave77

https://tbpl.mozilla.org/php/getParsedLog.php?id=14995768&tree=Mozilla-Inbound
w64-ix-slave81

https://tbpl.mozilla.org/php/getParsedLog.php?id=14997901&tree=Mozilla-Inbound
w64-ix-slave70
Note the cause could very well partly be bug 788954, combined with a dead-lock in pymake.
Whiteboard: [capacity] → [orange][purple][capacity]
Blocks: 438871
Bug 788954 should have fixed this.
Depends on: 788954
I've merged bug 788954 to mozilla-inbound.

Pymake shouldn't have stalled in the first place, though. That's being tracked in bug 788971.
Pretty sure this is fixed.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Whiteboard: [orange][purple][capacity] → [orange][purple][capacity][fixed by bug 788954]
The timeout is still too long for anything.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Mmm. So it's 10800 seconds right now, which is 3 hours. The longest taking thing we have by far is PGO-linking xul.dll, which typically takes 80-85 minutes. Should we make the timeout 2 hours long?
Whiteboard: [orange][purple][capacity][fixed by bug 788954] → [purple][capacity][fixed by bug 788954]
Resolving WFM keyword:intermittent-failure bugs last modified >3 months ago, whose whiteboard contains none of:
{random,disabled,marked,fuzzy,todo,fails,failing,annotated,time-bomb,leave open}

There will inevitably be some false positives; for that (and the bugspam) I apologise. Filter on orangewfm.
Status: REOPENED → RESOLVED
Closed: 12 years ago → 11 years ago
Resolution: --- → WORKSFORME
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.