788917 - Frequent Windows build timeouts after "Leaving directory 'e:\builds\moz2_slave\m-in-w32\build\obj-firefox\modules\libjar\test'" (or, rarely, "\dom\workers")

Status: RESOLVED WORKSFORME

(Firefox Build System :: General, defect)

Description

[Phil Ringnalda (:philor)]

10800 seconds is a nasty long timeout, so we're losing Windows build capacity, with 60 minute jobs taking 180 minutes.

https://tbpl.mozilla.org/php/getParsedLog.php?id=14980802&tree=Firefox
WINNT 5.2 mozilla-central build on 2012-09-05 04:20:33 PDT for push e00f615159f1
slave: w64-ix-slave07
w64-ix-slave07
make.py[6]: Leaving directory 'e:\builds\moz2_slave\m-cen-w32\build\obj-firefox\dom\workers'
command timed out: 10800 seconds without output, attempting to kill

https://tbpl.mozilla.org/php/getParsedLog.php?id=14987823&tree=Mozilla-Inbound
slave: w64-ix-slave77
make.py[6]: Leaving directory 'e:\builds\moz2_slave\m-in-w32\build\obj-firefox\modules\libjar\test'
command timed out: 10800 seconds without output, attempting to kill

https://tbpl.mozilla.org/php/getParsedLog.php?id=14989826&tree=Mozilla-Inbound
w64-ix-slave17

https://tbpl.mozilla.org/php/getParsedLog.php?id=14992543&tree=Mozilla-Inbound
w64-ix-slave70

https://tbpl.mozilla.org/php/getParsedLog.php?id=14993214&tree=Mozilla-Inbound
w64-ix-slave77

https://tbpl.mozilla.org/php/getParsedLog.php?id=14995768&tree=Mozilla-Inbound
w64-ix-slave81

https://tbpl.mozilla.org/php/getParsedLog.php?id=14997901&tree=Mozilla-Inbound
w64-ix-slave70

Comment 1

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=15000508&tree=Mozilla-Inbound
w64-ix-slave77

Comment 2

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=15004521&tree=Mozilla-Inbound
w64-ix-slave10

Comment 3

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=15008505&tree=Firefox
w64-ix-slave83

Comment 4

[Mike Hommey [:glandium]]

Note the cause could very well partly be bug 788954, combined with a dead-lock in pymake.

Comment 5

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=15008457&tree=Mozilla-Inbound

Comment 6

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=15006178&tree=Mozilla-Inbound

Comment 7

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=15011523&tree=Firefox
w64-ix-slave08

Comment 8

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=15016307&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=15015919&tree=Mozilla-Inbound

Comment 9

[Rain [:rain1] (inactive)]

Bug 788954 should have fixed this.

Comment 10

[Rain [:rain1] (inactive)]

I've merged bug 788954 to mozilla-inbound.

Pymake shouldn't have stalled in the first place, though. That's being tracked in bug 788971.

Comment 11

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=15018219&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=15018331&tree=Mozilla-Inbound

Comment 12

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=15018952&tree=Mozilla-Inbound

Comment 13

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=15021457&tree=Mozilla-Inbound

(still below the merge)

Comment 14

[Rain [:rain1] (inactive)]

Pretty sure this is fixed.

Comment 15

[Mike Hommey [:glandium]]

The timeout is still too long for anything.

Comment 16

[Rain [:rain1] (inactive)]

Mmm. So it's 10800 seconds right now, which is 3 hours. The longest taking thing we have by far is PGO-linking xul.dll, which typically takes 80-85 minutes. Should we make the timeout 2 hours long?

Comment 17

[Ed Morley [:emorley]]

Resolving WFM keyword:intermittent-failure bugs last modified >3 months ago, whose whiteboard contains none of:
{random,disabled,marked,fuzzy,todo,fails,failing,annotated,time-bomb,leave open}

There will inevitably be some false positives; for that (and the bugspam) I apologise. Filter on orangewfm.