813949 - Windows builds timing out after "processing firefox\firefox.exe on https://signing1.build.scl1.mozilla.com:9110"

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Ed Morley [:emorley]]

eg:
{
Compressing...
e:\builds\moz2_slave\m-in-w32\build\toolkit\mozapps\installer\packager.mk:866:0$ cd ../../dist && python e:/builds/moz2_slave/m-in-w32/tools/release/signing/signtool.py --cachedir e:/builds/moz2_slave/m-in-w32/signing_cache -t e:/builds/moz2_slave/m-in-w32/token -n e:/builds/moz2_slave/m-in-w32/nonce -c e:/builds/moz2_slave/m-in-w32/tools/release/signing/host.cert -H signing1.build.scl1.mozilla.com:9110 -H signing2.build.scl1.mozilla.com:9110 -f signcode -i '*.dll' -i '*.exe' -x 'D3DCompiler*.dll' -x 'd3dx9*.dll' -x 'msvc*.dll' --nsscmd "e:/builds/moz2_slave/m-in-w32/build/obj-firefox/browser/installer/../../dist/bin/shlibsign.exe -v -i" firefox && zip -r9D firefox-20.0a1.en-US.win32.zip firefox -x \*/.mkdir.done
2012-11-21 02:59:34,433 - 837c3eaebd047dd93afff1fc2ca6132bc50b5d07: exists in the cache; copying to firefox\AccessibleMarshal.dll
2012-11-21 02:59:34,451 - 837c3eaebd047dd93afff1fc2ca6132bc50b5d07: OK
2012-11-21 02:59:34,516 - a24baee4189ba04b7b9eb7cf270ea764a515e4ce: exists in the cache; copying to firefox\breakpadinjector.dll
2012-11-21 02:59:34,760 - a24baee4189ba04b7b9eb7cf270ea764a515e4ce: OK
2012-11-21 02:59:34,948 - 6401dfb5ad9ff9e00a793dbe06169716628be0f8: exists in the cache; copying to firefox\crashreporter.exe
2012-11-21 02:59:34,982 - 6401dfb5ad9ff9e00a793dbe06169716628be0f8: OK
2012-11-21 02:59:35,085 - 34e1232b852cb4116e9c706b41e41142aefb69fb: processing firefox\firefox.exe on https://signing1.build.scl1.mozilla.com:9110

command timed out: 1200 seconds without output, attempting to kill
SIGKILL failed to kill process
using fake rc=-1
program finished with exit code -1

remoteFailed: [Failure instance: Traceback from remote host -- Traceback (most recent call last):
}

https://tbpl.mozilla.org/php/getParsedLog.php?id=17235579&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=17235691&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=17235694&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=17237680&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=17237468&tree=Mozilla-Inbound

We seem to have had one green build since:
https://tbpl.mozilla.org/php/getParsedLog.php?id=17238511&tree=Mozilla-Inbound

...however I'm not sure if the other retriggers will come back green, and even if they do, it would be good to handle this failure mode properly (ie: if it was just a network issue with https://signing1.build.scl1.mozilla.com, then we should have a proper timeout message).

Comment 1

[[e-mail] (:bhearsum)]

The dep instance on signing1 was hung. I stopped it which should stop the bleeding (builds should retry and eventually end up on signing2). I'm having trouble getting it restarted. We should be fine for awhile, I think signing2 can handle the load right now.

Comment 2

[Ed Morley [:emorley]]

Thank you :-)

Comment 3

[[e-mail] (:bhearsum)]

I had to kill these even harder, there were some hung process which causes more jobs to fail until about 30min ago :(. Hope to have the signing1 dep instance restarted in the next couple of hours.

Comment 4

[[e-mail] (:bhearsum)]

Finally managed to get the instance restarted.