394841 - win32 buildbot/tryserver stuck in flame land

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Ed Lee :Mardak]

It all started with

http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1188868537.1188872331.14258.gz

Failure: buildbot.slave.commands.TimeoutError: SIGKILL failed to kill process


then
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1188873125.1188874761.21602.gz

mozilla\build - The directory is not empty.
mozilla\objdir\xpcom\glue - The process cannot access the file because it is being used by another process.
mozilla\objdir\xpcom - The process cannot access the file because it is being used by another process.
mozilla\objdir - The process cannot access the file because it is being used by another process.
The process cannot access the file because it is being used by another process.
cvs checkout: warning: failed to open /home/cltbld/.cvspass for reading: No such file or directory
U mozilla/client.mk
cvs checkout: warning: failed to open /home/cltbld/.cvspass for reading: No such file or directory
cvs [checkout aborted]: could not chdir to mozilla/build/autoconf: Permission denied
/bin/sh: mozilla/build/autoconf/mozconfig2client-mk: No such file or directory
client.mk:521: /cygdrive/d/buildbot/sendchange-slave/sendchange-win32/mozilla/.mozconfig.mk: No such file or directory
make: *** No rule to make target `/cygdrive/d/buildbot/sendchange-slave/sendchange-win32/mozilla/.mozconfig.mk'.  Stop.


and again

http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1188884164.1188885696.22756.gz

Comment 1

[Chris Cooper [:coop] (he/him)]

Slave was hung (unkillable without killing the shell or using taskmgr). Keyboard input is screwy too, so I'm going to reboot the VM and manually clobber the build dir before starting it back up.

Comment 2

[Chris Cooper [:coop] (he/him)]

Slave is back up and clobbered. Please reopen if it still refuses to build.

Comment 3

[Ed Lee :Mardak]

It's no longer burning.. it's just nowhere to be found ;)
http://tinderbox.mozilla.org/showbuilds.cgi?tree=MozillaTry

I submitted the patch ~30 minutes ago and waited a bit hoping that it was just slow to start, but still nothing.

Comment 4

[Chris Cooper [:coop] (he/him)]

When restarting last time, I re-ran the same command as whoever had started the slave previously which, as it turns out, was wrong. 

You can start buildbot with 'buildbot start .' from the actual slave dir directory and it won't complain, but it doesn't look like that actually works. Backing up a dir and running 'buildbot start sendchange-slave' seems to work fine.

Comment 5

[Ed Lee :Mardak]

It's down again, but the build I submitted went through okay. vlad's came soon after mine but that resulted in the same error as before.

Failure: buildbot.slave.commands.TimeoutError: SIGKILL failed to kill process
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1188941534.1188946655.30641.gz

Comment 6

[Ed Lee :Mardak]

Down again with SIGKILL issue.

There was a separate failure for something else earlier though..
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1189305564.1189311320.5675.gz
Generating PQG Params: An I/O error occurred during security authorization.
make[5]: *** [d:/buildbot/sendchange-slave/sendchange-win32/mozilla/objdir/dist/lib/softokn3.chk] Error 1

Comment 7

[Ed Lee :Mardak]

-> blocker, we're trying to get testing on windows for download resume

Comment 8

[Ed Lee :Mardak]

sev -> normal, not getting it in for M8

Comment 9

[Chris Cooper [:coop] (he/him)]

Unstuck. Again.