840461 - ix-mn-w864-002 (and others) unable to reboot

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Carsten Book [:Tomcat]]

seems the reboot script fails on the windows 8 test maschines

  TMP=C:\Users\cltbld\AppData\Local\Temp
  USERDOMAIN=IX-MN-W864-002
  USERDOMAIN_ROAMINGPROFILE=IX-MN-W864-002
  USERNAME=cltbld
  USERPROFILE=C:\Users\cltbld
  WINDIR=C:\windows
 using PTY: False
The system cannot find the path specified.
program finished with exit code 1
elapsedTime=0.100000

from some win8 docs it seems that shutdown /r /o /f /t 00 should work from any place - however woth investigating why the slaves not rebooting

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Can you paste the whole step log?

Comment 2

[Carsten Book [:Tomcat]]

'c:/mozilla-build/python27/python' '-u' 'scripts/external_tools/count_and_reboot.py' '-f' '../reboot_count.txt' '-n' '1' '-z'
 in dir C:\slave\test\. (timeout 1200 secs)
 watching logfiles {}
 argv: ['c:/mozilla-build/python27/python', '-u', 'scripts/external_tools/count_and_reboot.py', '-f', '../reboot_count.txt', '-n', '1', '-z']
 environment:
  ALLUSERSPROFILE=C:\ProgramData
  APPDATA=C:\Users\cltbld\AppData\Roaming
  COMMONPROGRAMFILES=C:\Program Files (x86)\Common Files
  COMMONPROGRAMFILES(X86)=C:\Program Files (x86)\Common Files
  COMMONPROGRAMW6432=C:\Program Files\Common Files
  COMPUTERNAME=IX-MN-W864-002
  COMSPEC=C:\windows\system32\cmd.exe
  FP_NO_HOST_CHECK=NO
  HOMEDRIVE=C:
  HOMEPATH=\Users\cltbld
  LOCALAPPDATA=C:\Users\cltbld\AppData\Local
  LOGONSERVER=\\IX-MN-W864-002
  NUMBER_OF_PROCESSORS=8
  OS=Windows_NT
  PATH=C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0\;C:\mozilla-build\python;C:\mozilla-build\python\Scripts;C:\mozilla-build\msys\bin;C:\mozilla-build\vim\vim72;C:\mozilla-build\wget;C:\mozilla-build\info-zip;C:\CoreUtils\bin;C:\mozilla-build\buildbotve\scripts
  PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
  PROCESSOR_ARCHITECTURE=x86
  PROCESSOR_ARCHITEW6432=AMD64
  PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 30 Stepping 5, GenuineIntel
  PROCESSOR_LEVEL=6
  PROCESSOR_REVISION=1e05
  PROGRAMDATA=C:\ProgramData
  PROGRAMFILES=C:\Program Files (x86)
  PROGRAMFILES(X86)=C:\Program Files (x86)
  PROGRAMW6432=C:\Program Files
  PROMPT=$P$G
  PROPERTIES_FILE=C:\slave\test/buildprops.json
  PSMODULEPATH=C:\windows\system32\WindowsPowerShell\v1.0\Modules\
  PUBLIC=C:\Users\Public
  PWD=C:\slave\test
  SYSTEMDRIVE=C:
  SYSTEMROOT=C:\windows
  TEMP=C:\Users\cltbld\AppData\Local\Temp
  TMP=C:\Users\cltbld\AppData\Local\Temp
  USERDOMAIN=IX-MN-W864-002
  USERDOMAIN_ROAMINGPROFILE=IX-MN-W864-002
  USERNAME=cltbld
  USERPROFILE=C:\Users\cltbld
  WINDIR=C:\windows
 using PTY: False
The system cannot find the path specified.
program finished with exit code 1
elapsedTime=0.100000

Comment 3

[Carsten Book [:Tomcat]]

script is http://hg.mozilla.org/users/armenzg_mozilla.com/mozharness/file/7a2d0c165b2e/external_tools/count_and_reboot.py

maybe (just as idea that came into my mind) 

    1.26 +elif sys.platform == "win32":
    1.27 +    # Windows
    1.28 +    def reboot():
    1.29 +        os.system("shutdown -f -r -t 0")
    1.30 +	# After starting the shutdown, we go to sleep since the system can
    1.31 +	# take a few minutes to shut everything down and reboot
    1.32 +	time.sleep(600)
    1.33 +

need to be updated to platform 64bit also

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

I verified that sys.platform == "win32" on those systems, so that's not it.

It's hard to tell which "path specified" the system cannot find.  Is it "shutdown"?  I don't expect so, since that's right in C:\windows\system32, which is the second element in %PATH%.  Is it count_and_reboot.py?  Or python?

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

if you run 'python foo.py', the error message says
 python: can't open file 'foo.py': [Errno 2] No such file or directory
so that rules out being unable to find count_and_reboot.py.

Comment 6

[Armen [:armenzg]]

It should have been python\python rather than python27\python

c:/mozilla-build/python27/python -u scripts/external_tools/count_and_reboot.py -f ../reboot_count.txt -n 1 -z

c:/mozilla-build/python/python -u scripts/external_tools/count_and_reboot.py -f ../reboot_count.txt -n 1 -z

I reconfigured the master to fix it but after the new IT additions we will have python under python27.

Comment 7

[Carsten Book [:Tomcat]]

hm what this really fixed, seems the systems still have problems to reboot

http://dev-master01.build.scl1.mozilla.com:8042/builders/Rev3%20WINNT%206.2%20cedar%20opt%20test%20mochitest-other/builds/65/steps/reboot/logs/stdio

'c:/mozilla-build/python27/python' '-u' 'scripts/external_tools/count_and_reboot.py' '-f' '../reboot_count.txt' '-n' '1' '-z'
 in dir C:\slave\test\. (timeout 1200 secs)
 watching logfiles {}
 argv: ['c:/mozilla-build/python27/python', '-u', 'scripts/external_tools/count_and_reboot.py', '-f', '../reboot_count.txt', '-n', '1', '-z']
 environment:
  ALLUSERSPROFILE=C:\ProgramData
  APPDATA=C:\Users\Administrator\AppData\Roaming
  COMMONPROGRAMFILES=C:\Program Files (x86)\Common Files
  COMMONPROGRAMFILES(X86)=C:\Program Files (x86)\Common Files
  COMMONPROGRAMW6432=C:\Program Files\Common Files
  COMPUTERNAME=IX-MN-W864-002
  COMSPEC=C:\windows\system32\cmd.exe
  FP_NO_HOST_CHECK=NO
  LOCALAPPDATA=C:\Users\Administrator\AppData\Local
  NUMBER_OF_PROCESSORS=8
  OS=Windows_NT
  PATH=C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0\;C:\mozilla-build\python;C:\mozilla-build\python\Scripts;C:\mozilla-build\msys\bin;C:\mozilla-build\vim\vim72;C:\mozilla-build\wget;C:\mozilla-build\info-zip;C:\CoreUtils\bin;C:\mozilla-build\buildbotve\scripts
  PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
  PROCESSOR_ARCHITECTURE=x86
  PROCESSOR_ARCHITEW6432=AMD64
  PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 30 Stepping 5, GenuineIntel
  PROCESSOR_LEVEL=6
  PROCESSOR_REVISION=1e05
  PROGRAMDATA=C:\ProgramData
  PROGRAMFILES=C:\Program Files (x86)
  PROGRAMFILES(X86)=C:\Program Files (x86)
  PROGRAMW6432=C:\Program Files
  PROMPT=$P$G
  PROPERTIES_FILE=C:\slave\test/buildprops.json
  PSMODULEPATH=C:\windows\system32\WindowsPowerShell\v1.0\Modules\
  PUBLIC=C:\Users\Public
  PWD=C:\slave\test
  SYSTEMDRIVE=C:
  SYSTEMROOT=C:\windows
  TEMP=C:\Users\ADMINI~1\AppData\Local\Temp
  TMP=C:\Users\ADMINI~1\AppData\Local\Temp
  USERDOMAIN=IX-MN-W864-002
  USERNAME=Administrator
  USERPROFILE=C:\Users\Administrator
  WINDIR=C:\windows
 using PTY: False
The system cannot find the path specified.
program finished with exit code 1
elapsedTime=0.101000

Comment 8

[Carsten Book [:Tomcat]]

verified fixed