1304092 - Win XP xpcshell/mozmill failure: WindowsError: [Error 2] The system cannot find the file specified

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[aleth [:aleth]]

Last good Wed Sep 14, 13:10:34 mc 501e27643
First bad Wed Sep 14, 18:52:12 mc 8a494adbc

========= Started 'bash -c ...' exception (results: 4, elapsed: 0 secs) (at 2016-09-19 22:10:23.446342) =========
 'bash' '-c' 'python archiver_client.py mozharness --repo mozilla-central --rev default --destination scripts --debug'
  in dir C:\slave\test\. (timeout 1200 secs)
  watching logfiles {}
  argv: ['bash', '-c', 'python archiver_client.py mozharness --repo mozilla-central --rev default --destination scripts --debug']
  environment:
   ALLUSERSPROFILE=C:\Documents and Settings\All Users
   APPDATA=C:\Documents and Settings\cltbld.T-XP32-IX-125\Application Data
   COMMONPROGRAMFILES=C:\Program Files\Common Files
   COMPUTERNAME=T-XP32-IX-125
   COMSPEC=C:\WINDOWS\system32\cmd.exe
   DCLOCATION=SCL3
   DNSSUFFIX=wintest.releng.scl3.mozilla.com
   FP_NO_HOST_CHECK=NO
   HOMEDRIVE=C:
   HOMEPATH=\Documents and Settings\cltbld.T-XP32-IX-125
   KTS_HOME=C:\Program Files\KTS
   KTS_VERSION=1.19c
   LOGONSERVER=\\T-XP32-IX-125
   MONDIR=C:\Monitor_config\
   MOZBUILDDIR=C:\mozilla-build\
   NUMBER_OF_PROCESSORS=8
   OS=Windows_NT
   OURDRIVE=C:
   PATH=C:\Program Files\NVIDIA Corporation\PhysX\Common;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\system32\WindowsPowerShell\v1.0;C:\mozilla-build\python27;C:\mozilla-build\python27\Scripts;C:\mozilla-build\msys\bin;C:\mozilla-build\vim\vim72;C:\mozilla-build\wget;C:\mozilla-build\info-zip;C:\CoreUtils\bin;C:\mozilla-build\buildbotve\scripts
   PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.PSC1
   PROCESSOR_ARCHITECTURE=x86
   PROCESSOR_IDENTIFIER=x86 Family 6 Model 30 Stepping 5, GenuineIntel
   PROCESSOR_LEVEL=6
   PROCESSOR_REVISION=1e05
   PROGRAMFILES=C:\Program Files
   PROMPT=$P$G
   PWD=C:\slave\test
   RUNLOGFILE=C:\slave\\runslave.log
   SESSIONNAME=Console
   SLAVEDIR=C:\slave\
   SYSTEMDRIVE=C:
   SYSTEMROOT=C:\WINDOWS
   TEMP=C:\DOCUME~1\CLTBLD~1.T-X\LOCALS~1\Temp
   TEST1=testie
   TMP=C:\DOCUME~1\CLTBLD~1.T-X\LOCALS~1\Temp
   USERDOMAIN=T-XP32-IX-125
   USERNAME=cltbld
   USERPROFILE=C:\Documents and Settings\cltbld.T-XP32-IX-125
   WINDIR=C:\WINDOWS
  using PTY: False
 2016-09-19 21:10:22,963 Setting DEBUG logging.

 2016-09-19 21:10:22,963 "default" was passed as the revision. Querying remote repository for corresponding rev hash of current default tip

 Traceback (most recent call last):

   File "archiver_client.py", line 313, in <module>

     main()

   File "archiver_client.py", line 296, in main

     options, args = options_args()

   File "archiver_client.py", line 262, in options_args

     options.rev = custom_mozharness_options(options)

   File "archiver_client.py", line 286, in custom_mozharness_options

     new_rev = subprocess.Popen(cmd, stdout=subprocess.PIPE).communicate()[0].strip()

   File "c:\mozilla-build\python27\lib\subprocess.py", line 679, in __init__

     errread, errwrite)

   File "c:\mozilla-build\python27\lib\subprocess.py", line 896, in _execute_child

     startupinfo)

 WindowsError: [Error 2] The system cannot find the file specified

 program finished with exit code 1
 elapsedTime=0.313000
========= master_lag: 0.01 =========

Comment 1

[Justin Wood (:Callek)]

1:25 PM <Callek> aleth: ooo the file not found is `https://dxr.mozilla.org/build-central/source/tools/buildfarm/utils/archiver_client.py#283`
1:27 PM <Callek> aleth: whats the time window for the bustage, (actual time, not csets)
1:28 PM <aleth> Wed Sep 14 13:10 ... 18:52
1:29 PM <Callek> aleth: markco arr: soo looks like that bug is *caused by* Bug 1302863 somehow
1:29 PM <Callek> as in, there seems to be something that was/is/whatever on the machines prior to that that isn't on them after

Comment 2

[aleth [:aleth]]

Note the file that is not being found is hg, so that points at a path/cwd issue.

Comment 3

[Jorg K (CEST = GMT+2)]

Aleth, thanks for filing this, I've seen the failures, but since bug 1219989 was always erroneously suggested, I didn't take any action. Also seen on C-A since the last merge on 20th Sept. 2016.

Comment 4

[ISHIKAWA, Chiaki]

Also seen on comm-central (try-comm-central).
E.g.:
https://treeherder.mozilla.org/logviewer.html#?job_id=21633&repo=try-comm-central#L189

Comment 5

[Mark Cornmesser [:markco]]

HG has been upgraded on all of the XP nodes. In that process hg should have been added into the path. 

Have we seen anymore of these errors in the past week, and if so do we have specific machines that this is happening on?

Comment 6

[aleth [:aleth]]

(In reply to Mark Cornmesser [:markco] from comment #5)
> Have we seen anymore of these errors in the past week, and if so do we have
> specific machines that this is happening on?

As far as I can tell, this was last seen on c-c on Monday Sep 26, so it looks like we're good - thanks!

Comment 7

[Jorg K (CEST = GMT+2)]

Last seen today:

XP:
https://treeherder.mozilla.org/#/jobs?repo=comm-central&revision=35368039b787ac417dbe605ad5a3287d6c89cd60&selectedJob=49265

Win 7:
https://treeherder.mozilla.org/#/jobs?repo=comm-central&revision=35368039b787ac417dbe605ad5a3287d6c89cd60&selectedJob=49268

Maybe I'm missing something.

Comment 8

[aleth [:aleth]]

(In reply to Jorg K (GMT+2) from comment #7)
> Last seen today:
> 
> XP:
> https://treeherder.mozilla.org/#/jobs?repo=comm-
> central&revision=35368039b787ac417dbe605ad5a3287d6c89cd60&selectedJob=49265
> 
> Win 7:
> https://treeherder.mozilla.org/#/jobs?repo=comm-
> central&revision=35368039b787ac417dbe605ad5a3287d6c89cd60&selectedJob=49268
> 
> Maybe I'm missing something.

Same file-not-found error message, but unrelated. This bug is about when the file not found is hg and pulling the repo fails, see earlier comments.