838485 - mozharness setup "rm -rf scripts" buildstep needs to set RETRY on failure

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Phil Ringnalda (:philor)]

It's hard to tell right now, since Windows on Cedar has been completely busted for a while, but it looks like the failure rate from permission denied during the rm -rf scripts step on Windows is about equal to the entire intermittent failure rate on non-mozharness Windows from all test failures. Like bug 838325, it needs to set RETRY on any non-zero result.

Comment 1

[Aki Sasaki (not active)]

Attachment: stop assuming it's 'hg'

Comment 2

[Aki Sasaki (not active)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=19477493&tree=Cedar&full=1

========= Started set props: script_repo_revision (results: 2, elapsed: 22 secs) (at 2013-02-05 20:03:08.078936) =========
'hg' 'id' '-i'
 in dir C:\\talos-slave\test\scripts (timeout 1200 secs)
 watching logfiles {}
 argv: ['hg', 'id', '-i']
 environment:
  ALLUSERSPROFILE=C:\Documents and Settings\All Users
  APPDATA=C:\Documents and Settings\cltbld\Application Data
  BOOTMODE=BKSTD
  CLIENTNAME=Console
  COMMONPROGRAMFILES=C:\Program Files\Common Files
  COMPUTERNAME=TALOS-R3-XP-098
  COMSPEC=C:\WINDOWS\system32\cmd.exe
  FP_NO_HOST_CHECK=NO
  HOMEDRIVE=C:
  HOMEPATH=\Documents and Settings\cltbld
  LOGONSERVER=\\TALOS-R3-XP-098
  NUMBER_OF_PROCESSORS=2
  OS=Windows_NT
  PATH=C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;c:\Program Files\gnuwin32\bin;c:\Python24\;c:\Python24\scripts;c:\Program Files\Vim\vim72
  PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH
  PROCESSOR_ARCHITECTURE=x86
  PROCESSOR_IDENTIFIER=x86 Family 6 Model 23 Stepping 10, GenuineIntel
  PROCESSOR_LEVEL=6
  PROCESSOR_REVISION=170a
  PROGRAMFILES=C:\Program Files
  PROMPT=$P$G
  PWD=C:\talos-slave\test\scripts
  SESSIONNAME=Console
  SYSTEMDRIVE=C:
  SYSTEMROOT=C:\WINDOWS
  TEMP=C:\DOCUME~1\cltbld\LOCALS~1\Temp
  TMP=C:\DOCUME~1\cltbld\LOCALS~1\Temp
  USERDOMAIN=TALOS-R3-XP-098
  USERNAME=cltbld
  USERPROFILE=C:\Documents and Settings\cltbld
  WINDIR=C:\WINDOWS
 using PTY: False
'hg' is not recognized as an internal or external command,
operable program or batch file.
program finished with exit code 1
elapsedTime=0.110000
script_repo_revision: "'hg' is not recognized as an internal or external command,\r\noperable program or batch file."
========= Finished set props: script_repo_revision (results: 2, elapsed: 22 secs) (at 2013-02-05 20:03:30.629737) =========

Comment 3

[Aki Sasaki (not active)]

Er, I suppose that's bug 838492.

Comment 4

[Phil Ringnalda (:philor)]

Yeah, that's the red now, but if you go back to before that, you have https://tbpl.mozilla.org/php/getParsedLog.php?id=19201366&tree=Cedar and https://tbpl.mozilla.org/php/getParsedLog.php?id=19056336&tree=Cedar and https://tbpl.mozilla.org/php/getParsedLog.php?id=19018960&tree=Cedar and https://tbpl.mozilla.org/php/getParsedLog.php?id=19015241&tree=Cedar and https://tbpl.mozilla.org/php/getParsedLog.php?id=18903054&tree=Cedar.

Probably not really quite as high as the intermittent test failure rate, but still significant.

Comment 5

[Aki Sasaki (not active)]

Attachment: add a log_eval_func to the clobber step

Using http://hg.mozilla.org/build/buildbotcustom/file/21b0f0b5e41c/process/factory.py#l462 as a model.

Comment 6

[Armen [:armenzg]]

Comment on attachment 710541 [details] [diff] [review]
stop assuming it's 'hg'

I had to use this exact patch on Friday for jetpack/mozharness mode.

Comment 7

[Aki Sasaki (not active)]

Comment on attachment 710541 [details] [diff] [review]
stop assuming it's 'hg'

http://hg.mozilla.org/build/buildbotcustom/rev/deba5135d777

Comment 8

[Aki Sasaki (not active)]

Comment on attachment 710544 [details] [diff] [review]
add a log_eval_func to the clobber step

http://hg.mozilla.org/build/buildbotcustom/rev/1f5fc9412a5c

Comment 9

[Aki Sasaki (not active)]

In production.