794760 - 'clobber build tools' step should haltOnFailure and set RETRY

Status: VERIFIED FIXED

(Release Engineering :: General, defect)

Description

[Ed Morley [:emorley]]

In bug 692715, we get:

{
========= Started clobber build tools failed (results: 2, elapsed: 59 secs) (at 2012-09-26 19:23:55.591877) =========
'rm' '-rf' 'tools'
 in dir C:\talos-slave\test\. (timeout 1200 secs)
 watching logfiles {}
 argv: ['rm', '-rf', 'tools']
<snip>
rm: cannot lstat `tools/release/l10n': Permission denied
program finished with exit code 1
elapsedTime=55.612000
========= Finished clobber build tools failed (results: 2, elapsed: 59 secs) (at 2012-09-26 19:24:55.223317) =========

========= Started clone build tools failed (results: 2, elapsed: 2 secs) (at 2012-09-26 19:24:55.224105) =========
'hg' 'clone' 'http://hg.mozilla.org/build/tools' 'tools'
 in dir C:\talos-slave\test\. (timeout 1320 secs)
 watching logfiles {}
 argv: ['hg', 'clone', 'http://hg.mozilla.org/build/tools', 'tools']
<snip>
abort: destination 'tools' already exists
program finished with exit code -1
elapsedTime=0.901000
========= Finished clone build tools failed (results: 2, elapsed: 2 secs) (at 2012-09-26 19:24:57.424973) =========

========= Started set props: toolsdir (results: 0, elapsed: 0 secs) (at 2012-09-26 19:24:57.425594) =========
'bash' '-c' 'pwd'
 in dir C:\talos-slave\test\tools (timeout 1200 secs)
 watching logfiles {}
 argv: ['bash', '-c', 'pwd']
<snip>
C:\talos-slave\test\tools
program finished with exit code 0
elapsedTime=0.101000
toolsdir: 'C:\\talos-slave\\test\\tools'
========= Finished set props: toolsdir (results: 0, elapsed: 0 secs) (at 2012-09-26 19:24:57.659934) =========

========= Started set props: purge_target (results: 2, elapsed: 0 secs) (at 2012-09-26 19:24:57.660570) =========
'python' 'tools/buildfarm/maintenance/purge_builds.py' '-s' '1.0' '-n' 'info' '-n' 'rel-*' '..' '/mock/users/cltbld/home/cltbld/build'
 in dir C:\talos-slave\test\. (timeout 3600 secs)
 watching logfiles {}
 argv: ['python', 'tools/buildfarm/maintenance/purge_builds.py', '-s', '1.0', '-n', 'info', '-n', 'rel-*', '..', '/mock/users/cltbld/home/cltbld/build']
<snip>
python: can't open file 'tools/buildfarm/maintenance/purge_builds.py': [Errno 2] No such file or directory
program finished with exit code 2
elapsedTime=0.101000
purge_target: '1.0GB'
========= Finished set props: purge_target (results: 2, elapsed: 0 secs) (at 2012-09-26 19:24:57.849780) =========
}

TBPL now recognises "^abort: " type failures, and thus is able to display:
https://tbpl.mozilla.org/php/getParsedLog.php?id=15575462&branch=mozilla-inbound
...but the root cause of bug 692715 was the step before.

I'm going to add TBPL parser support for "rm: cannot lstat `.*': Permission denied" (or similar), but it would help to halt the run on the clobber build tools step, so we don't waste time trying later steps, and so the TBPL annotated summary displays just:

> rm: cannot lstat `.*': Permission denied

and not

> rm: cannot lstat `.*': Permission denied
> abort: destination 'tools' already exists

Comment 1

[Ed Morley [:emorley]]

Attachment: Patch v1

Not sure who best to ask to review these types of patches, otherwise I'd try to distribute the load a bit more; sorry!

What are your thoughts on adding haltOnFailure=True to more of the tools type steps in factory.py (in another bug)? From glancing through it seems like many of them should haltOnFailure and yet don't - and so potentially waste time/make failures harder to spot in the logs for the average dev looking at them.

Comment 3

[Phil Ringnalda (:philor)]

Mine was better on two counts, RETRY and "unicorn ponies pooping candy rainbows" :)

Comment 4

[Ed Morley [:emorley]]

(In reply to Phil Ringnalda (:philor) from comment #3)
> Mine was better on two counts, RETRY and "unicorn ponies pooping candy
> rainbows" :)

I think you'll always beat me on the humorous bug comments front :-)

Comment 5

[Ed Morley [:emorley]]

Attachment: haltOnFailure & RETRY

Comment 6

[Ed Morley [:emorley]]

Comment on attachment 665297 [details] [diff] [review]
haltOnFailure & RETRY

Thank you for the review :-)

http://hg.mozilla.org/build/buildbotcustom/rev/68cfc61dca7f

Comment 7

[Aki Sasaki (not active)]

This is now live.

Comment 8

[Ed Morley [:emorley]]

Thank you :-)

Comment 9

[Phil Ringnalda (:philor)]

And, given enough time to remember that https://tbpl.mozilla.org/?noignore=1&jobname=6.1%20x64 will show it two or three times per push, working exactly as intended.