733658 - Try failure with "python: can't open file 'tools/buildfarm/maintenance/count_and_reboot.py': [Errno 2] No such file or directory"

Status: RESOLVED DUPLICATE of bug 712205

(Release Engineering :: General, defect)

Description

[Daniel Holbert [:dholbert]]

Just got a build failure on a Try build.

Build metadata:
builder: try-win32-debug
slave: w32-ix-slave23
starttime: 1331090581.42
results: retry (5)
builduid: cb970dc7268d43859b5591a31a73a15c
revision: 2a9660ea49110703eb9bbe2ec726cedacf90ec99
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-2a9660ea4911/try-win32-debug/try-win32-debug-build2603.txt.gz

The failure:
{
========= Started maybe rebooting failed (results: 2, elapsed: 0 secs) ==========
'python' 'tools/buildfarm/maintenance/count_and_reboot.py' '-f' '../reboot_count.txt' '-n' '1' '-z'
 in dir e:\builds\moz2_slave\try-w32-dbg\. (timeout 1200 secs)
 watching logfiles {}
 argv: ['python', 'tools/buildfarm/maintenance/count_and_reboot.py', '-f', '../reboot_count.txt', '-n', '1', '-z']
 environment:
[---SNIP---]
python: can't open file 'tools/buildfarm/maintenance/count_and_reboot.py': [Errno 2] No such file or directory
program finished with exit code 2
elapsedTime=0.125000
======== Finished maybe rebooting failed (results: 2, elapsed: 0 secs) ========
}

Comment 1

[Daniel Holbert [:dholbert]]

For some reason, Try is sending me a few emails per minute for the last 10 minutes with a new instance of this failure (and a differently-numbered log), for the same platform (try-win32-debug), for the same push.

I've gotten 13 such emails so far. Their logs are (including the one from comment 0):
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-2a9660ea4911/try-win32-debug/try-win32-debug-build2603.txt.gz
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-2a9660ea4911/try-win32-debug/try-win32-debug-build2604.txt.gz
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-2a9660ea4911/try-win32-debug/try-win32-debug-build2605.txt.gz
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-2a9660ea4911/try-win32-debug/try-win32-debug-build2606.txt.gz
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-2a9660ea4911/try-win32-debug/try-win32-debug-build2607.txt.gz
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-2a9660ea4911/try-win32-debug/try-win32-debug-build2608.txt.gz
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-2a9660ea4911/try-win32-debug/try-win32-debug-build2609.txt.gz
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-2a9660ea4911/try-win32-debug/try-win32-debug-build2610.txt.gz
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-2a9660ea4911/try-win32-debug/try-win32-debug-build2611.txt.gz
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-2a9660ea4911/try-win32-debug/try-win32-debug-build2612.txt.gz
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-2a9660ea4911/try-win32-debug/try-win32-debug-build2613.txt.gz

(I'm considering this possible-endless-spamming a distinct bug from what I've reported here. If it continues, I'll file a separate bug on it (and set up a mail filter. :))

Comment 2

[Daniel Holbert [:dholbert]]

(er, that was only 11 logs -- I accidentally omitted the last 2 because I saw the number "13" as the suffix and thought I'd pasted all of 'em. :)  Anyway -- I went ahead and filed bug 733663 on the possible infinite-spam-loop, and I gave the complete list of 13 failure logs (so far) there.)

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

There was a glitch at hg.m.o which caused errors for a few slaves cloning build/tools, where the reboot script lives. The job fails but buildbot is set to retry it in that case. The cycle repeats until build/tools can be cloned.

Comment 4

[Chris Cooper [:coop] (he/him)]

I'm going to dupe this to bug 712205 which should avoid this class of failures by not returning slaves to the pool that don't already have a tools checkout available. Not much we can do about the occasional hg failures otherwise.