613953 - Build / repack uploads should be retried if they fail

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Chris AtLee [:catlee]]

Regular and release builds and repacks often hit problems uploading to stage (bug 610399).

Instead of dying, we should re-try the upload a few times.

Comment 1

[[e-mail] (:bhearsum)]

Planning to fix this this quarter.

Comment 2

[[e-mail] (:bhearsum)]

The repack part of this was fixed in bug 613970.

Comment 3

[[e-mail] (:bhearsum)]

Attachment: [untested] add retry.py a bunch of places

This patch adds retry.py to a bunch of places, including (most) uploads. Still doing a bunch of testing on it.

Comment 4

[[e-mail] (:bhearsum)]

Attachment: refined version

Due to the upload_errors stuff I added in bug 661401 this got a bit more complicated. In cases where we have to retry at least once, but succeed in the end, there will be error messages caught by the log_eval_func and the overall status will get set to RETRY. To workaround that, all of the Retrying* steps will always succeed if the return code is 0, putting the onus on retry.py to exit correctly.

I tested this by using 'iptables -A OUTPUT -d 10.2.71.82 -j REJECT' on mv-moz2-linux-ix-slave01 to cause quick Connection Refused messages. I tested the succeeds-on-first-attempt, succeeds-on-subsequent-attempt, and never-succeeds scenarios. The first two resulted in SUCCESS, the latter in RETRY.

Comment 5

[[e-mail] (:bhearsum)]

Comment on attachment 538049 [details] [diff] [review]
refined version

Landed on the default branch of buildbotcustom.

Comment 6

[[e-mail] (:bhearsum)]

This made it to production.

Comment 7

[Aki Sasaki (not active)]

Backed out due to posix path to retry.py borking 192 win32 .

Comment 8

[[e-mail] (:bhearsum)]

I can't find any errors like the one you describe on http://tbpl.mozilla.org/?tree=Firefox3.6, can you point me at the specific issue?

Comment 9

[[e-mail] (:bhearsum)]

Found it: http://tinderbox.mozilla.org/showlog.cgi?log=Firefox3.6/1307745554.1307746489.759.gz&fulltext=1

python /e/builds/moz2_slave/192-w32-unittest/tools/buildfarm/utils/retry.py -s 1 -r 5 python 192-w32-unittest/tools/clobberer/clobberer.py -s tools -t 168 http://build.mozilla.org/clobberer/index.php mozilla-1.9.2 WINNT 5.2 mozilla-1.9.2 unit test 192-w32-unittest mw32-ix-slave14 http://buildbot-master08.build.scj1.mozilla.com:8001/
 in dir e:\builds\moz2_slave\192-w32-unittest\.. (timeout 3600 secs)
 watching logfiles {}
 argv: ['python', '/e/builds/moz2_slave/192-w32-unittest/tools/buildfarm/utils/retry.py', '-s', '1', '-r', '5', 'python', '192-w32-unittest/tools/clobberer/clobberer.py', '-s', 'tools', '-t', '168', 'http://build.mozilla.org/clobberer/index.php', 'mozilla-1.9.2', 'WINNT 5.2 mozilla-1.9.2 unit test', '192-w32-unittest', 'mw32-ix-slave14', u'http://buildbot-master08.build.scj1.mozilla.com:8001/']

python: can't open file '/e/builds/moz2_slave/192-w32-unittest/tools/buildfarm/utils/retry.py': [Errno 2] No such file or directory
program finished with exit code 2
elapsedTime=0.110000

Comment 10

[[e-mail] (:bhearsum)]

Attachment: full patch, with fix for UnittestBuildFactory/MozillaClobber

Same as before, except I've added the "pwd -W" toolsdir fix in UnittestBuildFactory, because it uses MercurialCloneCommand (which uses retry.py), and I've turned MozillaClobberer back into a ShellCommand, because toolsdir isn't set properly on Windows in MozillaBuildFactory. I ran some 1.9.2 and m-c builds in staging, including unittests - they worked fine.

Comment 11

[[e-mail] (:bhearsum)]

Comment on attachment 538926 [details] [diff] [review]
full patch, with fix for UnittestBuildFactory/MozillaClobber

This is on the default branch again, heading to production later today.

Comment 12

[[e-mail] (:bhearsum)]

Haven't seen any additional fallout.

Comment 13

[John Ford [:jhford] CET/CEST Berlin Time]

this is causing errors with make upload in scratchbox commands.

Comment 14

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: fix scratchboxcommand fallout

This patch should fix the root problem.  I haven't had a chance to run test_masters yet, but I have tested that the list comprehension is valid

>>> [str(x) for x in [1,2,3,'john']]
['1', '2', '3', 'john']

Comment 15

[[e-mail] (:bhearsum)]

Comment on attachment 539246 [details] [diff] [review]
fix scratchboxcommand fallout

I don't want to take this as a bustage fix. I'm attaching a safer fix.

Comment 16

[[e-mail] (:bhearsum)]

Attachment: work around limitations by forcing timeout to be a string

Comment 17

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 539248 [details] [diff] [review]
work around limitations by forcing timeout to be a string

r+ for bustage

Comment 18

[[e-mail] (:bhearsum)]

Comment on attachment 539246 [details] [diff] [review]
fix scratchboxcommand fallout

We didn't end up using this patch.

Comment 19

[[e-mail] (:bhearsum)]

Comment on attachment 539248 [details] [diff] [review]
work around limitations by forcing timeout to be a string

This was landed a few days ago.

Comment 20

[[e-mail] (:bhearsum)]

This is all done, again. bug 664211