576569 - some sendchanges retry forever

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[[e-mail] (:bhearsum)]

I found a build in staging today which was using the following for a sendchange:
python /builds/slave/mozilla-central-linux/tools/buildfarm/utils/retry.py -s 5 -t 1800 -r 0 buildbot sendchange --master localhost:9009 --username sendchange-unittest --branch mozilla-central-linux-opt-unittest --revision 7f420dc0636fd74a7bd2d5b67770514b2e0a7e8c http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1278028954/firefox-4.0b2pre.en-US.linux-i686.tar.bz2 http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1278028954/firefox-4.0b2pre.en-US.linux-i686.tests.zip

the -r 0 causes retry. to retry forever. We need to find and fix all instances of this to avoid unnecessary slave hangups.

Comment 1

[Rail Aliiev [:rail]]

Attachment: Don't retry forever

This patch should reproduce the old behavior.
If we set retries to 0, this step should be run once and shouldn't cause red result. If retries > 0, we should retry (retries times) and make the build red on send failure.

Comment 2

[[e-mail] (:bhearsum)]

Comment on attachment 455748 [details] [diff] [review]
Don't retry forever

Looks good

Comment 3

[Rail Aliiev [:rail]]

Attachment: Set retry count

Based on catlee's advise instead of using implicit behavior (0 for 1 retry without failure and n for n+1 retry with failure) let's use explicit retry count in *config.py (5 for production and 1 for staging). Right now it's applicable for "mozilla" directory only (the corresponding changes in SendChangeStep landed to buildbotcustom "buildbot-0.8.0" branch only).

Comment 4

[Rail Aliiev [:rail]]

Attachment: Set retry count

Updated version, includes test-master02.

Comment 5

[Rail Aliiev [:rail]]

Comment on attachment 459064 [details] [diff] [review]
Set retry count

Robert, could you review seamonkey part of this patch?

Comment 6

[Rail Aliiev [:rail]]

Comment on attachment 459064 [details] [diff] [review]
Set retry count

http://hg.mozilla.org/build/buildbot-configs/rev/a80491cc84e3