532228 - retrying sendchanges can bury buildbot master

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

If a buildbot master (ie pm01) gets busy from closely-spaced checkins then the sendchange's for unit test runs can get a very slow response. This causes the build to go into a warning state and confuse developers. 

Since bug 523904 we retry the sendchange, resulting in up to 5 requests for the same build. Some of these requests are merged away, some not. Currently there are 200 jobs waiting on pm01 for win32, about 60 for mac.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Make number of retries configurable

r+ from bhearsum on irc,
http://hg.mozilla.org/build/buildbotcustom/rev/0fda5ebdae4f

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Merge old style unit tests (non opt/debug)

r+ from bhearsum on irc
http://hg.mozilla.org/build/buildbotcustom/rev/6762cf582d75

We might back this out once the backlog is cleared.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Set config for unit tests to not retry sendchanges

r+ from bhearsum on irc
http://hg.mozilla.org/build/buildbot-configs/rev/588979445ea0

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Gozer, KaiRo - These changes might cause you some issues.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

pm01 & pm02 have been reconfigured with these changes (finished a few minutes ago). The backlog should start to clear out faster now.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Backlog is cleared. Leaving this open to discuss how much of this we want to keep permanently. Your thoughts catlee and bhearsum ?

Comment 7

[[e-mail] (:bhearsum)]

Until we can guarantee that we won't submit the same build twice I think we should leave on merging. With that on, I'd be OK having retries set to 1 or 2, maybe 3 - but given how bad things got yesterday I'm hesitant to crank it up higher.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Fix the retries and set a long timeout

We're still getting multiple retries on unit test sendchanges because the initial patch was incomplete (thanks for spotting that), and set a long enough timeout that even a 20 minute reconfig on pm will not drop changes.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 415712 [details] [diff] [review]
Fix the retries and set a long timeout

http://hg.mozilla.org/build/buildbotcustom/rev/4916ba025648

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Pretty sure this is no longer an issue.