440690 - Buildbot based nightlies (like Mozilla2) should keep attempting the nightly until it succeeds

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect, P3)

Description

[Nick Thomas [:nthomas] (UTC+12)]

We had a situation today where a bad checkin was made just before the nightly builds kicked off and they all burned. AFAICT our buildbot setup doesn't retry- until-success, so it's up to someone to force a build when builders are green again. I think retrying is a useful feature of the tinderbox client that we should bring over. Comments ?

Comment 1

[[e-mail] (:bhearsum)]

I think this is a great idea. We might be able to use a hacky BuildStep at the end of the nightlies to kick a new build if anything in the current one has failed, but really, I think this probably has to be done in Build, Builder, a Scheduler, or a combination.

Comment 2

[Chris Cooper [:coop] (he/him)]

Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

From triage: We already have nagios warning us if a nightly build is >28 hours old. If nightly build fails, RelEng wants to investigate why before we manually kick off new nightlies. Nagios should be enough, so closing.