849783 - release runner shouldn't sleep for so long after failing

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[[e-mail] (:bhearsum)]

Release runner sleeps after dying because we don't want to accidentally start a release that's had a problem. Due to recent changes I don't think this is necessary anymore. We have a few situations in which release runner can die:
1) Dying during polling - we definitely don't need to sleep after this, as no releases have been marked as ready yet.
2) Dying during release sanity - release sanity failures are almost always repeatable. If the release was tried again without changes, it will fail again. Additionally, the release will be marked as not ready before release runner exits.
3) Dying elsewhere during setup (eg, writing or pushing configs). These failures might not be repeatable, but we still mark the release as not ready if we hit them.

The only time that I can think of in which we'd retry a release without human intervention is if we failed to mark it as failed on ship-it.m.o. If we're paranoid about that, we should make it so we only sleep in that situation. Otherwise we should just never sleep.

Comment 1

[[e-mail] (:bhearsum)]

Attachment: wake up!!!

Comment 2

[[e-mail] (:bhearsum)]

Attachment: snooze

Comment 3

[[e-mail] (:bhearsum)]

Comment on attachment 724028 [details] [diff] [review]
snooze

I landed this, updated the checkout on bm36, and restarted release runner w/ supervisorctl.