1369383 - better way to shut off/on nightly updates

Status: RESOLVED FIXED

(Release Engineering Graveyard :: Applications: Balrog (backend), enhancement, P3)

Description

[[e-mail] (:bhearsum)]

It's relatively common to have the latest Nightly be in a bad state (eg: start up crash or major performance issues), and for us to want to temporarily disable updates to it. Right now the process for doing so is to poke someone with Balrog access (aka RelEng), and hope that they know how to do so. This process works OK as long as someone who knows enough about Balrog to do it is around, which isn't always the case.

This is a common enough case that we should probably find a more streamlined way of handling this. Depending which direction we go, the general "emergency shut off" in bug 1307169 might work. Catlee and I also talked about this recently and came up with the following process:
* Lock live channel to last good release
* After the bad nightlies stop submitting, copy the last known good release to the -latest blob, and point test channel rules at it
* Retriggered nightlies will submit to the -latest blob, and we can test on the test channel
* Once everything looks good, switch the live channel back to the -latest blob

One thing that may make this easier is to switch to a $product-$repo-$date-$revision format for blobs, to make it easier for whatever is doing the locking to find the correct previous nightly.

Comment 1

[Allan [:asilva]]

:catlee, :bhearsum, did the emergency shutoff fits well for the use case described in this bug?

Comment 2

[[e-mail] (:bhearsum)]

(In reply to Allan [:asilva] from comment #1)

:catlee, :bhearsum, did the emergency shutoff fits well for the use case
described in this bug?

IMO yes!