505392 - Release builders should be independently triggerable

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P4)

Description

[Chris AtLee [:catlee]]

If something goes wrong during a release, the current practice is to edit release_master.py, and replace successfully completed portions with dummy factories, and then re-initiate a "complete" release.  This requires somebody to ssh into production-master, edit release_master.py, reconfig, trigger the release, then undo the changes again; a very manual and error-prone process.

I think this is due to our use of Dependent Schedulers, particularly for the l10n repacks.

It would be great if each step of the release process could be kicked off independently, so resuming a release after some step fails is simple.

Comment 1

[[e-mail] (:bhearsum)]

Yeah, the DependentL10n Scheduler is the cause of this. We can already kick-off l10n_verification, updates, all the update verifies, and final verification without any config modification. If we didn't use the DependentL10n Scheduler, or if it worked through force build or some other manner we could use force build for any part of the release process.

One idea that was thrown out last week was to have a regular Scheduler that knows how to trigger the l10n builds, which could be kicked through sendchange if needed. This might be pretty simple by simply combining the buildbotcustom.l10n.Scheduler and L10nMixin objects.

Comment 2

[Chris Cooper [:coop] (he/him)]

Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.

Comment 3

[Armen [:armenzg]]

bhearsum's approach makes sense.

Comment 5

[[e-mail] (:bhearsum)]

I believe that this is fixed since bug 608008 landed. There is no part of the buildbot parts of the release automation that cannot be triggered through force build or sendchange. We're not yet in a state where downstream schedulers fire when things are triggered this way, but that's definitely a different bug, if wanted at all.