847424 - run sanity checks submitted releases as early as possible

Status: RESOLVED FIXED

(Release Engineering :: Applications: Shipit, defect, P3)

Description

[[e-mail] (:bhearsum)]

We've got the ability to run release sanity at will these days. I think it would be really great to know if it's going to pass before marking a release as "ready". This would let us avoid the mark as ready -> find failure -> edit -> repeat cycle that we've had on a few releases.

I'm thinking a better workflow could be:
* Submit release, which can't be marked as ready yet.
* Release runner runs release sanity, reports results (with specific error messages, if appropriate). If it's successful, the release can be marked as ready. If it fails, it can't.
* Human does review on ship it, marks as ready.
* Release runner commits changes, tags, reconfigs, starts release.

Comment 1

[[e-mail] (:bhearsum)]

We won't be able to do any of the checks that require tags in our buildbot repos as part of an initial release sanity run. Looks like verify_configs is the only check that cannot be done without the tags. It's still important to do this check later, though, so we should run release_sanity.py with --skip-verify-configs and --dryrun the first time, and then run it again when the release is marked as ready, but skip all of the other checks and only do verify_configs (because the buildbot repos will have been bumped/tagged.

We'll also need to adjust the verify_l10n_changesets and verify_l10n_dashboard checks not to pull the l10n_changsets file from the tagged buildbot-configs repo (we can probably pass in a local path instead.

Comment 2

[[e-mail] (:bhearsum)]

We talked about this at a planning meeting yesterday and decided it was a good idea, updating summary to reflect that.

Comment 3

[Rail Aliiev [:rail]]

Currently we split release sanity into 2 steps: everything that we can check before the builds are ready, and the rest. jlorenzo++