Closed Bug 1379602 Opened 3 years ago Closed 3 years ago

Fennec 57.0b7 accidentally triggered in automation instead of Fennec 55.0b7

Categories

(Release Engineering :: Release Automation: Other, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: mtabara, Assigned: mtabara)

Details

This is actually a quite interesting bug. Thanks :jcristau for heads-up of [0].

Context:
Because bug 1347635 isn't finished yet, we still have to follow-up the following process[1] for each Fennec release (beta/release/dot release,etc). That is, creating one subgraph encompassing the decision task + other post-release operations such as "mark release as shipped", "uptake monitoring", "push to releases", "version bump" and such. The decision task itself once ran, creates another graph - a nightly one - that uses the nightly CI as builds for Fennec beta.

For the first graph tasks (post release ops and such) we send over a bunch of  arguments via env vars as REVISION, BRANCH, VERSION and BUILD_NUMBER. While nightly graph gets most of these from in-tree (e.g. https://hg.mozilla.org/mozilla-central/file/tip/taskcluster/taskgraph/util/scriptworker.py#l394 ) the first graph relies on human, hence there's room for accidental mistakes.

So what happened last week?
* RelMan submitted the Fennec 55.0b7 in Ship-it and pinged us to manually trigger the corresponding graphs
* We used [1] to trigger it but instead of "55.0b7" pe specified "57.0b7". Bm85 bash history confirm this
* first graph (hence all post-release ops, push to releases and alike) take into account version 57.0b7 instead of 55.0b7.
* the nightly graph triggered by decision task ignores this issue because it looks up in the in-tree for grabbing its stuff (such as current version)
* because beetmover takes the version from in-tree, it correctly produces candidates 55.0b7[6]

That explains how:
* nightly graph triggered by decision task (that again, doesn't use any of the env var specified) is green[2]
* second graph[3] is completely wrong as all its routes lead to "57.0b7".
* mozilla-beta candidates_fennec is correctly green as it didn't take into account any of the  VERSION=57.0b7 wrongly specified
* Generate beetmover docker image is green as expected
* fennec mozilla-beta version bump - it actually run correctly producing [0], because the version is bogus. Normally it would have been a no-op
* fennec mozilla-beta uptake monitoring  failed as expected since "57.0b7" doesn't actually exist in mirrors
* fennec mozilla-beta mark release as shipped failed as expected as in Ship-it it was correctly submitted as 55.0b7 and *not* 57.0b7
* fennec mozilla-beta bouncer aliases never actually run since it's chained to (failing) uptake monitoring job
* fennec mozilla-beta bncr sub run successfully - we need to undo this.
* fennec mozilla-beta push to releases - it silently returned green although it didn't transfer any files from ~/candidates-55.0b7 -> ~/releases/57.0b7. We need to fix this, filed bug 1379599 to track it.

Actions:
* we need to undo fennec mozilla-beta bncr sub effects for 57.0b7
* we need to make sure having artifacts routed for 57.0b7 won't bite us in December when we'll actually deal with that given release
* we need to retrigger the first graph, skipping the decision task so that we push the proper bits under 55.0b7
* we need to backout the version bump changesets [4][5]

Conclusions: 
* it's always good to make sure all the tasks are green before marking the release as completed in releasewarrior
* we need better warnings so that we don't silently succeed - e.g. bug 1379599
* we need to make sure we automate this as soon as possible so that these values are no longer specified manually. automation FTW!

[0]: https://hg.mozilla.org/releases/mozilla-beta/rev/acfd0992ed18
[1]: https://github.com/mozilla/releasewarrior/blob/master/how-tos/fennec-temp-relpro.md#start-off-the-fennec-graph
[2]: https://tools.taskcluster.net/groups/aVI4rXb2TSu9k67ADC44QQ
[3]: https://tools.taskcluster.net/groups/3UmSrcH0S9qS9uTJyuEl8g
[4]: https://hg.mozilla.org/releases/mozilla-beta/rev/acfd0992ed18
[5]: https://hg.mozilla.org/releases/mozilla-beta/rev/8888750365b1
[6]: http://archive.mozilla.org/pub/mobile/candidates/55.0b7-candidates/
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #0)
> Actions:
> * we need to backout the version bump changesets [4][5]

done:
https://hg.mozilla.org/releases/mozilla-beta/rev/f6234a1e8f5c
https://hg.mozilla.org/releases/mozilla-beta/rev/24f20dadf233
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #0)
> Actions:
> * we need to retrigger the first graph, skipping the decision task so that
> we push the proper bits under 55.0b7

Done. See https://tools.taskcluster.net/groups/RK-qbnuKRlCYwwpPg82EJw and 
http://archive.mozilla.org/pub/mobile/releases/55.0b7/
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #0) 
> Actions:
> * we need to undo fennec mozilla-beta bncr sub effects for 57.0b7

Done. Removed from bouncer admin the entry for Product Fennec-57.0b7 with its corresponding locations.
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #0)
> Actions:
> * we need to make sure having artifacts routed for 57.0b7 won't bite us in
> December when we'll actually deal with that given release

Leftover: ^^
NI to myself to discuss this at the post-mortem.
Flags: needinfo?(mtabara)
Flags: needinfo?(mtabara)
Flags: needinfo?(mtabara)
Rok says this shouldn't be a problem as routes are being created/updated when we'll have the next release. I'm thikining the same as the future real 57.0b7 will have a different revision, hence other routes.

I left the item in the post-mortem meeting to confirm with :rail but we can close this bug for now.
Status: NEW → RESOLVED
Closed: 3 years ago
Flags: needinfo?(mtabara)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.