Closed Bug 1379602 Opened 3 years ago Closed 3 years ago
.0b7 accidentally triggered in automation instead of Fennec 55 .0b7
This is actually a quite interesting bug. Thanks :jcristau for heads-up of . Context: Because bug 1347635 isn't finished yet, we still have to follow-up the following process for each Fennec release (beta/release/dot release,etc). That is, creating one subgraph encompassing the decision task + other post-release operations such as "mark release as shipped", "uptake monitoring", "push to releases", "version bump" and such. The decision task itself once ran, creates another graph - a nightly one - that uses the nightly CI as builds for Fennec beta. For the first graph tasks (post release ops and such) we send over a bunch of arguments via env vars as REVISION, BRANCH, VERSION and BUILD_NUMBER. While nightly graph gets most of these from in-tree (e.g. https://hg.mozilla.org/mozilla-central/file/tip/taskcluster/taskgraph/util/scriptworker.py#l394 ) the first graph relies on human, hence there's room for accidental mistakes. So what happened last week? * RelMan submitted the Fennec 55.0b7 in Ship-it and pinged us to manually trigger the corresponding graphs * We used  to trigger it but instead of "55.0b7" pe specified "57.0b7". Bm85 bash history confirm this * first graph (hence all post-release ops, push to releases and alike) take into account version 57.0b7 instead of 55.0b7. * the nightly graph triggered by decision task ignores this issue because it looks up in the in-tree for grabbing its stuff (such as current version) * because beetmover takes the version from in-tree, it correctly produces candidates 55.0b7 That explains how: * nightly graph triggered by decision task (that again, doesn't use any of the env var specified) is green * second graph is completely wrong as all its routes lead to "57.0b7". * mozilla-beta candidates_fennec is correctly green as it didn't take into account any of the VERSION=57.0b7 wrongly specified * Generate beetmover docker image is green as expected * fennec mozilla-beta version bump - it actually run correctly producing , because the version is bogus. Normally it would have been a no-op * fennec mozilla-beta uptake monitoring failed as expected since "57.0b7" doesn't actually exist in mirrors * fennec mozilla-beta mark release as shipped failed as expected as in Ship-it it was correctly submitted as 55.0b7 and *not* 57.0b7 * fennec mozilla-beta bouncer aliases never actually run since it's chained to (failing) uptake monitoring job * fennec mozilla-beta bncr sub run successfully - we need to undo this. * fennec mozilla-beta push to releases - it silently returned green although it didn't transfer any files from ~/candidates-55.0b7 -> ~/releases/57.0b7. We need to fix this, filed bug 1379599 to track it. Actions: * we need to undo fennec mozilla-beta bncr sub effects for 57.0b7 * we need to make sure having artifacts routed for 57.0b7 won't bite us in December when we'll actually deal with that given release * we need to retrigger the first graph, skipping the decision task so that we push the proper bits under 55.0b7 * we need to backout the version bump changesets  Conclusions: * it's always good to make sure all the tasks are green before marking the release as completed in releasewarrior * we need better warnings so that we don't silently succeed - e.g. bug 1379599 * we need to make sure we automate this as soon as possible so that these values are no longer specified manually. automation FTW! : https://hg.mozilla.org/releases/mozilla-beta/rev/acfd0992ed18 : https://github.com/mozilla/releasewarrior/blob/master/how-tos/fennec-temp-relpro.md#start-off-the-fennec-graph : https://tools.taskcluster.net/groups/aVI4rXb2TSu9k67ADC44QQ : https://tools.taskcluster.net/groups/3UmSrcH0S9qS9uTJyuEl8g : https://hg.mozilla.org/releases/mozilla-beta/rev/acfd0992ed18 : https://hg.mozilla.org/releases/mozilla-beta/rev/8888750365b1 : http://archive.mozilla.org/pub/mobile/candidates/55.0b7-candidates/
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #0) > Actions: > * we need to backout the version bump changesets  done: https://hg.mozilla.org/releases/mozilla-beta/rev/f6234a1e8f5c https://hg.mozilla.org/releases/mozilla-beta/rev/24f20dadf233
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #0) > Actions: > * we need to retrigger the first graph, skipping the decision task so that > we push the proper bits under 55.0b7 Done. See https://tools.taskcluster.net/groups/RK-qbnuKRlCYwwpPg82EJw and http://archive.mozilla.org/pub/mobile/releases/55.0b7/
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #0) > Actions: > * we need to undo fennec mozilla-beta bncr sub effects for 57.0b7 Done. Removed from bouncer admin the entry for Product Fennec-57.0b7 with its corresponding locations.
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #0) > Actions: > * we need to make sure having artifacts routed for 57.0b7 won't bite us in > December when we'll actually deal with that given release Leftover: ^^ NI to myself to discuss this at the post-mortem.
Rok says this shouldn't be a problem as routes are being created/updated when we'll have the next release. I'm thikining the same as the future real 57.0b7 will have a different revision, hence other routes. I left the item in the post-mortem meeting to confirm with :rail but we can close this bug for now.
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.