1608882 - Migrate Fennec Nightly to Fenix Nightly

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, enhancement)

Description

[Johan Lorenzo [:jlorenzo]]

This requires us to turn off Fennec Nightly first, by:

0. Make push-apk support rollout percentages on other tracks than production. Google Play seems to support it these days. I know there's at least [0] to do. There may be more.
1. Removing the cron entry[1].
2. Removing the force cron hook[2] (in order to make sure nobody inadvertently triggers a new Fennec that would be published, thus erase user data)
3. Copy these bits of the push-apk configuration to the mobile pool. By the way, if we could stop naming it "aurora", that would be great. If you wonder why Nightly is named this way, see bug 1357351 comment 1.
4. Merge the push-apk task kind[4] into the one in Fenix.

Note there is no signing-related worked because we've signed Fenix with the Fennec key for quite some time, already.

[0] https://github.com/mozilla-releng/mozapkpublisher/blob/26cbc9f8b47f12978f8ba1bc05bf8c09e62c1ec4/mozapkpublisher/push_apk.py#L131
[1] https://searchfox.org/mozilla-esr68/rev/838ffc0d8db638590b42fa9e684be751dc060439/.cron.yml#85-99
[2] https://hg.mozilla.org/ci/ci-configuration/file/6c40940b46d8d934c5b0aaa935ab242326e2a4a1/projects.yml#l87
[3] https://github.com/mozilla-releng/scriptworker-scripts/blob/e1bd2ae0fe836ac50b8833f4df989e82cee01ca7/pushapkscript/docker.d/worker.yml#L36-L42
[4] https://searchfox.org/mozilla-esr68/rev/838ffc0d8db638590b42fa9e684be751dc060439/taskcluster/ci/push-apk/kind.yml#40-50
[5] https://github.com/mozilla-mobile/fenix/blob/399df170629bbb164ed84912287248559dddc31a/taskcluster/ci/push-apk/kind.yml#L30

Comment 1

[Johan Lorenzo [:jlorenzo]]

(In reply to Johan Lorenzo [:jlorenzo] from comment #0)

0. Make push-apk support rollout percentages on other tracks than production. Google Play seems to support it these days. I know there's at least [0] to do. There may be more.

Good news! Mitch took care of it about 6 months ago in [1]. It was just the help message being outdated. I'd like to make sure the full stack works. I plan to land a small patch that rolls out the next Fennec Nightly.

[1] https://github.com/mozilla-releng/mozapkpublisher/pull/219/commits/918ec6b9a4357cc95c4a4631a500f905c7e1870b#diff-0eca0f1f55bbb0ccdc047ed838061cc0R241

Comment 2

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1608882 - part 1: Test whether Fennec Nightly can be rolled out before migrating over to Fenix r=jcristau

Comment 3

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 9120738 [details]
Bug 1608882 - part 1: Test whether Fennec Nightly can be rolled out before migrating over to Fenix r=jcristau

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: Fennec Nightly is only present on ESR68. In order to make sure we can slowly rollout Fenix on the Nightly userbase, I'd like to make sure our stack supports such a use case. I plan to backout this patch if the test is successful.
• User impact if declined: We may deploy Fenix to 100% of the userbase at once.
• Fix Landed on Version: N/A. This patch is ESR-68 only.
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): It's a one-liner that deploys Fenix to 99% of the userbase, instead of 100%. Worst thing that can happen: we miss 1 day of nightly because the infra fails somewhere.
• String or UUID changes made by this patch: None

Comment 5

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9120738 [details]
Bug 1608882 - part 1: Test whether Fennec Nightly can be rolled out before migrating over to Fenix r=jcristau

Needed to test Fennec -> Fenix migration. Approved for ESR68.

Comment 6

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr68/rev/49285dbee11e

Comment 7

[Johan Lorenzo [:jlorenzo]]

Attachment: [Google Play] Fennec Nightly after 99% rollout was requested

Excellent! It worked as expected => the push-apk stack correctly sent the new APKs to 99% rollout.

2020-01-15 12:16:14,204 - mozapkpublisher.common.store - INFO - Track "beta" updated with: {'releases': [{'status': 'inProgress', 'userFraction': 0.99, 'versionCodes': ['2015675727', '2015675721', '2015675723', '2015675725']}]}
2020-01-15 12:16:14,205 - mozapkpublisher.common.store - DEBUG - Track update response: {'track': 'beta', 'releases': [{'name': '', 'versionCodes': ['2015675721', '2015675723', '2015675725', '2015675727'], 'userFraction': 0.99, 'status': 'inProgress'}]}
2020-01-15 12:16:14,206 - googleapiclient.discovery - INFO - URL being requested: POST https://www.googleapis.com/androidpublisher/v3/applications/org.mozilla.fennec_aurora/edits/07646663629160204000:commit?alt=json
2020-01-15 12:16:17,255 - mozapkpublisher.common.store - INFO - Changes committed

https://firefoxci.taskcluster-artifacts.net/WSR9QiDHSj6jxGdW8VDndg/0/public/logs/live_backing.log

Ryan, would you mind backing out my change and make the latest nightly available to 100% of the user base?

Comment 8

[Ryan VanderMeulen [:RyanVM]]

Backed out:
https://hg.mozilla.org/releases/mozilla-esr68/rev/7d85cbdd3ea34c2d5b940c92387699eea3901ace

Comment 9

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9120738 [details]
Bug 1608882 - part 1: Test whether Fennec Nightly can be rolled out before migrating over to Fenix r=jcristau

Clearing the approval flag to get this off the needs-uplift radar.

Comment 10

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1608882 - part 1: Fenix, expose dedicated fennec-nighty hook r=mtabara

Comment 11

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: [pushapkscript] Fix Fennec-nightly related credentials in cloudopsinfra

Comment 12

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1608882 - part 2: Add missing scopes to let the new fennec-nightly cron job be green r=mtabara

Comment 13

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1608882 - part 2: Deactivate Fennec nightly hook so it does not mess up the Fenix transition r=RyanVM

Comment 14

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 9121849 [details]
Bug 1608882 - part 2: Deactivate Fennec nightly hook so it does not mess up the Fenix transition r=RyanVM

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: ESR patch only because Fennec Nightly is shipped off this tree.
• User impact if declined: Fennec users who have migrated to Fenix will be sent back to Fennec causing major data migration problems.
• Fix Landed on Version:
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): It's just a cron job removed.
• String or UUID changes made by this patch: None

Comment 15

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9121849 [details]
Bug 1608882 - part 2: Deactivate Fennec nightly hook so it does not mess up the Fenix transition r=RyanVM

Removes the cron job for shipping Fennec Nightly builds from ESR68 to avoid interfering with the Fenix migration. Approved for ESR68.

Comment 16

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9121849 [details]
Bug 1608882 - part 2: Deactivate Fennec nightly hook so it does not mess up the Fenix transition r=RyanVM

Pushed to ESR68:
https://hg.mozilla.org/releases/mozilla-esr68/rev/2ebdfc0a64f11b02b65824f1c6bea38e2f9e9cd8

Also removing the approval flag to keep this off the needs-uplift radar.

Comment 17

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1608882 - part 3: Grant mobile releases the ability to trigger the new fennec-nightly hook r=mtabara

Comment 18

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1608882 - part 4: Remove `nightly-android` force hook r=mtabara

Comment 19

[Johan Lorenzo [:jlorenzo]]

The migration has started for several weeks and is still being rolled out[1]. We've published 7 builds so far and there has been no need to update the pipeline since build 2. We're now looking at migrating beta with a similar pipeline. Thus, I'm closing this bug. Feel free to reopen something needs to happen with nightly.

[1] https://arewefenixyet.com/

Comment 20

[Johan Lorenzo [:jlorenzo]]

Attachment: [fenix] part 1: Move Fennec Nightly to a dedicated hook

Comment 21

[Johan Lorenzo [:jlorenzo]]

Attachment: [fenix] part 2: Let fennec-nightly upload APKs on Google Play's alpha track

Comment 22

[Johan Lorenzo [:jlorenzo]]

Attachment: [scriptworker-scripts] Fix mobile staging instance and enable fennec-nightly

Comment 23

[Johan Lorenzo [:jlorenzo]]

Attachment: [fenix] let fennec-nightly tasks ride along vanilla fenix nightlies

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/37802f546527
[fenix] Bug 1608882 - part 1: Move Fennec Nightly to a dedicated hook
https://hg.mozilla.org/integration/autoland/rev/c580fa36db17
[fenix] Bug 1608882 - part 2: Let fennec-nightly upload APKs on Google Play's alpha track