1648785 - Fenix: Upload signed APKs to Github releases

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Uploading, enhancement)

Description

[Johan Lorenzo [:jlorenzo]]

Context

Since the first mobile project[1], we have relied on Github Releases to kick off release graphs. With bug 1568466 coming, the process will change: shipit[2] will start them. We do want to keep Github Releases around because this remains a way to tell Github users a new release has shipped.

In bug 1635488, we're adding tree-related operations like file bumps to treescript. As far as I understand, we won't use any Github-specific features. Github Releases are specific to this product. Let's create a new scriptworker-script to deal with Github specific operations.

Workflow

Thanks to this new worker, we can automate the publication of APKs to Github. So far, it has been done manually[3]. Tasks run by this worker will be idempotent:

1. It will create a Github release if it doesn't exist already, otherwise it will update it
2. It will check whether artifacts are attached to this release and whether they were correctly uploaded. If not, it will upload APKs
3. It will check that the 2 previous operations succeeded by pulling the latest data and verifying it matches the expected one.

Security concerns

I worked on a script that works locally for me. I see one thing to call out before going to production: how should the worker authenticate to Github?

a) First of all, the worker needs to have restricted access to perform the operations described above on a single repository only. I don't think we want to put several projects at risk if we end up leaking credentials
b) Then, the type of authentication should allow these aforementioned operations.

I looked into the settings of one of my projects and point b) can only be achieved by a personal token. SSH deploy keys can't be used and there is no project-specific tokens. Sadly, personal tokens can't be narrowed down to given project, scopes are given globally. Per this stackoverflow post[5], the best strategy is to create a "machine user"[6]. Therefore, in order to please point a), we would need to create one machine user per Github project. I personally don't see this as a big issue, we do the same with Google Play.

:hwine, does the "machine user" strategy sounds good to you? If you wish, we can set up a quick call to provide you more context.

[1] Firefox Focus, now deprecated on Android.
[2] https://shipit.mozilla-releng.net/
[3] https://github.com/mozilla-mobile/fenix/releases, see not so many releases have APKs attached to it.
[4] https://developer.github.com/apps/building-oauth-apps/understanding-scopes-for-oauth-apps/#available-scopes
[5] https://stackoverflow.com/questions/26372417/github-oauth2-token-how-to-restrict-access-to-read-a-single-private-repo
[6] https://developer.github.com/v3/guides/managing-deploy-keys/#machine-users

Comment 1

[Johan Lorenzo [:jlorenzo]]

Attachment: [scriptworker-scripts] Create githubscript

Comment 2

[Johan Lorenzo [:jlorenzo]]

Chatted with Hal and Releng on Zoom. Let's create a user that has write access to repos. No need to create a user per project. We don't want to reuse existing machine users are used for scriptworker and have read-only access. Let's not grant them any write ones.

Hal told me we don't need to set up an RRA for now. Our conversation was good enough to keep going.

Comment 3

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: [fenix] Let taskcluster upload APKs to Github releases

Backfilling Johan's Fenix PR for consistency.

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: [cloudops-infra] add githubscript scriptworker deployment config

Comment 5

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: [k8s] add github configs in Kubernetes

Comment 6

[Mihai Tabara [:mtabara]⌚️GMT]

SOPS updated too, sans the machine token.

Comment 7

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1648785 - part 1: Enable github-publication to fenix (including jlorenzo's) r=mtabara

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/e6040d8500c3
part 1: Enable github-publication to fenix (including jlorenzo's) r=mtabara

Comment 9

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1648785 - add githubscript clients. r=#releng

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/ec8992892669
add githubscript clients. r=releng-reviewers,aki

Comment 11

[Mihai Tabara [:mtabara]⌚️GMT]

FYI: githubscript workers are now fully functioning.

Comment 12

[Johan Lorenzo [:jlorenzo]]

Attachment: [scriptworker-scripts] part 4 to 7: Fix integration issues

Comment 13

[Johan Lorenzo [:jlorenzo]]

Exciting news! Today's Fenix 82.0.0-beta1 was uploaded to Github releases[1].

One step I forgot: We had to give this account[2] the write access to the Fenix repo. I also had to provide more scope to the token that was generated on this account. Giving the repo scope[3] worked. Although it gives too broad permissions, in my opinion. I just restricted the token to repo:status, repo_deployment, and public_repo. I couldn't test this out because of a Chain of Trust issue I don't understand. I'll see how the next beta (scheduled next week) behaves.

Keeping bug open in the meantime.

[1] https://github.com/mozilla-mobile/fenix/releases/tag/v82.0.0-beta.1
[2] https://github.com/mozilla-release-automation-bot
[3] https://developer.github.com/apps/building-oauth-apps/understanding-scopes-for-oauth-apps/

Comment 14

[Johan Lorenzo [:jlorenzo]]

Attachment: [scriptworker-scripts] Githubscript should not alter target_commitish on a GitHub release

We had a hiccup when this went live, but this has been fixed with this patch.

I got the permissions sorted out too.

It's been working on Fenix 82 for a few betas now. The last example is b5:

• Green github-release task https://firefox-ci-tc.services.mozilla.com/tasks/CQ-eDE7wRCeD_Azop5HxWw. Curated logs:

2020-10-13 18:55:44,049 - githubscript.github - INFO - Release Firefox Beta 82.0.0-beta.5 already exists. Making sure it has the latest data...
2020-10-13 18:55:44,049 - githubscript.github - INFO - Field "is_prerelease" differ. Expected: True. Got: False
2020-10-13 18:55:44,049 - githubscript.github - INFO - Existing release will be updated.
2020-10-13 18:55:44,273 - githubscript.github - INFO - Making sure the latest artifacts are present...
2020-10-13 18:55:44,617 - githubscript.github - DEBUG - Existing release has the following artifacts attached: []
2020-10-13 18:55:44,618 - githubscript.github - INFO - Artifact "fenix-82.0.0-beta.5-arm64-v8a.apk" does not exist on Github. Uploading...
2020-10-13 18:55:44,618 - githubscript.github - DEBUG - Uploading artifact "fenix-82.0.0-beta.5-arm64-v8a.apk"...
2020-10-13 18:55:44,618 - githubscript.github - INFO - Artifact "fenix-82.0.0-beta.5-armeabi-v7a.apk" does not exist on Github. Uploading...
2020-10-13 18:55:44,618 - githubscript.github - DEBUG - Uploading artifact "fenix-82.0.0-beta.5-armeabi-v7a.apk"...
2020-10-13 18:55:44,619 - githubscript.github - INFO - Artifact "fenix-82.0.0-beta.5-x86.apk" does not exist on Github. Uploading...
2020-10-13 18:55:44,619 - githubscript.github - DEBUG - Uploading artifact "fenix-82.0.0-beta.5-x86.apk"...
2020-10-13 18:55:44,630 - githubscript.github - INFO - Artifact "fenix-82.0.0-beta.5-x86_64.apk" does not exist on Github. Uploading...
2020-10-13 18:55:44,631 - githubscript.github - DEBUG - Uploading artifact "fenix-82.0.0-beta.5-x86_64.apk"...
2020-10-13 18:55:49,451 - githubscript.github - INFO - All artifacts have been uploaded. Making sure everything went fine...
2020-10-13 18:55:52,041 - githubscript.github - INFO - Everything is sane!
2020-10-13 18:55:52,078 - githubscript.script - INFO - Done!
exit code: 0

• Github release was correctly updated and artifacts were automatically uploaded https://github.com/mozilla-mobile/fenix/releases/tag/v82.0.0-beta.5

I'm closing this bug then, we're all done!

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ee458e579195
[fenix] Bug 1648785 - Let taskcluster upload APKs to Github releases (https://github.com/mozilla-mobile/fenix/pull/12907)