1635488 - add git and github support in treescript

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, task)

Description

[Mihai Tabara [:mtabara]⌚️GMT]

The goal here is to get Fenix on ship-it. For that to happen, there are two things that need to happen:

1. Fenix to be able to schedule an action task that generates a graph, which creates a Github release. Other sanity checks around version found in-tree and bump the version once the release is done.

Expected Github operations:

• github push (to master if there is no branch-protection) or pull-request (if there is)
• github create release

2. Bumping the version number in-tree requires manuevring git repos. For this we need to extend treescript to understand git as well.

Expected supported git operations:

• pull
• push
• tag
• count commits (used for sanity checks to compare local git commits to expected ones to be pushed in a release)
• (optional) sign commit

For git operations, we're leaning towards using subprocesses and call the underlying git modules directly for a handful of reasons:
a) easier to update the git within the Docker images should we want to
b) more maintenance expertise within RelEng when it comes to vanilla git commands, rather than any of the APIs (https://gitpython.readthedocs.io/en/stable/)
c) (speculation) speed - using native tools is likely faster than using the Python wrapper around them, since Python runs in a single process.

Comment 1

[Simon Fraser [:sfraser] ⌚️GMT]

https://pygithub.readthedocs.io/en/latest/index.html may be worth investigating, too. Example:

>>> repo = g.get_repo("PyGithub/PyGithub")
>>> contents = repo.get_contents("test.txt", ref="test")
>>> repo.update_file(contents.path, "more tests", "more tests", contents.sha, branch="test")
{'commit': Commit(sha="b06e05400afd6baee13fff74e38553d135dca7dc"), 'content': ContentFile(path="test.txt")}

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

Coordinated with Johan again on this topic to align our strategy, dropping some thoughts.

1. In the glorious future, Ship-it will replace Github releases so we won't have to rely on them anymore. That said, we can solely rely on the https://github.com/mozilla-mobile/fenix/blob/master/version.txt, similarly to what we do on gecko trees. That file shall be read by the Ship-it frontend and updated via treescript for the future Fenix releases.

2. For a while, we'll have both Github releases and Ship-it at the same time, which will probably be a bit confusing since we'll have to keep that file manually at sync. But once the migration is over, we should be good to go.

3. RelMan/Mobile go on Ship-it, they choose the revision and the version number for the Fenix release and trigger the release. Under the hood, an action task is being scheduled which schedules the release graph. Alongside them, there will be a treescript task too. Given that at the time of action-task creation we already have the version, we can already bake the version and nextVersion within the version-bump's task payload.

Version bump treescript task will look something like:
task.payload: version, nextVersion

logic under the hood in treescript:
a) pull https://github.com/mozilla-mobile/fenix/,
b) adjust the version value from version to nextVersion
c) add the files
d) commit
e) push to Github (push if ssh key, pull-request if branch protection, etc)

These is the basic backbone to begin with. We could inject some sanity checks along the way including but not limited to:
i) between a) and b) ensure the version in the payload corresponds to the one in the actual version.txt file in the repo
ii) between d) and e) to ensure git out (or wathever the equivalent is for hg out) pushes only one commit and nothing more
iii) import/export the commit and inject an Autograph signing task in between
iv) etc

Comment 3

[Mihai Tabara [:mtabara]⌚️GMT]

Note to self:

• confirmed with :hwine that we can use a bot, within a more restrictive group, that can automatically push to master. That entails:
  a) generate the ssh key for that bot and keep that in SOPS
  b) carefully restrict the actions possible via the Github configs and have SecOps sign-off that

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

Callek and myself pair-programmed on this today, with some great results:

• we have a proof-of-concept (version bump, sanity checks, github interactions, sanity checks, possible extensions) for both https://gitpython.readthedocs.io/en/stable/ and https://pygithub.readthedocs.io/en/latest/index.html
• pasting in a bit the results for both
• decision going forward is to use https://gitpython.readthedocs.io/en/stable/ for the reasons that are following in the comments

Comment 5

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: https://gitpython.readthedocs.io/ POC

Pros

• straightforward to implement, we already have MVP
• can check against race-conditions because of avoiding force-pushing
• tags are easy - https://gitpython.readthedocs.io/en/stable/reference.html?highlight=push#git.repo.base.Repo.create_tag
• more clarity in operations
• possibility of adding more sanity checks because of the verbose nature of the code
• able to push via ssh-key

Cons

• PRs difficult to implement as they are not natively supported
• we need to implement sanity checks to ensure things are as expected, as opposed
  to having them already there via the Github API wrapper
• less guarantees for idempotency - doable but needs extra work

Results: https://github.com/MihaiTabara/fenix/commit/7b6b8c40e7d5da10b7d73a10305a6ae9f044d909#diff-5fb01c6f8b54d96c62af3415c51ffad3

Comment 6

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: https://pygithub.readthedocs.io/ POC

Pros

• easy to implement, we already have MVP
• can check against race-conditions because of the sha
• PRs are easy to add if we need this - https://pygithub.readthedocs.io/en/latest/examples/PullRequest.html#create-a-new-pull-request
• tags are easy - https://pygithub.readthedocs.io/en/latest/github_objects/Tag.html?highlight=tag#github.Tag.Tag

Cons

• confusing non-granular write access for the Github token or Github user
• possible rate-limit future concern https://developer.github.com/v3/#rate-limiting
• side-effects of functions are not aparent at callsite
• possible deprecation of the Github APIs in the future versions
• only one file per changeset/commit via update_file
• harder to extend for future requests

Results: https://github.com/MihaiTabara/fenix/commit/2b79439542114e81145ee1de61126006631eca59#diff-5fb01c6f8b54d96c62af3415c51ffad3

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

Un-assigning this since it's a TEAM effort.

Comment 8

[Mihai Tabara [:mtabara]⌚️GMT]

• same treescript, but a different COT product for safety of COT
• we need to guard with scopes per branch/project since in the mobile world we can have more projects using this task, unlike gecko where we have per-level scopes

We can carve out two pieces of work here:

• treescript backbone to make room for git world
• the actual git module with operations for all of these (basically an extension of https://bugzilla.mozilla.org/attachment.cgi?id=9150567&action=edit )

Comment 9

[Mihai Tabara [:mtabara]⌚️GMT]

Note to self: Johan pointed out that we need to bump https://github.com/mozilla-mobile/fenix/pull/12907/files#diff-e5307118f57349f9bf5d4be8b0d92aef as well when we do the version bump.

Comment 10

[Mihai Tabara [:mtabara]⌚️GMT]

Had a chat with Johan on this today.

Places we need to touch:

• taskgraph - Fenix standalone version bump tasks + git tags since we're here
• instances - all the SOPS, cloudops-infra
• treescript git work + docker configuration + worker.yml
• ciadmin - scopes administration

Comment 11

[Johan Lorenzo [:jlorenzo]]

Tracked in JIRA as https://jira.mozilla.com/browse/RELENG-89. Chatted with :mtabara, I'm going to handle this bug.

Comment 12

[Johan Lorenzo [:jlorenzo]]

Attachment: [scriptworker-script] part 1: [treescript] Expose commit() function in mercurial

Comment 13

[Johan Lorenzo [:jlorenzo]]

Attachment: [scriptworker-scripts] part 2 & 3: Dynamically choose vcs module

Comment 14

[Johan Lorenzo [:jlorenzo]]

Attachment: [scriptworker-scripts] part 4: Support version bumps on github repos

Comment 15

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: [fenix] add taskgraph version bump task

Comment 16

[Johan Lorenzo [:jlorenzo]]

Attachment: [k8s-autoscale] Add mobile treescript workers

Comment 17

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1635488 - part 1: Create new TC clients for the mobile treescript instances r=mtabara

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/20818c90d9eb
part 1: Create new TC clients for the mobile treescript instances r=mtabara

Comment 19

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1635488 - Bump attrs (and other deps) so that fenix can use mozilla-version r=#releng-reviewers

mozilla-version requires attrs>=19.2.0 https://github.com/mozilla-releng/mozilla-version/pull/73

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/taskgraph/rev/2b2622598df0
Bump attrs (and other deps) so that fenix can use mozilla-version r=releng-reviewers,bhearsum

Comment 21

[Johan Lorenzo [:jlorenzo]]

Attachment: [scriptworker-scripts] part 7 and 8: Fixup Thunderbird

Comment 22

[Johan Lorenzo [:jlorenzo]]

Attachment: [fenix] part 2: Add missing branch in version bump payload

Comment 23

[Johan Lorenzo [:jlorenzo]]

Attachment: [scriptworker-scripts] part 9 and 10: Fix git branch that cannot be checked out

Comment 24

[Johan Lorenzo [:jlorenzo]]

Attachment: [scriptworker-scripts] part 11: Do not use the `branch@{u}` syntax

Comment 25

[Johan Lorenzo [:jlorenzo]]

Attachment: [scriptworker-scripts] part 12: Set upstream branch at push time

Comment 26

[Johan Lorenzo [:jlorenzo]]

Attachment: [scriptworker-scripts] part 13: Fix set-upstream flag

Comment 27

[Johan Lorenzo [:jlorenzo]]

Attachment: [scriptworker-scripts] part 14: Fix branch parameter

Comment 28

[Johan Lorenzo [:jlorenzo]]

This is now fixed and has been deployed to production.

2020-12-01 11:02:21,379 - git.cmd - DEBUG - Popen(['git', 'push', '--porcelain', '--set-upstream', '--verbose', 'origin', 'relbot/AC-67.0.6'], cwd=/app/workdir/src, universal_newlines=True, shell=None, istream=None)
2020-12-01 11:02:25,596 - treescript.git - INFO - Push done succesfully!
2020-12-01 11:02:25,598 - treescript.git - INFO - Resetting repo state to match upstream's...
2020-12-01 11:02:25,598 - git.cmd - DEBUG - Popen(['git', 'reset', '--hard', 'origin/relbot/AC-67.0.6', '--'], cwd=/app/workdir/src, universal_newlines=False, shell=None, istream=None)
2020-12-01 11:02:25,623 - git.cmd - DEBUG - Popen(['git', 'clean', '-fdx'], cwd=/app/workdir/src, universal_newlines=False, shell=None, istream=None)
2020-12-01 11:02:25,637 - treescript.git - INFO - Repo state reset.
2020-12-01 11:02:25,637 - treescript.script - INFO - Done!
exit code: 0

https://firefox-ci-tc.services.mozilla.com/tasks/eI8dFWdETrW-vYSu2SMUqA/runs/1/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FeI8dFWdETrW-vYSu2SMUqA%2Fruns%2F1%2Fartifacts%2Fpublic%2Flogs%2Flive_backing.log

The branch was wrong, but it's because of shipit. I'm going to work on this in another bug. This means this bug is now done \o/

Comment 30

[Pulsebot]

Pushed by ffxbld-merge:
https://hg.mozilla.org/releases/mozilla-beta/rev/94bc53ab64b6
[fenix] Bug 1635488 - add Fenix version-bump task. (https://github.com/mozilla-mobile/fenix/pull/16361)
https://hg.mozilla.org/releases/mozilla-beta/rev/916c09e43f29
[fenix] Bug 1635488 - part 2: Add missing branch in version bump payload (https://github.com/mozilla-mobile/fenix/pull/16650)