1986773 - Create Gecko Decision tasks for both hgmo and Github

Status: RESOLVED FIXED

(Release Engineering :: Firefox-CI Administration, task)

Description

[Andrew Halberstadt [:ahal]]

Because the Gecko CI migration will be very large, we'll want to do the migration piece-meal rather than all at once. In order to accomplish this, we'll need Decision tasks that run both via the traditional hgmo pulse events, as well as the taskcluster-github events.

Since this support will only be needed transitionally, I plan to setup .taskcluster.yml to be completely bi-modal. I.e, I will create two entirely separate task definitions that don't share any logic with one another.

Comment 1

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1986773 - [decision] Use VCS agnostic approach to obtain commit message, r?#taskgraph-reviewers

I don't think catching UnicodeDecodeError is necessary, but this exactly
matches upstream Taskgraph, so will make consolidating a bit easier in
the future.

Comment 2

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1986773 - [decision] Don't set hg specific parameters if running from git, r?#taskgraph-reviewers

Comment 3

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1986773 - [decision] Remove repository check in action builder, r?#taskgraph-reviewers

We'll soon be supporting running Decision tasks against Github in
addition to hg.mozilla.org. Also upstream Taskgraph doesn't have this
check at all, so just remove it rather than adjusting the regex.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/98465edd8259
https://hg.mozilla.org/integration/autoland/rev/3c21f49ee1dc
[decision] Remove repository check in action builder, r=taskgraph-reviewers,releng-reviewers,bhearsum
https://github.com/mozilla-firefox/firefox/commit/8aac306929d0
https://hg.mozilla.org/integration/autoland/rev/bb06b415873a
[decision] Use VCS agnostic approach to obtain commit message, r=taskgraph-reviewers,bhearsum
https://github.com/mozilla-firefox/firefox/commit/4cd8b5f6fdf7
https://hg.mozilla.org/integration/autoland/rev/ba228f2572d4
[decision] Don't set hg specific parameters if running from git, r=taskgraph-reviewers,bhearsum

Comment 5

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/3c21f49ee1dc
https://hg.mozilla.org/mozilla-central/rev/bb06b415873a
https://hg.mozilla.org/mozilla-central/rev/ba228f2572d4

Comment 6

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1986773 - [mozversioncontrol] Fetch git remotes more robustly, r?#releng-reviewers

It's possible for output of git remote -v to both contain less than
three parts and more than three parts.

Comment 7

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1986773 - [decision] Add target_tasks filtering for Github projects, r?#taskgraph-reviewers

This change:

1. Adds the "firefox" and "staging-firefox" projects
2. Adds filter_for_git_branch to the default filters (no-op on hg)
3. Implements a new filter_for_repo_type in the default filters
   (defaults to ["hg"]). Until tasks start setting this attribute to
   include "git", nothing will run on Github based graphs.

Comment 8

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1986773 - [ci] Stop testing third party 'attrmatch' function, r?#taskgraph-reviewers

Comment 9

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1986773 - [ci] Support upstream 'run-task' in job transforms, r?#taskgraph-reviewers

The Gecko version of run-task is specific to Mercurial. Now that we're
going to start running Gecko CI against Github repos, they will be using
the upstream version of run-task. This version of run-task expects a
REPOSITORIES env to be set.

This patch makes the job transforms match the upstream run ones a
bit more closely, which involves defining taskgraph.repositories in
config.yml.

Additionally this sets a few extra env variables that the run
transforms were setting, just for additional parity.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/e62372c880c9
https://hg.mozilla.org/integration/autoland/rev/7e269c982f59
[mozversioncontrol] Fetch git remotes more robustly, r=releng-reviewers,bhearsum
https://github.com/mozilla-firefox/firefox/commit/0afa1d31ba70
https://hg.mozilla.org/integration/autoland/rev/f400df67bf09
[decision] Add target_tasks filtering for Github projects, r=taskgraph-reviewers,bhearsum
https://github.com/mozilla-firefox/firefox/commit/c16d30d0e6c9
https://hg.mozilla.org/integration/autoland/rev/3d3d7f96e4df
[ci] Stop testing third party 'attrmatch' function, r=taskgraph-reviewers,jmaher

Comment 11

[amarc]

https://hg.mozilla.org/mozilla-central/rev/7e269c982f59
https://hg.mozilla.org/mozilla-central/rev/f400df67bf09
https://hg.mozilla.org/mozilla-central/rev/3d3d7f96e4df

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/9f13ca7bbe08
https://hg.mozilla.org/integration/autoland/rev/7db8f510ff7d
[ci] Support upstream 'run-task' in job transforms, r=taskgraph-reviewers,bhearsum

Comment 13

[agoloman]

https://hg.mozilla.org/mozilla-central/rev/7db8f510ff7d

Comment 14

[Toby Pilling [:tobyp]]

Hey there folks,
Do you have any advice on how to keep Thunderbird happy during these changes?

Comment 15

[Andrew Halberstadt [:ahal]]

Hey Toby, we gecko_taskgraph developers tend to break comm a lot, personally I can't even count how many times I've done it. The issue is that comm is basically using gecko_taskgraph as an unpinned dependency. At the same time, us gecko_taskgraph developers have no easy way to test our changes (and even if we did, worrying about keeping comm green isn't something we have the time or mandate to do).

I highly recommend that comm developers find a way to refactor their CI setup such that comm no longer depends on gecko_taskgraph this way. I haven't thought through everything, but here is what I would do if I were you:

1. Switch your Decision task to use the Decision image from Taskgraph
2. Invoke Taskgraph directly rather than going through mach
3. Hard fork any transforms / logic you need from gecko_taskgraph into comm_taskgraph and stop importing gecko_taskgraph entirely.

I'll be frank, this suggestion is not a trivial amount of work and there are probably all sorts of problems that will arise as you work through it. But until someone does the hard work of fixing comm's CI setup, we're going to carry on breaking you over and over :(

There is a bit of a time factor here as well, because once we're fully switched over to Git, we're going to be dropping support for Mercurial. This means that you'll either need to migrate to Git at the same time, or you'll be hard forking out of necessity anyway. So imo might as well do it ahead of time while there's not a ton of time pressure.

If this is something you want to tackle, I'm happy to provide guidance.

Comment 16

[Corey Bryant]

Hey Andrew,

Thanks for the insight. I'm not sure how much it would benefit us to hard fork gecko_taskgraph. That gets us into a larger issue of maintaining a fork that needs to keep up with upstream Firefox.

Thunderbird is built as an extension of Firefox. We depend on source code, CI, mach commands, etc from Firefox. For better or worse, we're a tightly integrated downstream user of Firefox in several areas, gecko_taskgraph being one of them. The dependency of Thunderbird on Firefox is all an unpinned dependency. Our sheriffs and others spend a lot of time fixing breakage when we pin to new versions of Thunderbird, and that's just the nature of the beast.

We can handle breakages. You don't need to keep comm green. However, I would like to ask that you consider us downstream when making code changes.

We are in the early stages of git migration. Hopefully you won't be dropping support for Mercurial before we migrate.

Thanks,
Corey

Comment 17

[Andrew Halberstadt [:ahal]]

That might be true overall, but I must insist you are mistaken when it comes to gecko_taskgraph, it doesn't have to be like this :)

What you should be depending on are:
https://github.com/taskcluster/taskgraph
https://github.com/mozilla-releng/mozilla-taskgraph

There we take great pains to follow semantic versioning and avoid breaking consumers. Those repos are designed to be used across projects, and you can pin their dependencies to a specific version, so you can deal with breakage on your own terms.

I've scanned the tasks in your repo, and I see you use a lot of similar kinds and are directly using a lot of the gecko_taskgraph transforms, especially for release tasks.

The piece you might be missing is that the grand vision is for mozilla-taskgraph to house all the common release type tasks, like signing, beetmover, balrog, etc. These are probably less Gecko-specific than you think. Other projects like mozilla-vpn-client also have transforms that deal with stuff like this. So my suggestion if you want to both stop depending on Gecko and not be responsible for maintaining it in the long term? Move it into mozilla-taskgraph instead of hard forking. I think you'll find I'll be eager to help, as you'll be helping us create transforms that benefit everyone.

I totally get that this isn't trivial and you have very limited resources to work on it. The good news, is that this is something that can be done gradually over time. So my goal for now is to try and convince you that this is the right approach and help you get started down this path. Then over the course of years, we slowly move logic into either comm_taskgraph or mozilla-taskgraph (or even the core taskgraph if it's generic enough).

p.s You can still use transforms from gecko_taskgraph, even if you don't use it as the entrypoint (i.e follow steps 1 and 2 that I outlined in comment 15). This alone should drastically cut back on the amount of breakage you receive from us. So even if you don't want to do any forking, I recommend you at least take those steps.

Comment 18

[Andrew Halberstadt [:ahal]]

We are in the early stages of git migration. Hopefully you won't be dropping support for Mercurial before we migrate.

We'll definitely keep supporting Mercurial in taskcluster/taskgraph until the last fxci repo is migrated off of Mercurial. And I think we can support it for a "reasonable" amount of time in gecko_taskgraph as well (I don't know what reasonable is yet). But if we ever get to a point where Gecko is fully off of Mercurial but comm isn't and there's no end in sight, then at that point hard forking Mercurial only stuff into comm taskgraph feels like it'll be the way to go.

Comment 19

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1986773 - Create a Decision task that gets triggered by Github events, r?#releng-reviewers

Comment 20

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1986773 - [ci] Add test parameters for push to firefox-main Github repo, r?#releng-reviewers

Comment 21

[Toby Pilling [:tobyp]]

Wow, thanks so much for taking the time to help on this. Truly appreciated

Comment 22

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1986773 - Fix misnamed argument in Git-based Decision tasks, r?#releng-reviewers

Comment 23

[Corey Bryant]

I've scanned the tasks in your repo, and I see you use a lot of similar kinds and are directly using a lot of the gecko_taskgraph transforms, especially for release tasks.

Yes, we do that for code re-use since we're built as an extension of firefox and use a lot of the same tasks.

The piece you might be missing is that the grand vision is for mozilla-taskgraph to house all the common release type tasks, like signing, beetmover, balrog, etc. These are probably less Gecko-specific than you think. Other projects like mozilla-vpn-client also have transforms that deal with stuff like this. So my suggestion if you want to both stop depending on Gecko and not be responsible for maintaining it in the long term? Move it into mozilla-taskgraph instead of hard forking. I think you'll find I'll be eager to help, as you'll be helping us create transforms that benefit everyone.

mozilla-taskgraph seems like a good approach for code re-use among projects. Are you in the process or planning to move tasks from gecko-taskgraph to mozilla-taskgraph, or are you suggesting we should do that?

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/801c99d92948
https://hg.mozilla.org/integration/autoland/rev/3bc325b8b7d2
Fix misnamed argument in Git-based Decision tasks, r=releng-reviewers,taskgraph-reviewers,bhearsum
https://github.com/mozilla-firefox/firefox/commit/0192d74fa3cf
https://hg.mozilla.org/integration/autoland/rev/2bba9b331b0c
Create a Decision task that gets triggered by Github events, r=releng-reviewers,bhearsum
https://github.com/mozilla-firefox/firefox/commit/8b2797900aea
https://hg.mozilla.org/integration/autoland/rev/d344424cf9c9
[ci] Add test parameters for push to firefox-main Github repo, r=releng-reviewers,bhearsum

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/366c806e01f4
https://hg.mozilla.org/integration/autoland/rev/75d400f79785
Revert "Bug 1986773 - [ci] Add test parameters for push to firefox-main Github repo, r=releng-reviewers,bhearsum" for causing py failures.

Comment 26

[Sandor Molnar[:smolnar]]

Backed out Add test parameters changeset for causing py failures

Backout link: https://hg.mozilla.org/integration/autoland/rev/75d400f797850981581308f10853f941b11aef84

Push with failures

Failure log -> taskcluster/test/test_generate_params.py::test_generate_graphs[firefox-main-onpush] TEST-UNEXPECTED-FAIL

Failure log -> Error running mach: mach taskgraph target --fast -p taskcluster/test/params --force-local-files-changed -o /builds/worker/artifacts/diffs/diff.txt

Comment 27

[Andrew Halberstadt [:ahal]]

I wasn't able to reproduce locally, the commit that caused the failure isn't necessary at this time, so I'll likely abandon the revision for now and attempt to add this parameter file again once we have real ones we can pull from firefox-main using the update script.

Comment 28

[Andrew Halberstadt [:ahal]]

mozilla-taskgraph seems like a good approach for code re-use among projects. Are you in the process or planning to move tasks from gecko-taskgraph to mozilla-taskgraph, or are you suggesting we should do that?

Sorry missed this. I'm suggesting you do that. This is something I push on from time to time when I'm working on something and see an opportunity to advance it. But it's never going to be a priority from leadership, so there's only so much time I can spend doing it unfortunately :(