1733950 - Make `gecko_taskgraph` depend on standalone taskgraph

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Andrew Halberstadt [:ahal]]

As part of the effort to merge the two taskgraphs, we need to get gecko_taskgraph to depend on the standalone one. This will be a large undertaking, so the initial goal here is to set up the dependency and make gecko_taskgraph use something trivial from standalone (like a util file or something).

This way we can make iterative progress on merging the two, and slowly manage the conversion piece by piece. This slow conversion will be handled in follow-up bugs (probably one per "logical area" of code being converted).

Comment 1

[Andrew Halberstadt [:ahal]]

In order to set up the dependency we have three options:

1. Vendor taskcluster-taskgraph
2. Use a mach virtualenv (without lockfile support)
3. Use a mach virtualenv (block on lockfile support)

With vendoring the taskgraph code will actually be checked into tree. This could cause some confusion (especially during the transition) as there will be duplicates of most taskgraph files. Though a PRO here is that we could make an emergency fix directly to the vendored code without needing to cut a new taskgraph release (and make the fix upstream after things have quieted down).

Alternatively mach's virtualenv system grew the ability to install packages from pypi recently. Though lockfile support isn't available yet which means there is no way to verify package hashes at the moment. This will hopefully be implemented by end of H2.

Given the lack of hash verification and the ability to make quick fixes by modifying in-tree files, I'm leaning towards option 1 here.

Comment 2

[Andrew Halberstadt [:ahal]]

(Note that modifying files under third_party/python will cause a test failure, so if this ever happens, it should be very temporary)

Comment 3

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1733950 - Update vendored copies of 'taskcluster', 'taskcluster_urls' and 'slugid', r?aki

I'd like to vendor the 'taskcluster-taskgraph' pacakge, but it requires slugid

= 2.0. But in order to update that we also need to update 'taskcluster' and
'taskcluster_urls' as the currently vendored versions don't work with
'slugid>=2.0' (due to slugid returning str rather than bytes here).

Version changes are:

• taskcluster 6.0.0 -> 44.2.2
• taskcluster_urls 11.0.0 -> 13.0.1
• slugid 1.0.7 -> 2.0.0

Comment 4

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1733950 - [WILL SQUASH] Fixes for latest package updates, r?aki

Created for ease of reviewing. Will squash this to avoid broken revisions in vcs.

Depends on D127680

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3b6a21dbb1aa
Update vendored copies of 'taskcluster', 'taskcluster_urls' and 'slugid', r=aki

Comment 6

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1733950 - Vendor 'taskcluster-taskgraph==1.0.1' into mozilla-central, r?#taskgraph-reviewers

This will slowly start being used by 'taskcluster/gecko_taskgraph'.

Comment 7

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1733950 - Use upstream's 'taskgraph.util.yaml' module, r?#taskgraph-reviewers

Gecko taskgraph is now an official consumer of standalone taskgraph.

I chose 'util/yaml.py' since both copies were identical. I've also synced the
test for them, so no coverage is being lost.

Depends on D127872

Comment 8

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/3b6a21dbb1aa

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/68f4eadd4fa3
Vendor 'taskcluster-taskgraph==1.0.1' into mozilla-central, r=taskgraph-reviewers,aki
https://hg.mozilla.org/integration/autoland/rev/3242e697112d
Use upstream's 'taskgraph.util.yaml' module, r=taskgraph-reviewers,aki

Comment 10

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/68f4eadd4fa3
https://hg.mozilla.org/mozilla-central/rev/3242e697112d