Closed Bug 1733950 Opened 2 months ago Closed 2 months ago

Make `gecko_taskgraph` depend on standalone taskgraph

Categories

(Firefox Build System :: Task Configuration, task)

task

Tracking

(firefox95 fixed)

RESOLVED FIXED
95 Branch
Tracking Status
firefox95 --- fixed

People

(Reporter: ahal, Assigned: ahal)

References

Details

Attachments

(3 files, 1 obsolete file)

As part of the effort to merge the two taskgraphs, we need to get gecko_taskgraph to depend on the standalone one. This will be a large undertaking, so the initial goal here is to set up the dependency and make gecko_taskgraph use something trivial from standalone (like a util file or something).

This way we can make iterative progress on merging the two, and slowly manage the conversion piece by piece. This slow conversion will be handled in follow-up bugs (probably one per "logical area" of code being converted).

In order to set up the dependency we have three options:

  1. Vendor taskcluster-taskgraph
  2. Use a mach virtualenv (without lockfile support)
  3. Use a mach virtualenv (block on lockfile support)

With vendoring the taskgraph code will actually be checked into tree. This could cause some confusion (especially during the transition) as there will be duplicates of most taskgraph files. Though a PRO here is that we could make an emergency fix directly to the vendored code without needing to cut a new taskgraph release (and make the fix upstream after things have quieted down).

Alternatively mach's virtualenv system grew the ability to install packages from pypi recently. Though lockfile support isn't available yet which means there is no way to verify package hashes at the moment. This will hopefully be implemented by end of H2.

Given the lack of hash verification and the ability to make quick fixes by modifying in-tree files, I'm leaning towards option 1 here.

(Note that modifying files under third_party/python will cause a test failure, so if this ever happens, it should be very temporary)

I'd like to vendor the 'taskcluster-taskgraph' pacakge, but it requires slugid

= 2.0. But in order to update that we also need to update 'taskcluster' and
'taskcluster_urls' as the currently vendored versions don't work with
'slugid>=2.0' (due to slugid returning str rather than bytes here).

Version changes are:

  • taskcluster 6.0.0 -> 44.2.2
  • taskcluster_urls 11.0.0 -> 13.0.1
  • slugid 1.0.7 -> 2.0.0

Created for ease of reviewing. Will squash this to avoid broken revisions in vcs.

Depends on D127680

Attachment #9244573 - Attachment is obsolete: true
Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/3b6a21dbb1aa
Update vendored copies of 'taskcluster', 'taskcluster_urls' and 'slugid', r=aki

This will slowly start being used by 'taskcluster/gecko_taskgraph'.

Gecko taskgraph is now an official consumer of standalone taskgraph.

I chose 'util/yaml.py' since both copies were identical. I've also synced the
test for them, so no coverage is being lost.

Depends on D127872

Status: ASSIGNED → RESOLVED
Closed: 2 months ago
Resolution: --- → FIXED
Target Milestone: --- → 95 Branch
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/68f4eadd4fa3
Vendor 'taskcluster-taskgraph==1.0.1' into mozilla-central, r=taskgraph-reviewers,aki
https://hg.mozilla.org/integration/autoland/rev/3242e697112d
Use upstream's 'taskgraph.util.yaml' module, r=taskgraph-reviewers,aki
You need to log in before you can comment on or make changes to this bug.