Closed Bug 1731296 Opened 3 years ago Closed 2 years ago

Start publishing standalone taskgraph to Pypi

Categories

(Firefox Build System :: Task Configuration, task)

task

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: ahal, Assigned: ahal)

Details

Attachments

(1 file)

We want to make Gecko taskgraph start depending on standalone taskgraph so we can begin merging the two. All our infrastructure in mozilla-central for this (both vendoring as well as installing at runtime), assumes the package is on Pypi.

It would likely be easier for other consumers (like mobile repos) to depend on a version specifier rather than a revision anyway (though I don't see harm in supporting both methods for the time being).

This bug will simply track releasing to Pypi. It will entail:

  1. Adding version constraints to requirements.in. Currently there are no constraints which has been fine up until now. But if we start publishing on Pypi it means we are claiming to be compatible with every version of our dependencies (which I can guarantee is not the case :p). I'd propose we use compatible release on minor version as a starting point. E.g: json-e ~= 4.0. This means that we claim to be compatible with all 4.x versions of json-e (which should work as long as they have been good about following SemVer). The reason we don't want to pin versions here is that consumers of taskgraph may be installing it into an env that needs those packages for other purposes. E.g, if we pin requests and then some other package they use requires a different version of requests, their dependency resolution will fail (and they would get annoyed at us for pinning requests).

  2. Package name. In the future, I'd love for standalone taskgraph to have all of the "mozilla" bits stripped out of it and live in the taskcluster Github org. In anticipation of that split, I want a separate package name for the "Mozilla" flavour of taskgraph (to avoid confusion down the road). Unfortunately the name taskgraph is already claimed on Pypi, so we'll have to get a bit creative here. How about tc-taskgraph-mozilla for the Mozilla flavor and just tc-taskgraph for the future generic variant? Bikeshedding welcome.

  3. Shoring up setup.py configuration (classifiers, etc) and README. We'll still want to use the requirements.txt files for local development. We should add a CONTRIBUTING.md file to make this clear as well as document the publishing process. Also adding a setup.cfg to prefer wheels.

  4. Versioning. I'd like to use SemVer which means starting at 0.1.0 where minor version bumps can contain backwards incompatible changes up until 1.0.0. At which point we relegate backwards incompatible changes to major version bumps.

Also adding a setup.cfg to prefer wheels.

Turns out this is unnecessary. I was thinking of universal=1 but this is only needed if the package supports both Py2 and Py3.

Assignee: nobody → ahal
Status: NEW → ASSIGNED
Attachment #9243745 - Attachment description: Bug 1731296 - Prepare taskgraph to release to pypi and version bump to 0.1.0, r?#taskgraph-reviewers → Bug 1731296 - Prepare taskgraph to release to pypi and version bump to 1.0.0, r?#taskgraph-reviewers

Ahal:

I think the main thing to bikeshed on here is the pypi name (package name is still just taskgraph for now). See https://bugzilla.mozilla.org/show_bug.cgi?id=1731296#c0 for why I chose tc-taskgraph-mozilla, but am open to alternative suggestions.

Me:

Are we planning on the taskcluster-owned package and this package to hard-fork and continue to each be supported at that point?
If the plan is to consolidate, we may want to remove the -mozilla.

Ahal:

Good question, missed this my first time around. I was kind of thinking there would be two packages.. the generic one and then the mozilla one. However we could alternatively have a single package and then put the Mozilla-specific logic into a transforms.mozilla submodule. I could be convinced to go either way. Maybe the bug or a zoom call would be a better discussion forum? Or do you think an RFC would be better?

Moving discussion into the bug; we can move it out if wanted.
Yeah, I'm leaning towards not forking unless there's a strong reason to. If we have ways around Mozilla requirements diverging from the standalone Taskcluster taskgraph requirements, then we can stay un-forked; if there is some reason why we have to hard fork, we can hard fork at that point rather than pre-assume we need to fork.

I haven't looked at details, though, so I may be missing reasons that may push us towards hard-forking.

I notice we've been using "fork" as the word here, but what I was actually thinking we might do in the future is "split" the repo. So e.g, generator.py, task.py, taskgraph.py and all the "core" taskgraph logic might someday move into a tc-taskgraph package that tc-taskgraph-mozilla consumes. And I think the only reason to split is if we wanted to push tc-taskgraph as something consumers outside of Mozilla might want to use. But there are no such consumers now so that's not a very compelling argument.

The only reason we might want mozilla in the package name now is that if in the future we do decide to split the repo, it will make the transition easier (as the Mozilla-based consumers of tc-taskgraph-mozilla can simply continue using tc-taskgraph-mozilla).

But yeah, I definitely don't want to get back into the situation of maintaining two forks of taskgraph :p

All that to say, I'm fine with using tc-taskgraph as the package name as well. Was just trying to better explain my reasoning for tc-taskgraph-mozilla.

Yeah, I can see that, and I still think we may want to decide we need to split the repo, and then fork the pypi module. If that happens, I imagine the order could be something like

  1. we start using the tc-taskgraph or taskcluster-taskgraph module name on pypi
  2. we find real life reasons that suggest splitting the module into two may be required
  3. discussion
  4. we decide that the platform taskgraph and Mozilla taskgraph should fork
  5. we fork and create a new tc-taskgraph-mozilla or taskcluster-taskgraph-mozilla type module, with major version bumps
  6. we transfer the tc-taskgraph or taskcluster-taskgraph module to the taskcluster team, if we haven't already
  7. we update our repos to use the proper package for each module as needed.

But if we don't get past that decision at step 4, then we can continue to use a single package for everything.
Not sure if you want to start a conversation with the taskcluster team or dig for mozilla-specific things in taskgraph now, or if we want to assume that we have a chance of supporting a single package for the foreseeable future. We already have ways to override taskgraph behavior in taskcluster/ci/config.yml -- if we continue along that route, we might be able to support Mozilla-isms without including them in the taskgraph package.

Sounds good, and wfm! Only tradeoff here is that step 7 wouldn't be necessary, but that's not a big deal.

Do you have a preference between tc-taskgraph and taskcluster-taskgraph (or any other ideas)? I prefer the former myself, but not strongly.

(In case it wasn't clear, I was also proposing we do steps 2-4 as well. Only difference was what the package gets called in the meantime).

(In reply to Andrew Halberstadt [:ahal] from comment #7)

Sounds good, and wfm! Only tradeoff here is that step 7 wouldn't be necessary, but that's not a big deal.

Yeah, but we might not have to maintain 2 packages this way :)

Do you have a preference between tc-taskgraph and taskcluster-taskgraph (or any other ideas)? I prefer the former myself, but not strongly.

Slight preference for taskcluster-taskgraph, but not strongly, so we can either go with your preference or put it to a vote.
(https://pypi.org/search/?q=taskcluster has a number of taskcluster packages; https://pypi.org/search/?q=tc only has tc-admin afaict.)

(In case it wasn't clear, I was also proposing we do steps 2-4 as well. Only difference was what the package gets called in the meantime).

Yeah, but we end up with 2 package names even if we don't need to split.

(In reply to Aki Sasaki [:aki] (he/him) (UTC-6) from comment #8)

Slight preference for taskcluster-taskgraph, but not strongly, so we can either go with your preference or put it to a vote.
(https://pypi.org/search/?q=taskcluster has a number of taskcluster packages; https://pypi.org/search/?q=tc only has tc-admin afaict.)

Good point about the prior art, taskcluster-taskgraph wfm. I've resubmitted the patch.

Pushed by ahalberstadt@mozilla.com: https://hg.mozilla.org/ci/taskgraph/rev/9cf2af8c161c Prepare taskgraph to release to pypi and version bump to 1.0.0, r=aki

Published:
https://pypi.org/project/taskcluster-taskgraph/1.0.0/

Also invited Aki to be an owner.

This is long fixed.

Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: