Start publishing standalone taskgraph to Pypi
Categories
(Firefox Build System :: Task Configuration, task)
Tracking
(Not tracked)
People
(Reporter: ahal, Assigned: ahal)
Details
Attachments
(1 file)
Bug 1731296 - Prepare taskgraph to release to pypi and version bump to 1.0.0, r?#taskgraph-reviewers
48 bytes,
text/x-phabricator-request
|
Details | Review |
We want to make Gecko taskgraph start depending on standalone taskgraph so we can begin merging the two. All our infrastructure in mozilla-central for this (both vendoring as well as installing at runtime), assumes the package is on Pypi.
It would likely be easier for other consumers (like mobile repos) to depend on a version specifier rather than a revision anyway (though I don't see harm in supporting both methods for the time being).
This bug will simply track releasing to Pypi. It will entail:
-
Adding version constraints to
requirements.in
. Currently there are no constraints which has been fine up until now. But if we start publishing on Pypi it means we are claiming to be compatible with every version of our dependencies (which I can guarantee is not the case :p). I'd propose we use compatible release on minor version as a starting point. E.g:json-e ~= 4.0
. This means that we claim to be compatible with all 4.x versions ofjson-e
(which should work as long as they have been good about following SemVer). The reason we don't want to pin versions here is that consumers oftaskgraph
may be installing it into an env that needs those packages for other purposes. E.g, if we pinrequests
and then some other package they use requires a different version ofrequests
, their dependency resolution will fail (and they would get annoyed at us for pinningrequests
). -
Package name. In the future, I'd love for standalone taskgraph to have all of the "mozilla" bits stripped out of it and live in the
taskcluster
Github org. In anticipation of that split, I want a separate package name for the "Mozilla" flavour of taskgraph (to avoid confusion down the road). Unfortunately the nametaskgraph
is already claimed on Pypi, so we'll have to get a bit creative here. How abouttc-taskgraph-mozilla
for the Mozilla flavor and justtc-taskgraph
for the future generic variant? Bikeshedding welcome. -
Shoring up
setup.py
configuration (classifiers, etc) and README. We'll still want to use therequirements.txt
files for local development. We should add a CONTRIBUTING.md file to make this clear as well as document the publishing process. Also adding asetup.cfg
to prefer wheels. -
Versioning. I'd like to use SemVer which means starting at
0.1.0
where minor version bumps can contain backwards incompatible changes up until1.0.0
. At which point we relegate backwards incompatible changes to major version bumps.
Assignee | ||
Comment 1•3 years ago
|
||
Also adding a setup.cfg to prefer wheels.
Turns out this is unnecessary. I was thinking of universal=1
but this is only needed if the package supports both Py2 and Py3.
Assignee | ||
Comment 2•3 years ago
|
||
Updated•3 years ago
|
Updated•3 years ago
|
Comment 3•3 years ago
|
||
Ahal:
I think the main thing to bikeshed on here is the pypi name (package name is still just taskgraph for now). See https://bugzilla.mozilla.org/show_bug.cgi?id=1731296#c0 for why I chose tc-taskgraph-mozilla, but am open to alternative suggestions.
Me:
Are we planning on the taskcluster-owned package and this package to hard-fork and continue to each be supported at that point?
If the plan is to consolidate, we may want to remove the -mozilla.
Ahal:
Good question, missed this my first time around. I was kind of thinking there would be two packages.. the generic one and then the mozilla one. However we could alternatively have a single package and then put the Mozilla-specific logic into a transforms.mozilla submodule. I could be convinced to go either way. Maybe the bug or a zoom call would be a better discussion forum? Or do you think an RFC would be better?
Moving discussion into the bug; we can move it out if wanted.
Yeah, I'm leaning towards not forking unless there's a strong reason to. If we have ways around Mozilla requirements diverging from the standalone Taskcluster taskgraph requirements, then we can stay un-forked; if there is some reason why we have to hard fork, we can hard fork at that point rather than pre-assume we need to fork.
I haven't looked at details, though, so I may be missing reasons that may push us towards hard-forking.
Assignee | ||
Comment 4•3 years ago
•
|
||
I notice we've been using "fork" as the word here, but what I was actually thinking we might do in the future is "split" the repo. So e.g, generator.py
, task.py
, taskgraph.py
and all the "core" taskgraph logic might someday move into a tc-taskgraph
package that tc-taskgraph-mozilla
consumes. And I think the only reason to split is if we wanted to push tc-taskgraph
as something consumers outside of Mozilla might want to use. But there are no such consumers now so that's not a very compelling argument.
The only reason we might want mozilla
in the package name now is that if in the future we do decide to split the repo, it will make the transition easier (as the Mozilla-based consumers of tc-taskgraph-mozilla
can simply continue using tc-taskgraph-mozilla
).
But yeah, I definitely don't want to get back into the situation of maintaining two forks of taskgraph :p
Assignee | ||
Comment 5•3 years ago
|
||
All that to say, I'm fine with using tc-taskgraph
as the package name as well. Was just trying to better explain my reasoning for tc-taskgraph-mozilla
.
Comment 6•3 years ago
|
||
Yeah, I can see that, and I still think we may want to decide we need to split the repo, and then fork the pypi module. If that happens, I imagine the order could be something like
- we start using the
tc-taskgraph
ortaskcluster-taskgraph
module name on pypi - we find real life reasons that suggest splitting the module into two may be required
- discussion
- we decide that the platform taskgraph and Mozilla taskgraph should fork
- we fork and create a new
tc-taskgraph-mozilla
ortaskcluster-taskgraph-mozilla
type module, with major version bumps - we transfer the
tc-taskgraph
ortaskcluster-taskgraph
module to the taskcluster team, if we haven't already - we update our repos to use the proper package for each module as needed.
But if we don't get past that decision at step 4, then we can continue to use a single package for everything.
Not sure if you want to start a conversation with the taskcluster team or dig for mozilla-specific things in taskgraph now, or if we want to assume that we have a chance of supporting a single package for the foreseeable future. We already have ways to override taskgraph behavior in taskcluster/ci/config.yml
-- if we continue along that route, we might be able to support Mozilla-isms without including them in the taskgraph package.
Assignee | ||
Comment 7•3 years ago
•
|
||
Sounds good, and wfm! Only tradeoff here is that step 7 wouldn't be necessary, but that's not a big deal.
Do you have a preference between tc-taskgraph
and taskcluster-taskgraph
(or any other ideas)? I prefer the former myself, but not strongly.
(In case it wasn't clear, I was also proposing we do steps 2-4 as well. Only difference was what the package gets called in the meantime).
Comment 8•3 years ago
|
||
(In reply to Andrew Halberstadt [:ahal] from comment #7)
Sounds good, and wfm! Only tradeoff here is that step 7 wouldn't be necessary, but that's not a big deal.
Yeah, but we might not have to maintain 2 packages this way :)
Do you have a preference between
tc-taskgraph
andtaskcluster-taskgraph
(or any other ideas)? I prefer the former myself, but not strongly.
Slight preference for taskcluster-taskgraph, but not strongly, so we can either go with your preference or put it to a vote.
(https://pypi.org/search/?q=taskcluster has a number of taskcluster packages; https://pypi.org/search/?q=tc only has tc-admin afaict.)
(In case it wasn't clear, I was also proposing we do steps 2-4 as well. Only difference was what the package gets called in the meantime).
Yeah, but we end up with 2 package names even if we don't need to split.
Assignee | ||
Comment 9•3 years ago
|
||
(In reply to Aki Sasaki [:aki] (he/him) (UTC-6) from comment #8)
Slight preference for taskcluster-taskgraph, but not strongly, so we can either go with your preference or put it to a vote.
(https://pypi.org/search/?q=taskcluster has a number of taskcluster packages; https://pypi.org/search/?q=tc only has tc-admin afaict.)
Good point about the prior art, taskcluster-taskgraph
wfm. I've resubmitted the patch.
Comment 10•3 years ago
|
||
Assignee | ||
Comment 11•3 years ago
|
||
Published:
https://pypi.org/project/taskcluster-taskgraph/1.0.0/
Also invited Aki to be an owner.
Comment 12•2 years ago
|
||
This is long fixed.
Description
•