1419881 - Implement static URLs that go to latest artifact with a varying name

Status: REOPENED

(Taskcluster :: Workers, enhancement, P5)

Description

[Steve Fink [:sfink] [:s:]]

Here's my use case: I want to provide a URL that goes to the latest version of an artifact produced by a taskcluster job (spidermonkey-sm-package), but the artifact has a version number in its name.

So the index URL https://index.taskcluster.net/v1/task/gecko.v2.mozilla-esr52.latest.firefox.sm-package-opt/artifacts/public/build/mozjs-52.4.0.tar.bz2 correctly goes to the latest build of mozjs-52.4.0.tar.bz2, but sadly that link is now a 404 because it hasn't been built in quite a while, as the current version is mozjs-52.5.1.tar.bz2.

Obviously, an easy way to solve this would be to remove the version number from the name. But that seems like it would lead to a lot of confusion.

One way of doing this would be to allow creating a "symlink" artifact, where the job says to create an artifact named mozjs.tar.bz2 which is actually mozjs-52.5.1.tar.bz2, and when you retrieve it, you get a HTTP 301 to mozjs-52.5.1. Or if the 301 is a bad idea, it could go to a simple file containing only a link to the mozjs-52.5.1 url.

Alternatively, the index could index artifacts itself, so the URL would be something like https://index.taskcluster.net/v1/task/gecko.v2.mozilla-esr52.latest.firefox.sm-package-opt/named-artifacts/mozjs.tar.bz2 (the exact keyword is a placeholder). I think it already does a 301 or 302 when you go to an index.t.n url?

Note that this isn't meant to be a full-on release mechanism. We have another location to store things we've properly blessed as releases that is manually populated. But we don't actually have real spidermonkey releases, and so pointing to the latest version on a particular branch would give people a way to have the latest security fixes in the absence of someone properly handling release management for SM.

Suggestions?

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

There is such a thing as a reference artifact
  https://docs.taskcluster.net/reference/platform/taskcluster-queue/references/api#createArtifact
so you could potentially implement this with such a reference.  However, you would need to implement support in the workers for that.  I wouldn't suggest hacking on docker-worker these days -- it's more or less in maintenance mode -- but this could perhaps be implemented in taskcluster-worker.

Comment 2

[Chris Cooper [:coop] (he/him)]

Found in triage.

Jonas asserts that the right thing to do is to name the file appropriately, and possibly add a README file alongside. While the "latest" links are a very historical Mozilla thing, they may be an anachronism from the ftp era.

Our new Object service model[1] could make this easier by addressing some of the security concerns, but this aspect is low priority.

1. https://github.com/taskcluster/taskcluster-rfcs/pull/116

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

I think Jonas is right, and also reference artifacts are a good solution here.  So this would be a good project for taskcluster-worker: adding a way to declare a reference artifact in a task payload.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Pete, would adding this feature to g-w make a good mentored bug?

Comment 5

[Pete Moore [:pmoore][:pete]]

I think there may be a better solution.

The artifact name serves two purposes:

1) It acts as an identifier to guard access to the artifact
2) It serves as a de-facto filename for the artifact content

Please note, when declaring an artifact in a worker task payload, the artifact name and artifact path may be different. For example, the an artifact may be located at "dist/firefox44.exe" but have artifact name "public/build/firefox".

When we serve an artifact, there is no reason that we could not serve it with a filename that is different to the artifact name.

I particularly dislike that artifact name implies artifact filename, since two distinct tasks could produce an artifact with the same filename (e.g. "build.tar.gz"), and given that many artifacts are published as the result of a directory artifact declaration in the task payload, artifact names can collide for completely different tasks. Yet only the artifact name is used to distinguish which clients should be able to retrieve the artifact.

Therefore I would be in favour of introducing a change whereby a different filename can be associated to an artifact than the artifact name, such that when it is served, it is served with the artifact filename rather than the artifact name.

In other words, you should be able to call e.g.

https://index.taskcluster.net/v1/task/gecko.v2.mozilla-esr52.latest.firefox.sm-package-opt/artifacts/public/build/mozjs.tar.bz2

but be redirected to a url such that you are served a file called mozjs-52.4.0.tar.bz2

This would all need to be backwardly compatible, so unless you specify you want to be served the filename, you'd still get the filename as the artifact name.

This probably warrants a separate RFC, but this is probably somewhat overdue anyway, due to artifact name being already overloaded, as explained at the start of this comment.

Comment 6

[John Ford [:jhford] CET/CEST Berlin Time]

We could use the content-disposition header to have "/v1/artifacts/public/build/firefox" automatically download "firefox-52.exe" when a default filename is not specified (e.g. curl -L would save to firefox-52.exe).  This would allow a response hander to see the value of the true filename using the HTTP spec as well.

Comment 7

[Steve Fink [:sfink] [:s:]]

Comment 5 makes a lot of sense to me, and would probably open up some additional use cases. Artifacts ought to be artifacts, not simply files, and so should have an identity determined by a name rather than a filename. Even if the two are often the same. It does mean that there are two types of URLs here, artifact URLs and file URLs. But heck, URLs point to resources, and both can be considered to be resources.

Comment 8

[Chris Cooper [:coop] (he/him)]

This is still valid, but relies on two potential improvements, the Object Service and Index v2, that we may not get to for a while.

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

(closed in bulk closure of old P5 bugs, representing that they are unlikely to ever be worked on)

Comment 10

[Pete Moore [:pmoore][:pete]]

Reopening inactive bugs, because they may still need attention. Historically, inactive bugs were closed, but this hides the fact there are genuine issues which have not been resolved.