queue: Introduce "alias" artifact

NEW
Unassigned

Status

Taskcluster
Queue
3 years ago
3 months ago

People

(Reporter: jonasfj, Unassigned)

Tracking

Details

(Reporter)

Description

3 years ago
We currently create artifacts on the form:
    public/build/firefox....<version>.....tar.bz2

This is hard to use in automation. And for people doing bisection by finding
artifacts using the index, it's not all that fun to list artifacts and use
regexps to find the right artifact. As opposed to just knowing that the
build is always:
    public/build/firefox.tar.bz2

We can and **should** implement this using "reference" artifacts. Which allows
you to create an artifact that just redirects to a URL. So we can make the
artifact public/build/firefox.tar.bz2 as a "reference" that redirects to the
public/build/firefox....<version>.....tar.bz2 artifact.

But this pattern won't work nicely when the artifact referenced is private.
Because you'll then need to sign the URL after it's been redirected.

To resolve this I propose an alias artifact, which can reference any other
artifact within the same task. The alias artifact has the same expiration
as the artifact it aliases. But it changes the scope required to access
the artifact.

Example:
  Consider artifact:
              private-1/build.tar.gz
  requires scope queue:get-artifact:private-1/build.tar.gz
  
  Now if we create an alias artifact, for private-1/build.tar.gz
              private-2/alias.tar.gz
  It requires scope queue:get-artifact:private-2/alias.tar.gz 
 
I propose that we do not require any additional scopes access or create
the artifact "private-2/alias.tar.gz". Creating an alias artifact can
move a private artifact to "public/" or to another scope. But a task
is only allowed to create aliases for it's own artifacts, not artifacts
from other tasks. So it's safe to assume that the task that creates
an alias public/build.tar.gz for the artifact private/build.tar.gz
has access to the file build.tar.gz before it was uploaded.

Thoughts? Is there a use-case where this isn't the case and we risk
elevating scopes (or making artifacts public) that we don't want to?
(In reply to Jonas Finnemann Jensen (:jonasfj) from comment #0)
> This is hard to use in automation. And for people doing bisection by finding
> artifacts using the index, it's not all that fun to list artifacts and use
> regexps to find the right artifact. As opposed to just knowing that the
> build is always:
>     public/build/firefox.tar.bz2

What about non-linux? This would at least be firefox.zip on Windows and firefox.dmg on OSX. Or what if we want to change the file extension from .tar.bz2 to .tar.sz or something - would automation need to know that linux packages have different extensions before & after the switchover date?
Err, I was thinking .xz, not .sz. Maybe someone will make a .sz zipping algorithm to make my comment valid :)
Component: TaskCluster → Queue
Product: Testing → Taskcluster

Updated

2 years ago
See Also: → bug 1318574
Do you still want this?
Flags: needinfo?(jopsen)
(Reporter)

Comment 4

3 months ago
I think this might be neat for private logs, and things like that. No immediate demand.
Flags: needinfo?(jopsen)
Hi, I wanted to work on this bug. I went through artifacts.js (https://github.com/taskcluster/taskcluster-queue/blob/master/src/artifacts.js#L10) and have a rough idea of how an artifact is created.

I would like to start with public artifacts, since it is easier to implement. If I get it right, we have to create/update the artifact public/build/firefox.tar.bz2 everytime we receive a request for public/build/firefox..<ver>...tar.bz2 

1) Do we have to maintain a separate table for storing the reference ? (firefox..<ver..  is referenced by firefox)
2) Any resources other than the docs that I can read and would help ?
Flags: needinfo?(jopsen)
(Reporter)

Comment 6

3 months ago
There is still a few design issues to be resolved around alias artifacts and private artifacts.
Maybe this is not a good bug right now, (don't worry I'd be happy to help you find one).
Flags: needinfo?(jopsen)
You need to log in before you can comment on or make changes to this bug.