1435403 - artifact builds should work on project and release branches

Status: NEW

(Firefox Build System :: General, defect)

Description

[Robert Helmer [:rhelmer]]

The mozilla-release repo is currently not in the list of CANDIDATE_TREES, so artifact builds do not work without modification:

https://searchfox.org/mozilla-central/rev/eeb7190f9ad6f1a846cd6df09986325b3f2c3117/python/mozbuild/mozbuild/artifacts.py#119

Attempting to push to try will yield:

 Exception: Could not find any candidate pushheads in the last 50 revisions.

There are many use cases for artifact builds on release, for instance the ability to test off-train releases like system add-on updates, shield studies, etc. as well as for quick feedback on dot-releases that are JS-only.

Comment 1

[Gregory Szorc [:gps]]

A big blocker to this is the extra overhead required to find artifacts. The way things work now is we have to basically walk the VCS DAG and make HTTP requests around Taskcluster to see if an artifact build archive is available for that revision + build configuration. We essentially have to do 1 HTTP request per revision + build config. A release repo is a different build config. So supporting all the repos adds more HTTP requests and latency.

There are a number of potential solutions to this problem though. But they aren't trivial for various reasons.

Comment 2

[Nick Alexander :nalexander [he/him]]

(In reply to Gregory Szorc [:gps] from comment #1)
> A big blocker to this is the extra overhead required to find artifacts. The
> way things work now is we have to basically walk the VCS DAG and make HTTP
> requests around Taskcluster to see if an artifact build archive is available
> for that revision + build configuration. We essentially have to do 1 HTTP
> request per revision + build config. A release repo is a different build
> config. So supporting all the repos adds more HTTP requests and latency.
> 
> There are a number of potential solutions to this problem though. But they
> aren't trivial for various reasons.

Aye.  How hard is it to determine tree/project/etc *in automation only* these days?  That is, can we use the TC environment to opt in to extra artifact trees for mozilla-{release,beta} and possibly even {alder,birch,...}?

I feel like this was frustratingly hard in the past, but maybe life is better now :)

Comment 3

[Gregory Szorc [:gps]]

In Taskcluster, we know which "project" a task is triggered from. Whether that gets passed down to the individual task in a manner that is readable by the task, I'm not sure. If it doesn't, it's trivial to add that to an environment variable for the task. This would be a taskgraph patch.

We could then modify the artifact walking code to take that environment variable into consideration. That's probably the easiest path forward. It also solves problems of artifact build tasks picking up artifacts from other repos/projects. So this sounds like a plan.

I might be missing something though. Maybe concerns over the current project/repo generating artifact build artifacts or not having TC index entries? I can't recall what special magic is all required to power artifact builds these days.