1181710 - Devise a way to provide Treeherder with a URL for retrieving build artifacts (to replace the ftp build directory link)

Status: RESOLVED INCOMPLETE

(Release Engineering :: General, defect)

Description

[Armen [:armenzg]]

STR:
1 - Click on https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound
2 - Click on buildbot build job (e.g. Linux mozilla-inbound build )
3 - Click on "Build: x86 linux32 linux"
4 - It takes you to http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux/1436373800/

30 days later, that link becomes a 404.

I assume that in TH we take the installer url property and drop the filename from it.
The interesting thing of pointing the users to the ftp directory is that they can see *all* artifacts that were uploaded.

Is there a way that we can point them to all the artifacts uploaded for that build on S3?

catlee tells me that the archive [1] will go away when we move to TC.

[1] http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/

Comment 1

[Ed Morley [:emorley]]

Treeherder uses the job log URL to generate that.

There are two parts to this:
1) Switch the log URL shown in builds-4hr to be S3 not FTP (Treeherder will pick this up seemlessly for logs themselves)
2) For the UI's derivation of the "build directory" come up with some other URL we can use per comment 0 here.

However this is a dupe of 1160410; unless you want to make this bug the releng part (ie: coming up with a way to add a URL for "all artifacts" to builds-4hr that we can fetch out; then bug 1160410 will make the treeherder changes?).

Comment 2

[Armen [:armenzg]]

It doesn't matter eitherway to me (dupe or not). I just wanted to make sure that this was not lost in the ftp -> S3 migration.

Comment 3

[Ed Morley [:emorley]]

(In reply to Ed Morley [:emorley] from comment #1)
> Treeherder uses the job log URL to generate that.
> 
> There are two parts to this:
> 1) Switch the log URL shown in builds-4hr to be S3 not FTP (Treeherder will
> pick this up seemlessly for logs themselves)
> 2) For the UI's derivation of the "build directory" come up with some other
> URL we can use per comment 0 here.
> 
> However this is a dupe of 1160410; unless you want to make this bug the
> releng part (ie: coming up with a way to add a URL for "all artifacts" to
> builds-4hr that we can fetch out; then bug 1160410 will make the treeherder
> changes?).

Making this about the releng (or taskcluster) changes needed so that Treeherder has a way of knowing what URL to use for build artifacts. This could be adding some content to builds-4hr, or providing a deterministic URL to access the artifacts (eg artifacts.foo.com/mozilla-inbound/<SHA>/<some job identifier that Treeherder can figure out>) etc.

The Treeherder work to _use_ this URL (once available), will occur in bug 1160410, but for now that bug is blocked on this one.

Comment 4

[Ed Morley [:emorley]]

Do you have any more news on the plan here? Treeherder's bug 1160410 needs a solution from the releng/taskcluster side before it can proceed. Thanks :-)

Comment 5

[Armen [:armenzg]]

I believe the right approach is to query the index.
I believe mshal has been working on the index and could help us determine what are all the artifacts uploaded for a job.

mshal:
* for a buildbot job, how could we determine which artifacts where uploaded?
* and the same for a taskcluster task

For context, we like developers being able to download from Treeherder any artifacts uploaded by a job so they can reproduce issues locally or look at screenshots or mini dumps.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

I don't think this is related to bug 1165405 (the S3 transition for the storage backend of ftp/archive.m.o) as we're still providing an http(s) interface as we were before, as well as propagating data through builds-4hr. 

The only thing I can think of here is to have mozharness emit the taskID into the buildbot properties, so that it's easy to get that from build-4hr.