1091003 - Add blobber urls to messages from NormalizedBuildConsumer

Status: RESOLVED FIXED

(Webtools :: Pulse, defect)

Description

[Andrew Halberstadt [:ahal]]

This would be very helpful for tools wishing to consume the newly uploaded structured log files, though obviously it would be useful for many other use cases as well.

Atm moment, consumers need to download the raw log and scrape the blobber urls out of there.

Comment 1

[Andrew Halberstadt [:ahal]]

I realized that PulseTestConsumer is something completely different and that buildbot test jobs use the normalized build exchange.

Comment 2

[Chris Manchester (limited bugmail, email directly)]

How does this relate to bug 986112?

Comment 3

[Jonathan Griffin (:jgriffin)]

I agree this would be useful.  It would require, AFAICT, adding buildbot properties to jobs corresponding to blobber files which have been uploaded.

Comment 4

[Jonathan Griffin (:jgriffin)]

(In reply to Chris Manchester [:chmanchester] from comment #2)
> How does this relate to bug 986112?

It looks like we already set a buildbot property for the blobber manifest (http://hg.mozilla.org/build/mozharness/file/190eb44f4d0e/mozharness/mozilla/blob_upload.py#l91) from bug 986112, so we'll need to grab this value in the pulse normalizer and extract the individual file paths.

Comment 5

[Andrew Halberstadt [:ahal]]

(In reply to Chris Manchester [:chmanchester] from comment #2)
> How does this relate to bug 986112?

The chain of discoverability is:
releng -> ateam -> developers

That bug is about making them discoverable to the next people down the pipeline, i.e us :) (via buildbot property). This bug is about making them discoverable to general tool creators.

Comment 6

[Andrew Halberstadt [:ahal]]

I just noticed that there's a "blobber_manifest_url" property in the non-normalized exchange which is a url to a manifest containing a json dump of uploaded files to their blobber url. It would be nice if the normalized exchange was able to just provide the list of uploaded files directly, so consumers can skip the step of downloading the manifest.

Comment 7

[Armen [:armenzg]]

If you need to see a property-per-uploaded-file, you can just call set_buildbot_property() per file and we can stop creating the summary of uploaded files.
http://hg.mozilla.org/build/mozharness/file/316033a17e14/mozharness/mozilla/buildbot.py#l96

Comment 8

[Andrew Halberstadt [:ahal]]

Yeah, I think having a single "blobber_uploads" property that contains a dictionary of file names to blobber urls would be more convenient, then consumers can skip the step of downloading an intermediate manifest. I'll file a separate bug for that.

Comment 9

[Andrew Halberstadt [:ahal]]

Attachment: Add 'blobber_files' to normalized data

Sorry for the unrelated cleanup, I couldn't help myself :p

Comment 10

[Andrew Halberstadt [:ahal]]

The 'blobber_files' property won't get set until the next reconfig, but this should be safe to merge anyway.

Comment 11

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8535155 [details] [review]
Add 'blobber_files' to normalized data

lgtm, thanks for all the property/type changes as well :)

Comment 12

[Andrew Halberstadt [:ahal]]

https://github.com/mozilla/pulsetranslator/commit/74177785e84de3922a5618db27483d4425e6c8eb
https://github.com/mozilla/pulsetranslator/commit/e91c247bf8963ce15eff16fdee716ff1750cfbdc