This would be very helpful for tools wishing to consume the newly uploaded structured log files, though obviously it would be useful for many other use cases as well. Atm moment, consumers need to download the raw log and scrape the blobber urls out of there.
I realized that PulseTestConsumer is something completely different and that buildbot test jobs use the normalized build exchange.
How does this relate to bug 986112?
I agree this would be useful. It would require, AFAICT, adding buildbot properties to jobs corresponding to blobber files which have been uploaded.
(In reply to Chris Manchester [:chmanchester] from comment #2) > How does this relate to bug 986112? It looks like we already set a buildbot property for the blobber manifest (http://hg.mozilla.org/build/mozharness/file/190eb44f4d0e/mozharness/mozilla/blob_upload.py#l91) from bug 986112, so we'll need to grab this value in the pulse normalizer and extract the individual file paths.
(In reply to Chris Manchester [:chmanchester] from comment #2) > How does this relate to bug 986112? The chain of discoverability is: releng -> ateam -> developers That bug is about making them discoverable to the next people down the pipeline, i.e us :) (via buildbot property). This bug is about making them discoverable to general tool creators.
I just noticed that there's a "blobber_manifest_url" property in the non-normalized exchange which is a url to a manifest containing a json dump of uploaded files to their blobber url. It would be nice if the normalized exchange was able to just provide the list of uploaded files directly, so consumers can skip the step of downloading the manifest.
If you need to see a property-per-uploaded-file, you can just call set_buildbot_property() per file and we can stop creating the summary of uploaded files. http://hg.mozilla.org/build/mozharness/file/316033a17e14/mozharness/mozilla/buildbot.py#l96
Yeah, I think having a single "blobber_uploads" property that contains a dictionary of file names to blobber urls would be more convenient, then consumers can skip the step of downloading an intermediate manifest. I'll file a separate bug for that.
Created attachment 8535155 [details] [review] Add 'blobber_files' to normalized data Sorry for the unrelated cleanup, I couldn't help myself :p
The 'blobber_files' property won't get set until the next reconfig, but this should be safe to merge anyway.
Comment on attachment 8535155 [details] [review] Add 'blobber_files' to normalized data lgtm, thanks for all the property/type changes as well :)