If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

Add blobber urls to messages from NormalizedBuildConsumer

RESOLVED FIXED

Status

Webtools
Pulse
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: ahal, Assigned: ahal)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Assignee)

Description

3 years ago
This would be very helpful for tools wishing to consume the newly uploaded structured log files, though obviously it would be useful for many other use cases as well.

Atm moment, consumers need to download the raw log and scrape the blobber urls out of there.
(Assignee)

Comment 1

3 years ago
I realized that PulseTestConsumer is something completely different and that buildbot test jobs use the normalized build exchange.
Summary: Add blobber urls to messages from PulseTestPublisher → Add blobber urls to messages from NormalizedBuildConsumer
How does this relate to bug 986112?
I agree this would be useful.  It would require, AFAICT, adding buildbot properties to jobs corresponding to blobber files which have been uploaded.
(In reply to Chris Manchester [:chmanchester] from comment #2)
> How does this relate to bug 986112?

It looks like we already set a buildbot property for the blobber manifest (http://hg.mozilla.org/build/mozharness/file/190eb44f4d0e/mozharness/mozilla/blob_upload.py#l91) from bug 986112, so we'll need to grab this value in the pulse normalizer and extract the individual file paths.
(Assignee)

Comment 5

3 years ago
(In reply to Chris Manchester [:chmanchester] from comment #2)
> How does this relate to bug 986112?

The chain of discoverability is:
releng -> ateam -> developers

That bug is about making them discoverable to the next people down the pipeline, i.e us :) (via buildbot property). This bug is about making them discoverable to general tool creators.
(Assignee)

Comment 6

3 years ago
I just noticed that there's a "blobber_manifest_url" property in the non-normalized exchange which is a url to a manifest containing a json dump of uploaded files to their blobber url. It would be nice if the normalized exchange was able to just provide the list of uploaded files directly, so consumers can skip the step of downloading the manifest.

Comment 7

3 years ago
If you need to see a property-per-uploaded-file, you can just call set_buildbot_property() per file and we can stop creating the summary of uploaded files.
http://hg.mozilla.org/build/mozharness/file/316033a17e14/mozharness/mozilla/buildbot.py#l96
(Assignee)

Comment 8

3 years ago
Yeah, I think having a single "blobber_uploads" property that contains a dictionary of file names to blobber urls would be more convenient, then consumers can skip the step of downloading an intermediate manifest. I'll file a separate bug for that.
(Assignee)

Updated

3 years ago
Depends on: 1107571
(Assignee)

Comment 9

3 years ago
Created attachment 8535155 [details] [review]
Add 'blobber_files' to normalized data

Sorry for the unrelated cleanup, I couldn't help myself :p
Assignee: nobody → ahalberstadt
Status: NEW → ASSIGNED
Attachment #8535155 - Flags: review?(jgriffin)
(Assignee)

Comment 10

3 years ago
The 'blobber_files' property won't get set until the next reconfig, but this should be safe to merge anyway.
Comment on attachment 8535155 [details] [review]
Add 'blobber_files' to normalized data

lgtm, thanks for all the property/type changes as well :)
Attachment #8535155 - Flags: review?(jgriffin) → review+
(Assignee)

Comment 12

3 years ago
https://github.com/mozilla/pulsetranslator/commit/74177785e84de3922a5618db27483d4425e6c8eb
https://github.com/mozilla/pulsetranslator/commit/e91c247bf8963ce15eff16fdee716ff1750cfbdc
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.