650989 - Some buildbot pulse messages are missing 'buildurl' property

Status: RESOLVED FIXED

(Webtools :: Pulse, defect, P3)

Description

[Jonathan Griffin (:jgriffin)]

Most buildbot pulse messages contain a buildurl property, however, a small percentage (seemingly randomly) do not.

I listen for messages with routing keys that match #.step.maybe_rebooting.finished, and if the message represents the end of a unit test run, I download and parse the log, in order to use the data in OrangeFactor.

99+% of the time, such messages contain a property like so:

      [
        "build_url", 
        "http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-macosx64/1303160474/firefox-6.0a1.en-US.mac.dmg", 
        "DownloadFile"
      ], 

I can use this to infer the location of the unit test log for downloading and parsing.

However, in less than 1% of cases, with no discernible pattern, the pulse message contains no url.  I'm attaching a pulse message which lacks the expected urls.

Comment 1

[Jonathan Griffin (:jgriffin)]

Attachment: pulse message without build url

Comment 2

[Mark Côté [:mcote]]

Not sure if this is still a issue but it bears investigation.

Comment 3

[Jonathan Griffin (:jgriffin)]

Catlee, this is making pulse notifications for emulator builds more or less useless.  Would it be difficult to add the build_url for emulator builds to the pulse messages?  If it is, we could guess the url, but that's more fragile and I'd rather avoid adding more fragility to the system.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

For similar issues in the past I modified code in the http://hg.mozilla.org/build/buildbotcustom/ repository. Maybe we have a condition here, which skips adding the buildurl.

Btw. in case of Mozmill we do not rely on this property but build it dynamically with the other information (version, locale, platform, etc) in mozdownload.

Comment 5

[Chris AtLee [:catlee]]

I expect that the b2g_build.py based builds do not set this property. It should be a trivial fix, given that we already collect the URL so we can use it in the sendchange.

Something in the upload() function should do this: http://hg.mozilla.org/build/mozharness/file/143f996273bb/scripts/b2g_build.py#l873

Comment 6

[Jonathan Griffin (:jgriffin)]

Attachment: Add tests and build urls to emulator buildbot properties,

This is a bit of a stab in the dark.  The format of the properties differs ('testsUrl' vs 'build_url') to follow conventions elsewhere.  'build_url' could also be 'fileUrl' or 'packageUrl', as all of these are used in different places.

Comment 7

[Chris AtLee [:catlee]]

Comment on attachment 8499044 [details] [diff] [review]
Add tests and build urls to emulator buildbot properties,

Review of attachment 8499044 [details] [diff] [review]:
-----------------------------------------------------------------

::: scripts/b2g_build.py
@@ +942,5 @@
>                  matches = glob.glob(os.path.join(dirs['abs_upload_dir'], 'b2g*crashreporter-symbols.zip'))
>                  if matches:
> +                    build_url = "%s/%s" % (download_url, os.path.basename(matches[0]))
> +                    downloadables.append(build_url)
> +                    self.set_buildbot_property('build_url', build_url, write_to_file=True)

this is actually the symbols_url. I think the build_url is in the block above.

we should be consistent and use build_url, symbols_url, tests_url, or packageUrl, symbolsUrl, testsUrl. It appears as if the camel case form is used for other types of builds.

Comment 8

[Jonathan Griffin (:jgriffin)]

Attachment: Add tests and build urls to emulator buildbot properties,

Review feedback addressed.

Comment 9

[Chris AtLee [:catlee]]

Comment on attachment 8500737 [details] [diff] [review]
Add tests and build urls to emulator buildbot properties,

Review of attachment 8500737 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm - do you want to add symbolsUrl there too?

Comment 10

[Jonathan Griffin (:jgriffin)]

(In reply to Chris AtLee [:catlee] from comment #9)
> Comment on attachment 8500737 [details] [diff] [review]
> Add tests and build urls to emulator buildbot properties,
> 
> Review of attachment 8500737 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> lgtm - do you want to add symbolsUrl there too?

Pulsebuildmonitor doesn't use it, but I'll add it for consistency, it may be helpful to someone!

Comment 11

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8500737 [details] [diff] [review]
Add tests and build urls to emulator buildbot properties,

https://hg.mozilla.org/build/mozharness/rev/e52e8837783f

Comment 12

[Jonathan Griffin (:jgriffin)]

Pulse is now publishing the necessary data for emulator builds.  Let's open a new bug for other missing data as needed.