1162664 - URLs in the "blobber_files" property in builds-4hr are sometimes null

Status: RESOLVED WONTFIX

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Ed Morley [:emorley]]

Treeherder inspects the contents of the "blobber_files" property (which is a json blob), to try and extract log URLs for non-standard logs (eg the json logs for the structured log viewer).

In bug 1162526 we've found instances where the json blob inside "blobber_files" has entries with a name and url, but where the url is 'null'.

eg:
\"browser-chrome-chunked_raw.log\": null,

https://emorley.pastebin.mozilla.org/8832771

Does anyone know why this is happening?

Comment 1

[Ed Morley [:emorley]]

The contents of the "blobber_files" property is to be read in from uploaded_files.json:
https://mxr.mozilla.org/build-central/source/mozharness/mozharness/mozilla/blob_upload.py#99

Which is output by blobberc.py.
https://github.com/mozilla/build-blobuploader/blob/master/blobberc.py#L243

And for whatever reason the blob_url isn't being set here:
https://github.com/mozilla/build-blobuploader/blob/fc9923f7e817bb5c7dfa93509984c777110c492d/blobberc.py#L147
...presumably because the upload failed.

Should we skip that file entry entirely if the upload wasn't successful?
Or at least retry? (or retry more)
Or use "upload failed" or the empty string as a URL instead?

Comment 2

[Chris AtLee [:catlee]]

pastebin is empty now

can you point to some full logs where this is happening?

I agree this is most likely due to the uploads failing. retrying may be an option, if we're not already doing that. One of our early design decisions for blobber was that failing uploads shouldn't impact the state of the job. Do we want to revisit that?

Comment 3

[Ed Morley [:emorley]]

(In reply to Chris AtLee [:catlee] from comment #2)
> pastebin is empty now
> 
> can you point to some full logs where this is happening?

I don't have an easy way to find some more, but the json blob mentioned was:

    "log_references": [{
      "url": "http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-1f185685074d/try-macosx64/try_snowleopard_test-mochitest-e10s-browser-chrome-3-bm108-tests1-macosx-build965.txt.gz",
      "name": "buildbot_text"
    },
    {
      "url": null,
      "name": "mozlog_json"
    }],

Also, comment 1 links to the code that is problematic - the issue can be seen by code inspection.

> I agree this is most likely due to the uploads failing. retrying may be an
> option, if we're not already doing that. One of our early design decisions
> for blobber was that failing uploads shouldn't impact the state of the job.
> Do we want to revisit that?

It's more that if the upload fails, we should:
(a) retry
(b) if it still fails, decide how we should handle this in the json blob. ie: null vs empty string vs some error property vs not including that log "name" at all etc etc?

Comment 4

[Ed Morley [:emorley]]

Wontfix given buildbot EOL