TaskCluster test jobs are skipping blobber uploads

RESOLVED FIXED

Status

Release Engineering
General Automation
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: jgriffin, Assigned: dustin)

Tracking

(Blocks: 1 bug)

unspecified
Dependency tree / graph

Firefox Tracking Flags

(firefox46 fixed)

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(1 attachment)

(Reporter)

Description

2 years ago
It looks like all of the TaskCluster test jobs are skipping blobber uploads, with the following message:

14:57:35     INFO - Running post-run listener: _upload_blobber_files
14:57:35  WARNING - Blob upload gear skipped. Missing cmdline options.

As a result, we're losing the structured logs for the tests, which will block us from turning off any buildbot jobs.
(Reporter)

Comment 1

2 years ago
Example log from a mochitest-gl run: https://public-artifacts.taskcluster.net/bF-f5WzeTT-u0SDBmr-6GQ/0/public/logs/live_backing.log
I don't think blobber makes sense to use from TaskCluster, which has its own built in artifacts upload support.

Maybe tests in TC should set $MOZ_UPLOAD_DIR to a subdirectory inside the TC artifacts directory?
That's a good point. The implication here is that mozharness scripts will need to know whether or not they are being invoked by buildbot or taskcluster. Maybe they already do?
(Assignee)

Comment 4

2 years ago
Well, they need to get the proper config to behave the right way in either case.  I don't think that's --invoked-from-taskcluster, so much as MOZ_UPLOAD_DIR=/home/worker/artifacts/blobber or something like that
Correct: https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/desktop_unittest.py#641

Simplest thing would be to make that a commandline option that defaults to 'abs_blob_upload_dir', and then taskcluster can override it.

Comment 6

2 years ago
That should work.

What are we using the structured logs? Are they used for ActiveData? Do we query the Buildbot property 'blobber_files'? We would have to discuss for a different approach for TC; Perhaps finding them in the list of artifacts.

blobber_files:{"mozilla-test-fail-screenshot_CC2kiM.png": "http://mozilla-releng-blobs.s3.amazonaws.com/blobs/mozilla-inbound/sha512/02007064bf5589bb3d1cfd05a116f20eaf6056a6370c878e5f256589d8bf47d226516b0580264241b47418c5c6aa1c71f500c6d50b3e2a1fe4169d5f03b6aef0", "browser-chrome-chunked_raw.log": "http://mozilla-releng-blobs.s3.amazonaws.com/blobs/mozilla-inbound/sha512/f28fc5c40232e55d2ef14c27798cf4b6437e65c0f5e728d461c16d0ece5bf39ab0d0176ed3954cbb0addb1837cd3c28e8bd713dc81e20c1f8b32f6547bf5d39f", "browser-chrome-chunked_errorsummary.log": "http://mozilla-releng-blobs.s3.amazonaws.com/blobs/mozilla-inbound/sha512/837ed213ba5573b25b6e10b34822a1e03923ef39890b91d6eb4e14652f2c41a35856b9b540badc7d3f5ac075a098bf9454d979b78286455f05439ef371f7b8e0"}
(Reporter)

Comment 7

2 years ago
We need them for both ActiveData and automatic classification in Treeherder. I agree it makes sense to move them to being TaskCluster artifacts.

Comment 8

2 years ago
We're on the same page there.

jgraham, ekyle: how do ActiveData and automatic classification grab the structured logs?

We're trying to *not* use Blobber as an upload mechanism from the test jobs when moving the jobs into TC, which has its own upload mechanisms.
We want to make sure the TC uploads will get to your pipelines.
ActiveData uses the `blobber_files` property found in the normalized pulse logs, and found in the buildbot JSON logs.

> "blobber_files": {
>     "xpcshell_raw.log": "http://mozilla-releng-blobs.s3.amazonaws.com/blobs/mozilla-inbound/sha512/7ff36184866022a0d34fa6fbcb109ef715d98e74a881c9564c112096d4a69fc75950835c06d0271c4a117f489d8e42df903ff39af5c12e0db4069cb44a264aac"
> }
Taskcluster uses a separate pulse stream. Maybe pulsetranslator should be the piece that listens to both the buildbot and taskcluster streams, and then normalizes them into a single stream so tools like ActiveData don't have to.

But this is getting a little off topic for this bug. Fixing mozharness scripts is the first step.
(Reporter)

Comment 11

2 years ago
(In reply to Andrew Halberstadt [:ahal] from comment #10)
> Taskcluster uses a separate pulse stream. Maybe pulsetranslator should be
> the piece that listens to both the buildbot and taskcluster streams, and
> then normalizes them into a single stream so tools like ActiveData don't
> have to.

Filed bug 1222361 for this.

Updated

2 years ago
Duplicate of this bug: 1222113

Comment 13

2 years ago
I will be looking into this.
Assignee: nobody → armenzg
(Assignee)

Updated

2 years ago
Assignee: armenzg → dustin
(Assignee)

Comment 14

2 years ago
Actually, I think there's very little to do here:

14:51:43     INFO - ENV: MOZ_UPLOAD_DIR is now /home/worker/workspace/build/blobber_upload_dir
...
[taskcluster] === Task Finished ===
[taskcluster] Artifact "public/logs/" not found at "/home/worker/build/upload/logs/"
[taskcluster] Artifact "public/test_info/" not found at "/home/worker/build/blobber_upload_dir/"

so we just need to fix the task def to pull from the right place.
(Assignee)

Comment 15

2 years ago
Created attachment 8698095 [details]
MozReview Request: Bug 1221553: fix getting artifacts from blobber upload dir; r?armenzg

Bug 1221553: fix getting artifacts from blobber upload dir; r?armenzg
Attachment #8698095 - Flags: review?(armenzg)

Comment 16

2 years ago
Comment on attachment 8698095 [details]
MozReview Request: Bug 1221553: fix getting artifacts from blobber upload dir; r?armenzg

https://reviewboard.mozilla.org/r/27795/#review24973

r+ if that try run goes well.
Attachment #8698095 - Flags: review?(armenzg) → review+

Comment 17

2 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/f2bd6a971f80

Comment 18

2 years ago
ekyle:
After the change in this bug, I believe all TC test jobs are going to start uploading their structured logs.
https://tools.taskcluster.net/task-inspector/#FZl3ofjGRE2t_f9hr7yXjg/0

You can see wpt_raw.log uploaded in the list of artifacts.
Excellent!  I will hook this up tomorrow!

Comment 20

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/f2bd6a971f80
Status: NEW → RESOLVED
Last Resolved: 2 years ago
status-firefox46: --- → fixed
Resolution: --- → FIXED
Blocks: 1223057

Updated

2 years ago
See Also: → bug 1235734
You need to log in before you can comment on or make changes to this bug.