1275674 - KeyError: 'abs_blob_upload_dir' at _resource_record_post_run

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

We are getting this failure for our firefox-ui-tests script since bug 1272202 has been fixed:

https://treeherder.mozilla.org/logviewer.html#?job_id=3949511&repo=mozilla-central#L963


 10:32:39  WARNING - could not upload resource usage JSON

 10:32:39  WARNING - Traceback (most recent call last):

 10:32:39  WARNING - 

 10:32:39  WARNING -   File "c:\jenkins\workspace\mozilla-central_update\mozharness\mozharness\base\python.py", line 512, in _resource_record_post_run

 10:32:39  WARNING -     upload_dir = self.query_abs_dirs()['abs_blob_upload_dir']

 10:32:39  WARNING - 

10:32:39 WARNING - KeyError: 'abs_blob_upload_dir'

Comment 1

[Gregory Szorc [:gps]]

Good times. See also bug 1272539 for a similar failure.

The resource usage uploader foo in mozharness needs to be taught about taskcluster. I have no idea how file uploads are managed from TC tasks in mozharness :/

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The interesting part is that resource usage for TC jobs is visible on Treeherder. Should there be a specific line for resource-usage.json, or is that invisible and we only see the actual values printed?

Pete, who is the one who knows about all that?

Comment 3

[Gregory Szorc [:gps]]

The resource usage printed on Treeherder is a result of "TinderboxPrint" lines in the log. Those are only a summary of the raw data that is captured.

The raw data is saved to a resource-usage.json file and that is supposed to be uploaded as an artifact. That is the part that is failing.

Comment 4

[Pete Moore [:pmoore][:pete]]

For taskcluster tasks, the worker takes care of publishing artifacts, so it shouldn't be necessary for the task to upload files anywhere. Instead, the task can declare artifacts, and those will automatically get uploaded. If a mozharness action takes care of the uploading for buildbot runs, probably this action can just be skipped in the TaskCluster equivalent task.

See "artifacts" section of https://docs.taskcluster.net/manual/execution/workers/docker-worker#queue-payload-schema

Hope that helps! Let me know if you need more info. Other people that might be able to help: wcosta, garndt, jlund.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

This warning comes clearly from the mozharness scripts and has nothing to do with taskcluster execution. I can perfectly reproduce it locally.

The reason is that our firefox-ui-tests scripts do not declare this path at all. That is because the scripts have never been designed to get run on buildbot. And that's why we do not need the `abs_blob_upload_dir` folder.

Greg, should we set this path anyway? Looks like that we place some useful content in there which we might want to have uploaded as artifact. Not sure what this file exactly is.

Comment 6

[Gregory Szorc [:gps]]

Yes, I think we should set "abs_blob_upload_dir" in all jobs/configurations so random files can be uploaded by *any* job. I guess firefox-ui-tests has no non-stdout artifacts? If it does, how is it uploading them?

I'm just not sure if there is something better than "abs_blob_upload_dir" to accomplish "upload this file."

FWIW the file it is trying to upload is metrics of system resource utilization during execution. The file lets us see how efficient our CPU utilization is, for example.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

We have our own reports (html, xunit) when the tests are triggered via mozmill-ci. I would like to change that so we also have the screenshots for tc bases tests. This will be done via bug 1283791.

As first step I will check what's the best place to define this dir, so maybe not every test job has to do it on its own.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

FYI this warning started to happen for us when Greg got his patch for uploading the resource_usage.json file landed via bug 1272202.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Bug 1275674 - Mozharness script for firefox-ui tests should use 'abs_blob_upload_dir'.

Review commit: https://reviewboard.mozilla.org/r/69374/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/69374/

Comment 10

[Jordan Lund (:jlund)]

Comment on attachment 8778136 [details]
Bug 1275674 - Mozharness script for firefox-ui tests should use 'abs_blob_upload_dir'.

https://reviewboard.mozilla.org/r/69374/#review67234

sorry, was on PTO last week

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/98ad8d33b747
Mozharness script for firefox-ui tests should use 'abs_blob_upload_dir'. r=jlund

Comment 12

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/98ad8d33b747

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Simple mozharness patch which will allow us to track resource usage of our tests. We could benefit from it when having it on aurora and beta.

Comment 14

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/65a7a3e61c3a

Comment 15

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/57dd286a94f0