Closed Bug 1275674 Opened 3 years ago Closed 3 years ago
Error: 'abs _blob _upload _dir' at _resource _record _post _run
58 bytes, text/x-review-board-request
We are getting this failure for our firefox-ui-tests script since bug 1272202 has been fixed: https://treeherder.mozilla.org/logviewer.html#?job_id=3949511&repo=mozilla-central#L963 10:32:39 WARNING - could not upload resource usage JSON 10:32:39 WARNING - Traceback (most recent call last): 10:32:39 WARNING - 10:32:39 WARNING - File "c:\jenkins\workspace\mozilla-central_update\mozharness\mozharness\base\python.py", line 512, in _resource_record_post_run 10:32:39 WARNING - upload_dir = self.query_abs_dirs()['abs_blob_upload_dir'] 10:32:39 WARNING - 10:32:39 WARNING - KeyError: 'abs_blob_upload_dir'
Good times. See also bug 1272539 for a similar failure. The resource usage uploader foo in mozharness needs to be taught about taskcluster. I have no idea how file uploads are managed from TC tasks in mozharness :/
The interesting part is that resource usage for TC jobs is visible on Treeherder. Should there be a specific line for resource-usage.json, or is that invisible and we only see the actual values printed? Pete, who is the one who knows about all that?
The resource usage printed on Treeherder is a result of "TinderboxPrint" lines in the log. Those are only a summary of the raw data that is captured. The raw data is saved to a resource-usage.json file and that is supposed to be uploaded as an artifact. That is the part that is failing.
For taskcluster tasks, the worker takes care of publishing artifacts, so it shouldn't be necessary for the task to upload files anywhere. Instead, the task can declare artifacts, and those will automatically get uploaded. If a mozharness action takes care of the uploading for buildbot runs, probably this action can just be skipped in the TaskCluster equivalent task. See "artifacts" section of https://docs.taskcluster.net/manual/execution/workers/docker-worker#queue-payload-schema Hope that helps! Let me know if you need more info. Other people that might be able to help: wcosta, garndt, jlund.
This warning comes clearly from the mozharness scripts and has nothing to do with taskcluster execution. I can perfectly reproduce it locally. The reason is that our firefox-ui-tests scripts do not declare this path at all. That is because the scripts have never been designed to get run on buildbot. And that's why we do not need the `abs_blob_upload_dir` folder. Greg, should we set this path anyway? Looks like that we place some useful content in there which we might want to have uploaded as artifact. Not sure what this file exactly is.
Yes, I think we should set "abs_blob_upload_dir" in all jobs/configurations so random files can be uploaded by *any* job. I guess firefox-ui-tests has no non-stdout artifacts? If it does, how is it uploading them? I'm just not sure if there is something better than "abs_blob_upload_dir" to accomplish "upload this file." FWIW the file it is trying to upload is metrics of system resource utilization during execution. The file lets us see how efficient our CPU utilization is, for example.
We have our own reports (html, xunit) when the tests are triggered via mozmill-ci. I would like to change that so we also have the screenshots for tc bases tests. This will be done via bug 1283791. As first step I will check what's the best place to define this dir, so maybe not every test job has to do it on its own.
Assignee: nobody → hskupin
Status: NEW → ASSIGNED
FYI this warning started to happen for us when Greg got his patch for uploading the resource_usage.json file landed via bug 1272202.
See Also: → 1272202
Review commit: https://reviewboard.mozilla.org/r/69374/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/69374/
Attachment #8778136 - Flags: review?(jlund)
Comment on attachment 8778136 [details] Bug 1275674 - Mozharness script for firefox-ui tests should use 'abs_blob_upload_dir'. https://reviewboard.mozilla.org/r/69374/#review67234 sorry, was on PTO last week
Attachment #8778136 - Flags: review?(jlund) → review+
Pushed by email@example.com: https://hg.mozilla.org/integration/autoland/rev/98ad8d33b747 Mozharness script for firefox-ui tests should use 'abs_blob_upload_dir'. r=jlund
Simple mozharness patch which will allow us to track resource usage of our tests. We could benefit from it when having it on aurora and beta.
You need to log in before you can comment on or make changes to this bug.