Closed Bug 1585161 Opened 6 years ago Closed 1 year ago

Intermittent [taskcluster:error] Uploading error artifact public/logs from file logs with message "Could not read directory 'C:\\tasks\\task_1569873924\\logs'", reason "file-missing-on-worker" and expiry 2020-09-29T16:24:51.919

Categories

(Infrastructure & Operations :: RelOps: Windows OS, defect, P5)

Product:
Infrastructure & Operations
Component:
RelOps: Windows OS
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: intermittent-bug-filer, Assigned: markco)

References

Details

(Keywords: intermittent-failure, regression, Whiteboard: [stockwell unknown])

Filed by: apavel [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=269128281&repo=mozilla-central
Full log: https://queue.taskcluster.net/v1/task/SNmVMIa1R9CMqSTOKOnYjw/runs/0/artifacts/public/logs/live_backing.log

[taskcluster 2019-09-30T20:02:05.290Z] === Task Starting ===
[taskcluster 2019-09-30T20:02:06.776Z] Uploading redirect artifact public/logs/live.log to URL http://10.6.2.20:49800/log/bCvoPDmoT0yhaKUHema62Q with mime type "text/plain; charset=utf-8" and expiry 2019-09-30T21:47:06.316Z
[taskcluster 2019-09-30T20:02:07.157Z] [mounts] Downloading task SeoNwAMcQMmEH877Ka2Uvw artifact public/build/mozharness.zip to downloads\TBrtFMi0RQGjEaxtkufYbg
[taskcluster 2019-09-30T20:02:07.881Z] [mounts] Downloaded 2333554 bytes with SHA256 b35ee6961ded73d5c3ea7aa069c0244bbb8a295cb8fb768e0cad9f8e5737a6bb from task SeoNwAMcQMmEH877Ka2Uvw artifact public/build/mozharness.zip to downloads\TBrtFMi0RQGjEaxtkufYbg
[taskcluster:warn 2019-09-30T20:02:07.881Z] [mounts] Download downloads\TBrtFMi0RQGjEaxtkufYbg of task SeoNwAMcQMmEH877Ka2Uvw artifact public/build/mozharness.zip has SHA256 b35ee6961ded73d5c3ea7aa069c0244bbb8a295cb8fb768e0cad9f8e5737a6bb but task payload does not declare a required value, so content authenticity cannot be verified
[taskcluster 2019-09-30T20:02:07.881Z] [mounts] Creating directory C:\tasks\task_1569872308 with permissions 0700
[taskcluster 2019-09-30T20:02:07.881Z] [mounts] Extracting zip file downloads\TBrtFMi0RQGjEaxtkufYbg to 'C:\tasks\task_1569872308'
[taskcluster 2019-09-30T20:02:11.931Z] [mounts] Downloading url https://hg.mozilla.org/mozilla-central/raw-file/e545376c33914c25a9252a74e09bd2b5b3e8ed76/taskcluster/scripts/run-task to downloads\KdkUVv-XT0CqvFaMOxSPDg
[taskcluster 2019-09-30T20:02:13.288Z] [mounts] Downloaded 31468 bytes with SHA256 c69f9ae1494ced424fb5fcc36989f28ef291779f76f61ac6e7a5a1be9a560684 from url https://hg.mozilla.org/mozilla-central/raw-file/e545376c33914c25a9252a74e09bd2b5b3e8ed76/taskcluster/scripts/run-task to downloads\KdkUVv-XT0CqvFaMOxSPDg
[taskcluster:warn 2019-09-30T20:02:13.288Z] [mounts] Download downloads\KdkUVv-XT0CqvFaMOxSPDg of url https://hg.mozilla.org/mozilla-central/raw-file/e545376c33914c25a9252a74e09bd2b5b3e8ed76/taskcluster/scripts/run-task has SHA256 c69f9ae1494ced424fb5fcc36989f28ef291779f76f61ac6e7a5a1be9a560684 but task payload does not declare a required value, so content authenticity cannot be verified
[taskcluster 2019-09-30T20:02:13.289Z] [mounts] Creating directory C:\tasks\task_1569872308 with permissions 0700
[taskcluster 2019-09-30T20:02:13.289Z] [mounts] Copying downloads\KdkUVv-XT0CqvFaMOxSPDg to C:\tasks\task_1569872308\run-task
[taskcluster 2019-09-30T20:02:13.635Z] Executing command 0: C:/mozilla-build/python3/python3.exe run-task -- c:\mozilla-build\python\python.exe -u mozharness\scripts\web_platform_tests.py --cfg mozharness\configs\web_platform_tests\prod_config_windows_taskcluster.py --test-type=reftest --setpref=media.peerconnection.mtransport_process=false --setpref=network.process.enabled=false --download-symbols ondemand --total-chunk=4 --this-chunk=2
C:\tasks\task_1569872308>C:/mozilla-build/python3/python3.exe run-task -- c:\mozilla-build\python\python.exe -u mozharness\scripts\web_platform_tests.py --cfg mozharness\configs\web_platform_tests\prod_config_windows_taskcluster.py --test-type=reftest --setpref=media.peerconnection.mtransport_process=false --setpref=network.process.enabled=false --download-symbols ondemand --total-chunk=4 --this-chunk=2
[taskcluster 2019-09-30T20:02:13.945Z] Exit Code: -1073741515
[taskcluster 2019-09-30T20:02:13.945Z] User Time: 0s
[taskcluster 2019-09-30T20:02:13.945Z] Kernel Time: 62.5ms
[taskcluster 2019-09-30T20:02:13.945Z] Wall Time: 257.8558ms
[taskcluster 2019-09-30T20:02:13.945Z] Result: FAILED
[taskcluster 2019-09-30T20:02:13.945Z] === Task Finished ===
[taskcluster 2019-09-30T20:02:13.945Z] Task Duration: 309.8575ms
[taskcluster:error] Uploading error artifact public/logs from file logs with message "Could not read directory 'C:\tasks\task_1569872308\logs'", reason "file-missing-on-worker" and expiry 2020-09-29T16:24:53.133Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read directory 'C:\tasks\task_1569872308\logs'
[taskcluster:error] Uploading error artifact public/test_info from file build/blobber_upload_dir with message "Could not read directory 'C:\tasks\task_1569872308\build\blobber_upload_dir'", reason "file-missing-on-worker" and expiry 2020-09-29T16:24:53.133Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read directory 'C:\tasks\task_1569872308\build\blobber_upload_dir'
[taskcluster 2019-09-30T20:02:15.195Z] Uploading redirect artifact public/logs/live.log to URL https://queue.taskcluster.net/v1/task/RRCWit82Qyev0EatWlBXdg/runs/0/artifacts/public/logs/live_backing.log with mime type "text/plain; charset=utf-8" and expiry 2020-09-29T16:24:53.133Z
[taskcluster:error] exit status -1073741515
[taskcluster:error] file-missing-on-worker: Could not read directory 'C:\tasks\task_1569872308\logs'
[taskcluster:error] file-missing-on-worker: Could not read directory 'C:\tasks\task_1569872308\build\blobber_upload_dir'

I handle the Android (Linux) workers on Bitbar. I think :markco might know what's up with these.

Flags: needinfo?(aerickson) → needinfo?(mcornmesser)

Thank you.

It looks like the file should have been generated by the zip file.

[taskcluster 2019-09-30T20:02:07.881Z] [mounts] Creating directory C:\tasks\task_1569872308 with permissions 0700
[taskcluster 2019-09-30T20:02:07.881Z] [mounts] Extracting zip file downloads\TBrtFMi0RQGjEaxtkufYbg to 'C:\tasks\task_1569872308'

I don't think it is a configuration issue. I am going to redirect to Rob since he is the most familiar with the Yoga laptops.

Flags: needinfo?(mcornmesser) → needinfo?(rthijssen)

i think the interesting part of this error log is Exit Code: -1073741515 on the failing task command step.

  • -1073741515 converts to hex: FFFF FFFF C000 0135 or 0xC0000135
  • 0xC0000135 may map to: STATUS_DLL_NOT_FOUND, which expands to:

    {Unable To Locate Component} This application has failed to start because %hs was not found. Reinstalling the application might fix this problem.

if i'm barking up the right tree here, there's some application on this system that needs to be reinstalled. most likely something related to the .net framework or a windows sdk.

i will investigate further.

Assignee: nobody → rthijssen
Status: NEW → ASSIGNED
Flags: needinfo?(rthijssen)
Blocks: 1585433
Blocks: 1585578

Recent occurrences here have been classified in relation with https://bugzilla.mozilla.org/show_bug.cgi?id=1481291#c56

No longer blocks: 1585578

There are 21 total failures in the last 7 days: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2020-02-26&endday=2020-03-04&tree=trunk&bug=1585161

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=291531846&repo=autoland&lineNumber=50

Rob, are there any updates?

Flags: needinfo?(rthijssen)
Whiteboard: [stockwell needswork:owner]
Component: Workers → RelOps: Windows OS
Product: Taskcluster → Infrastructure & Operations

There are 36 total failures in the last 7 days on windows builds and linux64 builds

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=327612747&repo=autoland&lineNumber=61

[taskcluster 2021-01-24T04:30:42.065Z] Executing command 1: /usr/local/bin/python3 run-task -- /usr/local/bin/python2 -u 'mozharness/scripts/desktop_unittest.py' --cfg 'mozharness/configs/unittests/mac_unittest.py' --mochitest-suite=mochitest-webgl2-core '--setpref=media.peerconnection.mtransport_process=false' '--setpref=network.process.enabled=false' --enable-webrender '--setpref=layers.d3d11.enable-blacklist=false' --download-symbols ondemand --total-chunk=1 --this-chunk=1
[setup 2021-01-24T04:30:42.149Z] run-task started in /Users/cltbld/tasks/task_1611450905
[setup 2021-01-24T04:30:42.153Z] MOZ_FETCHES_DIR is /Users/cltbld/tasks/task_1611450905/fetches
[fetches 2021-01-24T04:30:42.153Z] fetching artifacts
[fetches 2021-01-24T04:30:42.154Z] executing ['/usr/local/bin/python3', '-u', 'fetch-content', 'task-artifacts']
[fetches 2021-01-24T04:30:42.263Z] attempt 1/5
[fetches 2021-01-24T04:30:42.263Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Z-cZ9Z09TMOg0iK86JEfhw/artifacts/public/build/fix-stacks.tar.xz to /Users/cltbld/tasks/task_1611450905/fetches/fix-stacks.tar.xz
[fetches 2021-01-24T04:30:42.263Z] attempt 1/5
[fetches 2021-01-24T04:30:42.263Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/etV2Amv_R6CGkEt_PWtP9w/artifacts/public/build/minidump_stackwalk.tar.xz to /Users/cltbld/tasks/task_1611450905/fetches/minidump_stackwalk.tar.xz
[fetches 2021-01-24T04:30:42.263Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Z-cZ9Z09TMOg0iK86JEfhw/artifacts/public/build/fix-stacks.tar.xz
[fetches 2021-01-24T04:30:42.263Z] Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/etV2Amv_R6CGkEt_PWtP9w/artifacts/public/build/minidump_stackwalk.tar.xz
[fetches 2021-01-24T04:30:42.264Z] fetch-content:180: DeprecationWarning: cafile, capath and cadefault are deprecated, use a custom context instead.
[fetches 2021-01-24T04:30:42.264Z] with urllib.request.urlopen(url, cafile=certifi.where()) if certifi else urllib.request.urlopen(url) as fh:
[fetches 2021-01-24T04:30:42.402Z] https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Z-cZ9Z09TMOg0iK86JEfhw/artifacts/public/build/fix-stacks.tar.xz resolved to 955352 bytes with sha256 41ff9e6c0820a0dcfef045bab526b391f3d9cced616d73ebbd81a63da4e18765 in 0.139s
[fetches 2021-01-24T04:30:42.402Z] Extracting /Users/cltbld/tasks/task_1611450905/fetches/fix-stacks.tar.xz to /Users/cltbld/tasks/task_1611450905/fetches
[fetches 2021-01-24T04:30:42.562Z] /Users/cltbld/tasks/task_1611450905/fetches/fix-stacks.tar.xz extracted in 0.160s
[fetches 2021-01-24T04:30:42.562Z] Removing /Users/cltbld/tasks/task_1611450905/fetches/fix-stacks.tar.xz
[taskcluster:error] Aborting task...
[taskcluster:warn 2021-01-24T05:00:42.157Z] no such process
[taskcluster 2021-01-24T05:00:42.157Z] === Task Finished ===
[taskcluster 2021-01-24T05:00:42.157Z] Task Duration: 30m0.094957325s
[taskcluster:error] Uploading error artifact public/logs from file logs with message "Could not read directory '/Users/cltbld/tasks/task_1611450905/logs'", reason "file-missing-on-worker" and expiry 2022-01-24T04:13:37.651Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read directory '/Users/cltbld/tasks/task_1611450905/logs'
[taskcluster:error] Uploading error artifact public/test_info from file build/blobber_upload_dir with message "Could not read directory '/Users/cltbld/tasks/task_1611450905/build/blobber_upload_dir'", reason "file-missing-on-worker" and expiry 2022-01-24T04:13:37.651Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read directory '/Users/cltbld/tasks/task_1611450905/build/blobber_upload_dir'
[taskcluster:error] Task aborted - max run time exceeded
[taskcluster:error] file-missing-on-worker: Could not read directory '/Users/cltbld/tasks/task_1611450905/logs'
[taskcluster:error] file-missing-on-worker: Could not read directory '/Users/cltbld/tasks/task_1611450905/build/blobber_upload_dir'

Mark can you assign someone to take a look?

Flags: needinfo?(rthijssen) → needinfo?(mcornmesser)
Whiteboard: [stockwell unknown] → [stockwell needswork:owner]

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=327612747&repo=autoland&lineNumber=61
This looks like a an OS X test Worker Type (gecko-t-osx-1014) settings:

From looking at the details in comment 79, this looks like it this happen across the all 3 OS we test, but it looks like the majority of these was one problem worker T-W1064-MS-555 . Which has since been quarantined.

With the exception of ms-555, because it is intermittent across platforms and environments it doesn't look like it is being caused by a local configuration issue.

dhouse: jmaher: Any ideas?

Flags: needinfo?(jmaher)
Flags: needinfo?(dhouse)

I think some of these cases fall into a task timeout of X minutes and the task itself is taking something like X-1 minutes, so any cleanup that happens will hit the timeout.

That is a theory, ideally we would see another pattern emerge in the coming weeks.

Flags: needinfo?(jmaher)
Flags: needinfo?(mcornmesser)

+1, from the last few failure logs, it looks those paths do not exist yet because the task did not start before the failure.

I think the actual failures for these tasks are not the missing files (like this second to last windows one looks like a zstd/fetch failure), since the failed collection of the missing artifacts is happening after the "Task Finished" entry.

examples from last 5:

https://treeherder.mozilla.org/logviewer?job_id=327612747&repo=autoland&lineNumber=61

[fetches 2021-01-24T04:30:42.402Z] https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Z-cZ9Z09TMOg0iK86JEfhw/artifacts/public/build/fix-stacks.tar.xz resolved to 955352 bytes with sha256 41ff9e6c0820a0dcfef045bab526b391f3d9cced616d73ebbd81a63da4e18765 in 0.139s
[fetches 2021-01-24T04:30:42.402Z] Extracting /Users/cltbld/tasks/task_1611450905/fetches/fix-stacks.tar.xz to /Users/cltbld/tasks/task_1611450905/fetches
[fetches 2021-01-24T04:30:42.562Z] /Users/cltbld/tasks/task_1611450905/fetches/fix-stacks.tar.xz extracted in 0.160s
[fetches 2021-01-24T04:30:42.562Z] Removing /Users/cltbld/tasks/task_1611450905/fetches/fix-stacks.tar.xz
[taskcluster:error] Aborting task...
[taskcluster:warn 2021-01-24T05:00:42.157Z] no such process
[taskcluster 2021-01-24T05:00:42.157Z] === Task Finished ===
[taskcluster 2021-01-24T05:00:42.157Z] Task Duration: 30m0.094957325s
[taskcluster:error] Uploading error artifact public/logs from file logs with message "Could not read directory '/Users/cltbld/tasks/task_1611450905/logs'", reason "file-missing-on-worker" and expiry 2022-01-24T04:13:37.651Z

https://treeherder.mozilla.org/logviewer?job_id=327463449&repo=mozilla-central&lineNumber=59

[fetches 2021-01-22T12:13:48.351Z] Extracting /home/cltbld/tasks/task_1611317624/fetches/minidump_stackwalk.tar.xz to /home/cltbld/tasks/task_1611317624/fetches
[fetches 2021-01-22T12:13:48.432Z] /home/cltbld/tasks/task_1611317624/fetches/minidump_stackwalk.tar.xz extracted in 0.082s
[fetches 2021-01-22T12:13:48.432Z] Removing /home/cltbld/tasks/task_1611317624/fetches/minidump_stackwalk.tar.xz
[taskcluster:error] Aborting task...
[taskcluster:warn 2021-01-22T12:38:48.081Z] no such process
[taskcluster 2021-01-22T12:38:48.081Z] === Task Finished ===
[taskcluster 2021-01-22T12:38:48.081Z] Task Duration: 25m0.000446038s
[taskcluster:error] Uploading error artifact public/logs from file logs with message "Could not read directory '/home/cltbld/tasks/task_1611317624/logs'", reason "file-missing-on-worker" and expiry 2022-01-22T10:22:37.486Z

https://treeherder.mozilla.org/logviewer?job_id=327284871&repo=autoland&lineNumber=40

[taskcluster 2021-01-20T19:09:44.391Z] [mounts] Copying C:\downloads\NMCsTmqbR-CTr6fXGQj1FA to C:\Users\task_1611168146\fetch-content
[taskcluster 2021-01-20T19:09:44.393Z] [mounts] Granting task_1611168146 full control of file 'C:\Users\task_1611168146\fetch-content'
[taskcluster 2021-01-20T19:09:44.405Z] Executing command 0: C:/mozilla-build/python3/python3.exe run-task -- c:\mozilla-build\python\python.exe -u mozharness\scripts\talos_script.py --cfg mozharness\configs\talos\windows_config.py --suite=damp --use-talos-json --setpref=media.peerconnection.mtransport_process=false --setpref=network.process.enabled=false --enable-webrender --setpref=layers.d3d11.enable-blacklist=false --download-symbols ondemand
[taskcluster 2021-01-20T19:09:44.420Z]    Exit Code: 3221225794
[taskcluster 2021-01-20T19:09:44.420Z]    User Time: 0s
[taskcluster 2021-01-20T19:09:44.420Z]  Kernel Time: 0s
[taskcluster 2021-01-20T19:09:44.420Z]    Wall Time: 14.9921ms
[taskcluster 2021-01-20T19:09:44.420Z]       Result: FAILED
[taskcluster 2021-01-20T19:09:44.420Z] === Task Finished ===
[taskcluster 2021-01-20T19:09:44.420Z] Task Duration: 14.9921ms
[taskcluster:error] Uploading error artifact public/logs from file logs with message "Could not read directory 'C:\\Users\\task_1611168146\\logs'", reason "file-missing-on-worker" and expiry 2022-01-20T19:01:36.094Z

https://treeherder.mozilla.org/logviewer?job_id=327236557&repo=mozilla-central&lineNumber=40

[taskcluster 2021-01-20T12:06:29.618Z] [mounts] Granting task_1611144059 full control of file 'C:\Users\task_1611144059\fetch-content'
[taskcluster 2021-01-20T12:06:29.630Z] Executing command 0: C:/mozilla-build/python3/python3.exe run-task -- c:\mozilla-build\python\python.exe -u mozharness\scripts\raptor_script.py --cfg mozharness\configs\raptor\windows_config.py --setpref=extensions.logging.enabled=true --no-conditioned-profile --app=firefox --cold --test=raptor-tp6-imdb-firefox --project=mozilla-central --enable-webrender --download-symbols ondemand
[taskcluster 2021-01-20T12:06:29.645Z]    Exit Code: 3221225794
[taskcluster 2021-01-20T12:06:29.645Z]    User Time: 0s
[taskcluster 2021-01-20T12:06:29.645Z]  Kernel Time: 0s
[taskcluster 2021-01-20T12:06:29.645Z]    Wall Time: 15.145ms
[taskcluster 2021-01-20T12:06:29.645Z]       Result: FAILED
[taskcluster 2021-01-20T12:06:29.645Z] === Task Finished ===
[taskcluster 2021-01-20T12:06:29.645Z] Task Duration: 15.145ms
[taskcluster:error] Uploading error artifact public/logs from file logs with message "Could not read directory 'C:\\Users\\task_1611144059\\logs'", reason "file-missing-on-worker" and expiry 2022-01-20T09:55:29.846Z

https://treeherder.mozilla.org/logviewer?job_id=327228684&repo=autoland&lineNumber=48

[setup 2021-01-20T09:32:56.704Z] MOZ_FETCHES_DIR is C:/Users/task_1611131256/fetches
[fetches 2021-01-20T09:32:56.704Z] fetching artifacts
[fetches 2021-01-20T09:32:56.711Z] executing ['C:\\mozilla-build\\python3\\python3.exe', '-u', 'fetch-content', 'task-artifacts']
[fetches 2021-01-20T09:32:56.822Z] Traceback (most recent call last):
[fetches 2021-01-20T09:32:56.822Z]   File "fetch-content", line 30, in <module>
[fetches 2021-01-20T09:32:56.822Z]     import zstandard
[fetches 2021-01-20T09:32:56.822Z] ValueError: source code string cannot contain null bytes
[taskcluster 2021-01-20T09:32:56.852Z]    Exit Code: 1
[taskcluster 2021-01-20T09:32:56.852Z]    User Time: 0s
[taskcluster 2021-01-20T09:32:56.852Z]  Kernel Time: 0s
[taskcluster 2021-01-20T09:32:56.852Z]    Wall Time: 329.1717ms
[taskcluster 2021-01-20T09:32:56.852Z]       Result: FAILED
[taskcluster 2021-01-20T09:32:56.852Z] === Task Finished ===
[taskcluster 2021-01-20T09:32:56.852Z] Task Duration: 329.1717ms
[taskcluster:error] Uploading error artifact public/logs from file logs with message "Could not read directory 'C:\\Users\\task_1611131256\\logs'", reason "file-missing-on-worker" and expiry 2022-01-20T09:11:11.055Z
Flags: needinfo?(dhouse)

Perhaps the windows task runs are capturing the failure cause and the others are not:
https://treeherder.mozilla.org/logviewer?job_id=327224165&repo=autoland&lineNumber=42

[fetches 2021-01-20T06:22:08.692Z] fetching artifacts
[fetches 2021-01-20T06:22:08.699Z] executing ['C:\\mozilla-build\\python3\\python3.exe', '-u', 'fetch-content', 'task-artifacts']
[fetches 2021-01-20T06:22:08.804Z] Traceback (most recent call last):
[fetches 2021-01-20T06:22:08.804Z]   File "fetch-content", line 30, in <module>
[fetches 2021-01-20T06:22:08.804Z]     import zstandard
[fetches 2021-01-20T06:22:08.804Z] ValueError: source code string cannot contain null bytes
[taskcluster 2021-01-20T06:22:08.834Z]    Exit Code: 1
[taskcluster 2021-01-20T06:22:08.834Z]    User Time: 0s
[taskcluster 2021-01-20T06:22:08.834Z]  Kernel Time: 15.625ms
[taskcluster 2021-01-20T06:22:08.834Z]    Wall Time: 326.5141ms
[taskcluster 2021-01-20T06:22:08.834Z]       Result: FAILED
[taskcluster 2021-01-20T06:22:08.834Z] === Task Finished ===
[taskcluster 2021-01-20T06:22:08.834Z] Task Duration: 326.5141ms
[taskcluster:error] Uploading error artifact public/logs from file logs with message "Could not read directory 'C:\\Users\\task_1611123377\\logs'", reason "file-missing-on-worker" and expiry 2022-01-20T05:04:21.954Z

Recent log:
https://treeherder.mozilla.org/logviewer?job_id=335284482&repo=autoland&lineNumber=100

There's a big spike in these failures in the last day.

There are 72 total failures in the last 2 days on

[fetches 2021-04-02T13:54:56.185Z] Removing /home/cltbld/tasks/task_161737169447753/fetches/fix-stacks.tar.xz
[fetches 2021-04-02T13:54:56.186Z]
[fetches 2021-04-02T13:54:56.187Z] inflating: wasm-misc/embenchen/src/wasm_lua_binarytrees.c.js
[fetches 2021-04-02T13:54:56.189Z] inflating: wasm-misc/embenchen/src/wasm_lua_binarytrees.c.wasm
[fetches 2021-04-02T13:54:56.190Z] inflating: wasm-misc/embenchen/src/wasm_skinning.js
[fetches 2021-04-02T13:54:56.190Z] inflating: wasm-misc/embenchen/src/wasm_skinning.wasm
[fetches 2021-04-02T13:54:56.191Z] inflating: wasm-misc/embenchen/src/wasm_zlib.c.js
[fetches 2021-04-02T13:54:56.191Z] inflating: wasm-misc/embenchen/src/wasm_zlib.c.wasm
[fetches 2021-04-02T13:54:56.191Z] inflating: wasm-misc/embenchen/worker.js
[fetches 2021-04-02T13:54:56.192Z] inflating: wasm-misc/index.html
[fetches 2021-04-02T13:54:56.192Z] /home/cltbld/tasks/task_161737169447753/fetches/wasm-misc.zip extracted in 0.106s
[fetches 2021-04-02T13:54:56.192Z] Removing /home/cltbld/tasks/task_161737169447753/fetches/wasm-misc.zip
[fetches 2021-04-02T13:54:56.193Z] PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"name": "fetch_content", "value": 0.26750746900000166, "lowerIsBetter": true, "shouldAlert": false, "subtests": []}]}
[fetches 2021-04-02T13:54:56.203Z] finished fetching artifacts
[task 2021-04-02T13:54:56.203Z] executing ['/usr/bin/python3', '-u', 'mozharness/scripts/raptor_script.py', '--cfg', 'mozharness/configs/raptor/linux_config.py', '--setpref=extensions.logging.enabled=true', '--no-conditioned-profile', '--test=raptor-wasm-misc', '--project=autoland', '--download-symbols', 'ondemand']
[task 2021-04-02T13:54:56.392Z] Usage: raptor_script.py [options]
[task 2021-04-02T13:54:56.392Z]
[task 2021-04-02T13:54:56.392Z] raptor_script.py: error: no such option: --no-conditioned-profile
[taskcluster 2021-04-02T13:54:56.408Z] Exit Code: 2
[taskcluster 2021-04-02T13:54:56.408Z] User Time: 711.489ms
[taskcluster 2021-04-02T13:54:56.408Z] Kernel Time: 90.947ms
[taskcluster 2021-04-02T13:54:56.408Z] Wall Time: 602.315497ms
[taskcluster 2021-04-02T13:54:56.408Z] Result: FAILED
[taskcluster 2021-04-02T13:54:56.408Z] === Task Finished ===
[taskcluster 2021-04-02T13:54:56.408Z] Task Duration: 603.25855ms
[taskcluster:error] Uploading error artifact public/logs from file logs with message "Could not read directory '/home/cltbld/tasks/task_161737169447753/logs'", reason "file-missing-on-worker" and expiry 2022-04-02T12:20:15.001Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read directory '/home/cltbld/tasks/task_161737169447753/logs'
[taskcluster:error] Uploading error artifact public/test_info from file build/blobber_upload_dir with message "Could not read directory '/home/cltbld/tasks/task_161737169447753/build/blobber_upload_dir'", reason "file-missing-on-worker" and expiry 2022-04-02T12:20:15.001Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read directory '/home/cltbld/tasks/task_161737169447753/build/blobber_upload_dir'
[taskcluster 2021-04-02T13:54:56.667Z] Uploading redirect artifact public/logs/live.log to URL https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Yqfkf3LTQXKIqrx6Fw-9ow/runs/0/artifacts/public%2Flogs%2Flive_backing.log with mime type "text/plain; charset=utf-8" and expiry 2022-04-02T12:20:15.001Z
[taskcluster:error] exit status 2
[taskcluster:error] file-missing-on-worker: Could not read directory '/home/cltbld/tasks/task_161737169447753/logs'
[taskcluster:error] file-missing-on-worker: Could not read directory '/home/cltbld/tasks/task_161737169447753/build/blobber_upload_dir'

Mark can you please take a look? ALso, should the component be changed given these are not only on Win?

Flags: needinfo?(mcornmesser)
Whiteboard: [stockwell unknown] → [stockwell needswork:owner]

The "[taskcluster:error] file-missing-on-worker: Could not read directory '/..." is not the issue that is causing the failure but a result of the of the issue. In the log snippet in the above comment the problem is:

[task 2021-04-02T13:54:56.203Z] executing ['/usr/bin/python3', '-u', 'mozharness/scripts/raptor_script.py', '--cfg', 'mozharness/configs/raptor/linux_config.py', '--setpref=extensions.logging.enabled=true', '--no-conditioned-profile', '--test=raptor-wasm-misc', '--project=autoland', '--download-symbols', 'ondemand']
[task 2021-04-02T13:54:56.392Z] Usage: raptor_script.py [options]
[task 2021-04-02T13:54:56.392Z]
[task 2021-04-02T13:54:56.392Z] raptor_script.py: error: no such option: --no-conditioned-profile

The no such file error could be caused by an unrelated issue across the platforms, but It looks like many are hitting the max time , so there maybe a couple different things that need to be untangled.

jmaher: thoughts?

Flags: needinfo?(mcornmesser) → needinfo?(jmaher)
Flags: needinfo?(jmaher)
See Also: → 1745812

Hi Mark, I moved the assign on this from :grenade to you. I was looking at this because we've seen various failures with generic-worker failing on lost/corrupt files on macos and linux. But those are mostly with the cache files and not the logs directory.

This failure decreased and then about a month passed without it (or at least not matched to this bug), but then it appeared once this week (summary says 2, but only one is found with https://treeherder.mozilla.org/intermittent-failures/bugdetails?startday=2022-04-11&endday=2022-04-18&tree=trunk&bug=1585161).

Maybe this is a filesystem/generic-worker problem and not a problem with the task? This one records failing in 773 ms with the failure message "Could not read directory 'Z:\task_164997000655334\logs'" and so maybe it is failing to create that directory at task startup?

https://treeherder.mozilla.org/logviewer?job_id=374549397&repo=autoland

[taskcluster 2022-04-14T21:28:58.350Z] [mounts] Granting task_164997000655334 full control of file 'Z:\task_164997000655334\fetch-content'
[taskcluster 2022-04-14T21:28:58.365Z] Executing command 0: C:/mozilla-build/python3/python3.exe run-task -- c:\mozilla-build\python3\python3.exe -u mozharness\scripts\desktop_unittest.py --cfg mozharness\configs\unittests\win_unittest.py --setpref=webgl.out-of-process=false --setpref=security.sandbox.content.headless=false --setpref=webgl.out-of-process=true --setpref=security.sandbox.content.headless=true --setpref=fission.autostart=true --mochitest-suite=mochitest-webgl1-ext --setpref=media.peerconnection.mtransport_process=false --setpref=network.process.enabled=false --setpref=layers.d3d11.enable-blacklist=false --download-symbols true --total-chunk=1 --this-chunk=1
Z:\task_164997000655334>C:/mozilla-build/python3/python3.exe run-task -- c:\mozilla-build\python3\python3.exe -u mozharness\scripts\desktop_unittest.py --cfg mozharness\configs\unittests\win_unittest.py --setpref=webgl.out-of-process=false --setpref=security.sandbox.content.headless=false --setpref=webgl.out-of-process=true --setpref=security.sandbox.content.headless=true --setpref=fission.autostart=true --mochitest-suite=mochitest-webgl1-ext --setpref=media.peerconnection.mtransport_process=false --setpref=network.process.enabled=false --setpref=layers.d3d11.enable-blacklist=false --download-symbols true --total-chunk=1 --this-chunk=1 
[taskcluster 2022-04-14T21:28:59.141Z]    Exit Code: 3221226091
[taskcluster 2022-04-14T21:28:59.141Z]    User Time: 0s
[taskcluster 2022-04-14T21:28:59.141Z]  Kernel Time: 15.625ms
[taskcluster 2022-04-14T21:28:59.141Z]    Wall Time: 773.001ms
[taskcluster 2022-04-14T21:28:59.141Z]       Result: FAILED
[taskcluster 2022-04-14T21:28:59.141Z] === Task Finished ===
[taskcluster 2022-04-14T21:28:59.141Z] Task Duration: 776.0005ms
[taskcluster:error] Uploading error artifact public/logs from file logs with message "Could not read directory 'Z:\\task_164997000655334\\logs'", reason "file-missing-on-worker" and expiry 2023-04-14T21:00:53.591Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read directory 'Z:\task_164997000655334\logs'
[taskcluster:error] Uploading error artifact public/test_info from file build/blobber_upload_dir with message "Could not read directory 'Z:\\task_164997000655334\\build\\blobber_upload_dir'", reason "file-missing-on-worker" and expiry 2023-04-14T21:00:53.591Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read directory 'Z:\task_164997000655334\build\blobber_upload_dir'
[taskcluster 2022-04-14T21:28:59.341Z] Uploading link artifact public/logs/live.log to artifact public/logs/live_backing.log with expiry 2023-04-14T21:00:53.591Z
[taskcluster:error] exit status 3221226091
Assignee: rthijssen → mcornmesser
Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.