1667826 - T-W1064-MS-470 fails all tasks

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Windows OS, task)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Example log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=316899287&repo=mozilla-central&lineNumber=44

[taskcluster 2020-09-28T11:58:23.747Z] Worker Type (releng-hardware/gecko-t-win10-64-1803-hw) settings:
[taskcluster 2020-09-28T11:58:23.747Z] {
[taskcluster 2020-09-28T11:58:23.747Z] "config": {
[taskcluster 2020-09-28T11:58:23.747Z] "deploymentId": ""
[taskcluster 2020-09-28T11:58:23.747Z] },
[taskcluster 2020-09-28T11:58:23.747Z] "generic-worker": {
[taskcluster 2020-09-28T11:58:23.747Z] "engine": "multiuser",
[taskcluster 2020-09-28T11:58:23.747Z] "go-arch": "amd64",
[taskcluster 2020-09-28T11:58:23.747Z] "go-os": "windows",
[taskcluster 2020-09-28T11:58:23.747Z] "go-version": "go1.10.8",
[taskcluster 2020-09-28T11:58:23.747Z] "release": "https://github.com/taskcluster/generic-worker/releases/tag/v16.2.0",
[taskcluster 2020-09-28T11:58:23.747Z] "revision": "b321a877c8f1616d8912399f599c325171f15e95",
[taskcluster 2020-09-28T11:58:23.747Z] "source": "https://github.com/taskcluster/generic-worker/commits/b321a877c8f1616d8912399f599c325171f15e95",
[taskcluster 2020-09-28T11:58:23.747Z] "version": "16.2.0"
[taskcluster 2020-09-28T11:58:23.747Z] }
[taskcluster 2020-09-28T11:58:23.747Z] }
[taskcluster 2020-09-28T11:58:23.747Z] Task ID: Aj724BHsQdi__YKQGQKr-g
[taskcluster 2020-09-28T11:58:23.747Z] === Task Starting ===
[taskcluster 2020-09-28T11:58:24.518Z] Uploading redirect artifact public/logs/live.log to URL https://firefoxci-websocktunnel.services.mozilla.com/mdc2.T-W1064-MS-470.60099/log/TiHFnvdXQa2FRc-gJ2cXdw with mime type "text/plain; charset=utf-8" and expiry 2020-09-28T13:03:24.417Z
[taskcluster 2020-09-28T11:58:24.586Z] [mounts] Downloading task ZP-HREVNSKSpClj_oISG-w artifact public/build/mozharness.zip to C:\downloads\M-ZviUXRRgqOV8MbIr442w
[taskcluster 2020-09-28T11:58:26.142Z] [mounts] Downloaded 3910540 bytes with SHA256 610628ebc2fd728e48c7de7298d3ac5769b74478e03f9baca3f8bec65a8391d0 from task ZP-HREVNSKSpClj_oISG-w artifact public/build/mozharness.zip to C:\downloads\M-ZviUXRRgqOV8MbIr442w
[taskcluster:warn 2020-09-28T11:58:26.142Z] [mounts] Download C:\downloads\M-ZviUXRRgqOV8MbIr442w of task ZP-HREVNSKSpClj_oISG-w artifact public/build/mozharness.zip has SHA256 610628ebc2fd728e48c7de7298d3ac5769b74478e03f9baca3f8bec65a8391d0 but task payload does not declare a required value, so content authenticity cannot be verified
[taskcluster 2020-09-28T11:58:26.142Z] [mounts] Creating directory C:\Users\task_1601294019 with permissions 0700
[taskcluster 2020-09-28T11:58:26.142Z] [mounts] Extracting zip file C:\downloads\M-ZviUXRRgqOV8MbIr442w to 'C:\Users\task_1601294019'
[taskcluster 2020-09-28T11:58:26.294Z] [mounts] Granting task_1601294019 full control of directory 'C:\Users\task_1601294019'
[taskcluster:warn 2020-09-28T11:58:26.572Z] [mounts] No SHA256 specified in task mounts for urlcontent:https://hg.mozilla.org/mozilla-central/raw-file/9c47bca207be7bdaf827d73a24332e1c2aec2545/taskcluster/scripts/run-task - SHA256 from downloaded file C:\downloads\emrKmUGaSx6L6SeMNNxD4g is 826d7cf339abd7b732b6d0c4a874af15796a486ad653ac257f00583c95a68482.
[taskcluster 2020-09-28T11:58:26.572Z] [mounts] Creating directory C:\Users\task_1601294019 with permissions 0700
[taskcluster 2020-09-28T11:58:26.572Z] [mounts] Copying C:\downloads\emrKmUGaSx6L6SeMNNxD4g to C:\Users\task_1601294019\run-task
[taskcluster 2020-09-28T11:58:26.599Z] [mounts] Granting task_1601294019 full control of file 'C:\Users\task_1601294019\run-task'
[taskcluster:warn 2020-09-28T11:58:26.610Z] [mounts] No SHA256 specified in task mounts for urlcontent:https://hg.mozilla.org/mozilla-central/raw-file/9c47bca207be7bdaf827d73a24332e1c2aec2545/taskcluster/scripts/misc/fetch-content - SHA256 from downloaded file C:\downloads\KLAHOzGASFybsWhREXa-gQ is c3e084f90d9b034149c9be7ada12fc0ee41c0e66080ce463654e6ab3ff634c0d.
[taskcluster 2020-09-28T11:58:26.610Z] [mounts] Creating directory C:\Users\task_1601294019 with permissions 0700
[taskcluster 2020-09-28T11:58:26.610Z] [mounts] Copying C:\downloads\KLAHOzGASFybsWhREXa-gQ to C:\Users\task_1601294019\fetch-content
[taskcluster 2020-09-28T11:58:26.614Z] [mounts] Granting task_1601294019 full control of file 'C:\Users\task_1601294019\fetch-content'
[taskcluster 2020-09-28T11:58:26.625Z] Executing command 0: C:/mozilla-build/python3/python3.exe run-task -- c:\mozilla-build\python\python.exe -u mozharness\scripts\raptor_script.py --cfg mozharness\configs\raptor\windows_config.py --setpref=extensions.logging.enabled=true --app=firefox --cold --test=raptor-tp6-instagram-firefox --project=mozilla-central --download-symbols ondemand
C:\Users\task_1601294019>C:/mozilla-build/python3/python3.exe run-task -- c:\mozilla-build\python\python.exe -u mozharness\scripts\raptor_script.py --cfg mozharness\configs\raptor\windows_config.py --setpref=extensions.logging.enabled=true --app=firefox --cold --test=raptor-tp6-instagram-firefox --project=mozilla-central --download-symbols ondemand
[setup 2020-09-28T11:58:26.804Z] run-task started in C:\Users\task_1601294019
[setup 2020-09-28T11:58:26.805Z] MOZ_FETCHES_DIR is C:/Users/task_1601294019/fetches
[fetches 2020-09-28T11:58:26.805Z] fetching artifacts
[fetches 2020-09-28T11:58:26.815Z] executing ['C:\mozilla-build\python3\python3.exe', '-u', 'fetch-content', 'task-artifacts']
[fetches 2020-09-28T11:58:26.923Z] Traceback (most recent call last):
[fetches 2020-09-28T11:58:26.923Z] File "fetch-content", line 30, in <module>
[fetches 2020-09-28T11:58:26.923Z] import zstandard
[fetches 2020-09-28T11:58:26.923Z] ValueError: source code string cannot contain null bytes
[taskcluster 2020-09-28T11:58:26.957Z] Exit Code: 1
[taskcluster 2020-09-28T11:58:26.957Z] User Time: 0s
[taskcluster 2020-09-28T11:58:26.957Z] Kernel Time: 15.625ms
[taskcluster 2020-09-28T11:58:26.957Z] Wall Time: 331.9564ms
[taskcluster 2020-09-28T11:58:26.957Z] Result: FAILED
[taskcluster 2020-09-28T11:58:26.957Z] === Task Finished ===
[taskcluster 2020-09-28T11:58:26.957Z] Task Duration: 331.9564ms
[taskcluster:error] Uploading error artifact public/logs from file logs with message "Could not read directory 'C:\Users\task_1601294019\logs'", reason "file-missing-on-worker" and expiry 2021-09-28T09:51:23.416Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read directory 'C:\Users\task_1601294019\logs'
[taskcluster:error] Uploading error artifact public/test_info from file build/blobber_upload_dir with message "Could not read directory 'C:\Users\task_1601294019\build\blobber_upload_dir'", reason "file-missing-on-worker" and expiry 2021-09-28T09:51:23.416Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read directory 'C:\Users\task_1601294019\build\blobber_upload_dir'
[taskcluster 2020-09-28T11:58:27.270Z] Uploading redirect artifact public/logs/live.log to URL https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Aj724BHsQdi__YKQGQKr-g/runs/0/artifacts/public/logs/live_backing.log with mime type "text/plain; charset=utf-8" and expiry 2021-09-28T09:51:23.416Z
[taskcluster:error] exit status 1
[taskcluster:error] file-missing-on-worker: Could not read directory 'C:\Users\task_1601294019\logs'

Comment 1

[Mark Cornmesser [:markco]]

That is odd for a single node to be in this kind of state. I am in progress of redeploying these nodes. After this node has been re-image I will on unquarantine it and keep an eye on the test results for it.

Comment 2

[Mark Cornmesser [:markco]]

After re-imaging the node appears to be working and passing tests.