Open Bug 1765463 Opened 3 years ago Updated 4 months ago

Intermittent [taskcluster:error] [mounts] Could not fetch from url http://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/I_B1YcIGSDy_zLbvAAZqfg/artifacts/public/run-task into file downloads/c1Utl7z6RbWsAXhWzmn8Kw: Get https://firefox-ci-tc.services.

Categories

(Taskcluster :: General, defect, P5)

Product:
Taskcluster
Component:
General
Type:
defect

Tracking

(Not tracked)

People

(Reporter: intermittent-bug-filer, Unassigned)

Details

(Keywords: intermittent-failure)

Filed by: mlaza [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=375076637&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/avwOsenMTLuEIutMVxcRCA/runs/0/artifacts/public/logs/live_backing.log

[taskcluster 2022-04-19T23:47:38.149Z] Worker Type (proj-autophone/gecko-t-bitbar-gw-perf-a51) settings:
[taskcluster 2022-04-19T23:47:38.149Z]   {
[taskcluster 2022-04-19T23:47:38.149Z]     "config": {
[taskcluster 2022-04-19T23:47:38.149Z]       "deploymentId": ""
[taskcluster 2022-04-19T23:47:38.149Z]     },
[taskcluster 2022-04-19T23:47:38.149Z]     "generic-worker": {
[taskcluster 2022-04-19T23:47:38.149Z]       "engine": "simple",
[taskcluster 2022-04-19T23:47:38.149Z]       "go-arch": "amd64",
[taskcluster 2022-04-19T23:47:38.149Z]       "go-os": "linux",
[taskcluster 2022-04-19T23:47:38.149Z]       "go-version": "go1.13.7",
[taskcluster 2022-04-19T23:47:38.149Z]       "release": "https://github.com/taskcluster/taskcluster/releases/tag/v36.0.0",
[taskcluster 2022-04-19T23:47:38.149Z]       "revision": "b9cb11293f106f4330a42853fa3d2d15a9f13200",
[taskcluster 2022-04-19T23:47:38.149Z]       "source": "https://github.com/taskcluster/taskcluster/commits/b9cb11293f106f4330a42853fa3d2d15a9f13200",
[taskcluster 2022-04-19T23:47:38.149Z]       "version": "36.0.0"
[taskcluster 2022-04-19T23:47:38.149Z]     }
[taskcluster 2022-04-19T23:47:38.149Z]   }
[taskcluster 2022-04-19T23:47:38.149Z] Task ID: avwOsenMTLuEIutMVxcRCA
[taskcluster 2022-04-19T23:47:38.149Z] === Task Starting ===
[taskcluster 2022-04-19T23:47:38.438Z] Uploading redirect artifact public/logs/live.log to URL https://firefoxci-websocktunnel.services.mozilla.com/bitbar.a51-05.60099/log/YDpzdqRWTAy6mSZ11I0p_g with mime type "text/plain; charset=utf-8" and expiry 2022-04-20T01:02:38.388Z
[taskcluster 2022-04-19T23:47:38.825Z] [mounts] Downloading url https://hg.mozilla.org/mozilla-central/raw-file/ba973417adb8c738b0ab57328fd28133c77bcf5b/taskcluster/scripts/tester/test-linux.sh to downloads/VlFVk4KmSTOGdYU0NSdRhQ
[taskcluster:warn 2022-04-19T23:47:58.826Z] [mounts] Download of url https://hg.mozilla.org/mozilla-central/raw-file/ba973417adb8c738b0ab57328fd28133c77bcf5b/taskcluster/scripts/tester/test-linux.sh failed on this attempt: Get https://hg.mozilla.org/mozilla-central/raw-file/ba973417adb8c738b0ab57328fd28133c77bcf5b/taskcluster/scripts/tester/test-linux.sh: dial tcp: lookup hg.mozilla.org on 127.0.0.11:53: read udp 127.0.0.1:43552->127.0.0.11:53: i/o timeout
[taskcluster 2022-04-19T23:47:59.311Z] [mounts] Downloading url https://hg.mozilla.org/mozilla-central/raw-file/ba973417adb8c738b0ab57328fd28133c77bcf5b/taskcluster/scripts/tester/test-linux.sh to downloads/VlFVk4KmSTOGdYU0NSdRhQ
[taskcluster:warn 2022-04-19T23:48:19.312Z] [mounts] Download of url https://hg.mozilla.org/mozilla-central/raw-file/ba973417adb8c738b0ab57328fd28133c77bcf5b/taskcluster/scripts/tester/test-linux.sh failed on this attempt: Get https://hg.mozilla.org/mozilla-central/raw-file/ba973417adb8c738b0ab57328fd28133c77bcf5b/taskcluster/scripts/tester/test-linux.sh: dial tcp: lookup hg.mozilla.org on 127.0.0.11:53: read udp 127.0.0.1:56113->127.0.0.11:53: i/o timeout
[taskcluster 2022-04-19T23:48:19.842Z] [mounts] Downloading url https://hg.mozilla.org/mozilla-central/raw-file/ba973417adb8c738b0ab57328fd28133c77bcf5b/taskcluster/scripts/tester/test-linux.sh to downloads/VlFVk4KmSTOGdYU0NSdRhQ
[taskcluster:warn 2022-04-19T23:48:39.843Z] [mounts] Download of url https://hg.mozilla.org/mozilla-central/raw-file/ba973417adb8c738b0ab57328fd28133c77bcf5b/taskcluster/scripts/tester/test-linux.sh failed on this attempt: Get https://hg.mozilla.org/mozilla-central/raw-file/ba973417adb8c738b0ab57328fd28133c77bcf5b/taskcluster/scripts/tester/test-linux.sh: dial tcp: lookup hg.mozilla.org on 127.0.0.11:53: read udp 127.0.0.1:54237->127.0.0.11:53: i/o timeout
[taskcluster 2022-04-19T23:48:40.881Z] [mounts] Downloading url https://hg.mozilla.org/mozilla-central/raw-file/ba973417adb8c738b0ab57328fd28133c77bcf5b/taskcluster/scripts/tester/test-linux.sh to downloads/VlFVk4KmSTOGdYU0NSdRhQ
[taskcluster 2022-04-19T23:48:52.419Z] [mounts] Downloaded 8967 bytes with SHA256 31cb2e6a6c81f143c0685774456ad846c482c708aa6c64474d79d644fbea4634 from url https://hg.mozilla.org/mozilla-central/raw-file/ba973417adb8c738b0ab57328fd28133c77bcf5b/taskcluster/scripts/tester/test-linux.sh to downloads/VlFVk4KmSTOGdYU0NSdRhQ
[taskcluster:warn 2022-04-19T23:48:52.419Z] [mounts] Download downloads/VlFVk4KmSTOGdYU0NSdRhQ of url https://hg.mozilla.org/mozilla-central/raw-file/ba973417adb8c738b0ab57328fd28133c77bcf5b/taskcluster/scripts/tester/test-linux.sh has SHA256 31cb2e6a6c81f143c0685774456ad846c482c708aa6c64474d79d644fbea4634 but task payload does not declare a required value, so content authenticity cannot be verified
[taskcluster 2022-04-19T23:48:52.419Z] [mounts] Creating directory /builds/task_165041203866798 with permissions 0700
[taskcluster 2022-04-19T23:48:52.419Z] [mounts] Copying downloads/VlFVk4KmSTOGdYU0NSdRhQ to /builds/task_165041203866798/test-linux.sh
[taskcluster 2022-04-19T23:48:53.622Z] [mounts] Downloading url http://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/I_B1YcIGSDy_zLbvAAZqfg/artifacts/public/run-task to downloads/c1Utl7z6RbWsAXhWzmn8Kw
[taskcluster:warn 2022-04-20T00:04:39.987Z] [mounts] Download of url http://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/I_B1YcIGSDy_zLbvAAZqfg/artifacts/public/run-task failed on this attempt: Get https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/I_B1YcIGSDy_zLbvAAZqfg/artifacts/public/run-task: read tcp 10.7.205.142:36310->35.190.5.182:443: read: connection timed out
[taskcluster:error] [mounts] Could not fetch from url http://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/I_B1YcIGSDy_zLbvAAZqfg/artifacts/public/run-task into file downloads/c1Utl7z6RbWsAXhWzmn8Kw: Get https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/I_B1YcIGSDy_zLbvAAZqfg/artifacts/public/run-task: read tcp 10.7.205.142:36310->35.190.5.182:443: read: connection timed out
[taskcluster:error] [mounts] Could not fetch from url http://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/I_B1YcIGSDy_zLbvAAZqfg/artifacts/public/run-task into file downloads/c1Utl7z6RbWsAXhWzmn8Kw due to Get https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/I_B1YcIGSDy_zLbvAAZqfg/artifacts/public/run-task: read tcp 10.7.205.142:36310->35.190.5.182:443: read: connection timed out
[taskcluster 2022-04-20T00:04:40.526Z] [taskcluster-proxy] Successfully refreshed taskcluster-proxy credentials: task-client/avwOsenMTLuEIutMVxcRCA/0/on/bitbar/a51-05/until/1650414280.434
[taskcluster 2022-04-20T00:04:40.599Z] Uploading redirect artifact public/logs/live.log to URL https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/avwOsenMTLuEIutMVxcRCA/runs/0/artifacts/public%2Flogs%2Flive_backing.log with mime type "text/plain; charset=utf-8" and expiry 2023-04-19T21:51:21.886Z
[taskcluster:error] [mounts] Get https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/I_B1YcIGSDy_zLbvAAZqfg/artifacts/public/run-task: read tcp 10.7.205.142:36310->35.190.5.182:443: read: connection timed out

Hi Pete, could you please take a look at this recent spike in failures?

Recent failure log -> https://treeherder.mozilla.org/logviewer?job_id=522207361&repo=autoland&lineNumber=86

Flags: needinfo?(pmoore)
Summary: Intermittent [taskcluster:error] [mounts] Could not fetch from url http://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/I_B1YcIGSDy_zLbvAAZqfg/artifacts/public/run-task into file downloads/c1Utl7z6RbWsAXhWzmn8Kw: Get https://firefox-ci-tc.services. → Frequent [taskcluster:error] [mounts] Could not fetch from url http://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/I_B1YcIGSDy_zLbvAAZqfg/artifacts/public/run-task into file downloads/c1Utl7z6RbWsAXhWzmn8Kw: Get https://firefox-ci-tc.services.
Summary: Frequent [taskcluster:error] [mounts] Could not fetch from url http://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/I_B1YcIGSDy_zLbvAAZqfg/artifacts/public/run-task into file downloads/c1Utl7z6RbWsAXhWzmn8Kw: Get https://firefox-ci-tc.services. → Intermittent [taskcluster:error] [mounts] Could not fetch from url http://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/I_B1YcIGSDy_zLbvAAZqfg/artifacts/public/run-task into file downloads/c1Utl7z6RbWsAXhWzmn8Kw: Get https://firefox-ci-tc.services.

Mark, we're seeing a few issues on windows workers, jobs fail as an exception with logs like here and here or they have downloading issues like the one in this bug, recent failure log here, here or here. Link from treeherder.

This started on the 13th when there were a bunch of them and we're terminated a lot of workers to get rid of them but today it happened again on several jobs and workers. Could you please dig into why this is happening? I thought it might be due to a corrupted batch of windows workers but it seems it is happening again and it's pointing to something systemic.

Flags: needinfo?(mcornmesser)

(In reply to Cosmin Sabou [:CosminS] from comment #20)

Mark, we're seeing a few issues on windows workers, jobs fail as an exception with logs like here and here or they have downloading issues like the one in this bug, recent failure log here, here or here. Link from treeherder.

This started on the 13th when there were a bunch of them and we're terminated a lot of workers to get rid of them but today it happened again on several jobs and workers. Could you please dig into why this is happening? I thought it might be due to a corrupted batch of windows workers but it seems it is happening again and it's pointing to something systemic.

Sorry for the delay on my response. Are we still seeing these failures?

Flags: needinfo?(mcornmesser) → needinfo?(csabou)

None lately.

Flags: needinfo?(csabou)
You need to log in before you can comment on or make changes to this bug.