1921446 - Intermittent [taskcluster:error] Pulling docker image has failed.

Status: RESOLVED WORKSFORME

(Taskcluster :: Services, defect)

Description

[Julien Cristau [:jcristau] (back August 18)]

https://treeherder.mozilla.org/jobs?repo=mozilla-central&resultStatus=testfailed%2Cbusted%2Cexception&revision=a44bb440300f5b5f0de217495495df33b4d22fca&searchStr=linux had a bunch of failures from test jobs that couldn't pull their docker image, e.g. https://firefoxci.taskcluster-artifacts.net/AvE6x0aTSpuJ3YLLIhlRzA/0/public/logs/live_backing.log

[taskcluster 2024-09-27 10:42:00.342Z] Task ID: AvE6x0aTSpuJ3YLLIhlRzA
[taskcluster 2024-09-27 10:42:00.342Z] Worker ID: 1796653598321095994
[taskcluster 2024-09-27 10:42:00.342Z] Worker Group: us-central1-a
[taskcluster 2024-09-27 10:42:00.342Z] Worker Node Type: projects/887720501152/machineTypes/n2-standard-4
[taskcluster 2024-09-27 10:42:00.342Z] Worker Type: t-linux-xlarge-noscratch-gcp
[taskcluster 2024-09-27 10:42:00.342Z] Public IP: 34.42.224.19
[taskcluster 2024-09-27 10:42:00.342Z] Hostname: gecko-t-t-linux-xlarge-noscratch-gcp-oo9w-ts6sesrzcyj0dfwaq
[taskcluster 2024-09-27 10:42:00.342Z] using cache "gecko-level-3-checkouts-hg58-v3-df928252a170103b7408" -> /builds/worker/checkouts
[taskcluster 2024-09-27 10:42:00.342Z] using cache "gecko-level-3-tooltool-cache-v3-df928252a170103b7408" -> /builds/worker/tooltool-cache

[taskcluster 2024-09-27 10:42:01.215Z] Downloading artifact "public/image.tar.zst" from task ID: OOWZC4sZSj-kyp_37ID7Rw.
[taskcluster 2024-09-27 10:42:06.216Z] Download Progress: 55.43%
[taskcluster 2024-09-27 10:42:10.246Z] Downloaded artifact successfully.
[taskcluster 2024-09-27 10:42:10.247Z] Downloaded 1264.121 mb
[taskcluster 2024-09-27 10:42:10.249Z] Decompressing downloaded image
[taskcluster 2024-09-27 10:42:14.223Z] Downloading artifact "public/image.tar.zst" from task ID: OOWZC4sZSj-kyp_37ID7Rw.
[taskcluster 2024-09-27 10:42:19.225Z] Download Progress: 42.47%
[taskcluster 2024-09-27 10:42:24.226Z] Download Progress: 81.33%
[taskcluster 2024-09-27 10:42:27.002Z] Downloaded artifact successfully.
[taskcluster 2024-09-27 10:42:27.003Z] Downloaded 1264.121 mb
[taskcluster 2024-09-27 10:42:27.004Z] Decompressing downloaded image
[taskcluster 2024-09-27 10:42:30.471Z] Downloading artifact "public/image.tar.zst" from task ID: OOWZC4sZSj-kyp_37ID7Rw.
[taskcluster 2024-09-27 10:42:35.474Z] Download Progress: 47.46%
[taskcluster 2024-09-27 10:42:40.473Z] Download Progress: 99.64%
[taskcluster 2024-09-27 10:42:40.507Z] Downloaded artifact successfully.
[taskcluster 2024-09-27 10:42:40.507Z] Downloaded 1264.121 mb
[taskcluster 2024-09-27 10:42:40.508Z] Decompressing downloaded image
[taskcluster 2024-09-27 10:42:43.246Z] Downloading artifact "public/image.tar.zst" from task ID: OOWZC4sZSj-kyp_37ID7Rw.
[taskcluster 2024-09-27 10:42:48.249Z] Download Progress: 37.04%
[taskcluster 2024-09-27 10:42:53.249Z] Download Progress: 69.70%

[taskcluster:error] Error downloading "public/image.tar.zst" from task ID "OOWZC4sZSj-kyp_37ID7Rw". Error: Expected size is '1325526526' but received '1288952224' Next Attempt in: 14701.45 ms.
[taskcluster 2024-09-27 10:43:18.065Z] Download Progress: 45.19%
[taskcluster 2024-09-27 10:43:23.065Z] Download Progress: 87.05%
[taskcluster 2024-09-27 10:43:25.409Z] Downloaded artifact successfully.
[taskcluster 2024-09-27 10:43:25.409Z] Downloaded 1264.121 mb
[taskcluster 2024-09-27 10:43:25.409Z] Decompressing downloaded image

[taskcluster:error] Pulling docker image has failed.

[taskcluster:error] Error: Error loading docker image. Could not decompress image file.Exit Code: 1 Errors: 
DyICIQ/image.tar.zst : 0 MB...     

DyICIQ/image.tar.zst : 82 MB...     

DyICIQ/image.tar.zst : 150 MB...     

DyICIQ/image.tar.zst : 282 MB...     

DyICIQ/image.tar.zst : 382 MB...     

DyICIQ/image.tar.zst : 465 MB...     

DyICIQ/image.tar.zst : 558 MB...     

DyICIQ/image.tar.zst : 656 MB...     

DyICIQ/image.tar.zst : 775 MB...     

DyICIQ/image.tar.zst : 868 MB...     

DyICIQ/image.tar.zst : 956 MB...     

DyICIQ/image.tar.zst : 1044 MB...     

DyICIQ/image.tar.zst : 1147 MB...     

DyICIQ/image.tar.zst : 1234 MB...     
DyICIQ/image.tar.zst : Decoding error (36) : Corrupted block detected 

[taskcluster 2024-09-27 10:43:28.096Z] Unsuccessful task run with exit code: -1 completed in 87.755 seconds

Comment 1

[Julien Cristau [:jcristau] (back August 18)]

The artifact in question is https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/OOWZC4sZSj-kyp_37ID7Rw/runs/0/artifacts/public%2Fimage.tar.zst aka https://firefoxci.taskcluster-artifacts.net/OOWZC4sZSj-kyp_37ID7Rw/0/public/image.tar.zst aka https://storage.googleapis.com/firefoxcitc-public-artifacts/OOWZC4sZSj-kyp_37ID7Rw/0/public/image.tar.zst aka gs://firefoxcitc-public-artifacts/OOWZC4sZSj-kyp_37ID7Rw/0/public/image.tar.zst
When going through the CDN I intermittently got an error back when trying to download locally or in a gcp instance, e.g.

jcristau@instance-20240927-111232:~$ curl -L -o image.tar.zst https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/OOWZC4sZSj-kyp_37ID7Rw/runs/0/artifacts/public%2Fimage.tar.zst 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   127  100   127    0     0   1292      0 --:--:-- --:--:-- --:--:--  1295
 37 1264M   37  476M    0     0  73.1M      0  0:00:17  0:00:06  0:00:11 70.7M
curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)

Other times it worked fine (from the same client), getting 1325526526 bytes and sha256sum matching f36087105c1c9d80b04c11298cbcf79f401ff9f935d395d3e0a0847a0cb4dd23.

Comment 2

[Intermittent Failures Robot]

207 failures in 742 pushes (0.279 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 207

Platform and build breakdown:

• linux1804-64-ccov-qr: 48
  • opt: 48
• linux1804-64-tsan-qr: 156
  • opt: 156
• fenix-ui-test: 1
  • opt: 1
• windows11-64-2009-ccov-qr: 1
  • opt: 1
• linux1804-64-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1921446&startday=2024-09-27&endday=2024-09-27&tree=all

Comment 3

[Intermittent Failures Robot]

207 failures in 4889 pushes (0.042 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 207

Platform and build breakdown:

• linux1804-64-ccov-qr: 48
  • opt: 48
• linux1804-64-tsan-qr: 156
  • opt: 156
• fenix-ui-test: 1
  • opt: 1
• windows11-64-2009-ccov-qr: 1
  • opt: 1
• linux1804-64-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1921446&startday=2024-09-23&endday=2024-09-29&tree=all

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2] (away July 24th - August 16th)]

Julian, do we have to handle that separately beside bug 1635386 because it's some kind of other regression?

Comment 5

[Julien Cristau [:jcristau] (back August 18)]

There was a specific issue with that one image on a subset of workers for a short period last week. It seems to have gone away on its own.