1932466 - Perma Exception: Download failed, no more retries! | Download failed: size mismatch

Status: NEW

(Release Engineering :: General, defect, P2)

Description

[Treeherder Bug Filer]

Filed by: smolnar [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=483525355&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/ZBhuyN0qRUGX_dPNbQvk1w/runs/0/artifacts/public/logs/live_backing.log

---

Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/LcGMJ5QdT8OUk0kqVu9GXg/artifacts/public/build/clang.tar.zst
[fetches 2024-11-20T18:39:35.237Z] https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/LcGMJ5QdT8OUk0kqVu9GXg/artifacts/public/build/clang.tar.zst resolved to 7864272 bytes with sha256 97978193086485ea780444136bbeaf3dcb9b4ebce3717fdea555601272c16d31 in 0.403s
[fetches 2024-11-20T18:39:35.238Z] Download failed: size mismatch on https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/LcGMJ5QdT8OUk0kqVu9GXg/artifacts/public/build/clang.tar.zst: wanted 472193866; got 7864272
[fetches 2024-11-20T18:39:35.239Z] Traceback (most recent call last):
[fetches 2024-11-20T18:39:35.239Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/run-task/fetch-content", line 978, in <module>
[fetches 2024-11-20T18:39:35.239Z]     sys.exit(main())
[fetches 2024-11-20T18:39:35.240Z]              ^^^^^^
[fetches 2024-11-20T18:39:35.240Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/run-task/fetch-content", line 974, in main
[fetches 2024-11-20T18:39:35.240Z]     return args.func(args)
[fetches 2024-11-20T18:39:35.240Z]            ^^^^^^^^^^^^^^^
[fetches 2024-11-20T18:39:35.240Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/run-task/fetch-content", line 880, in command_task_artifacts
[fetches 2024-11-20T18:39:35.240Z]     fetch_urls(downloads)
[fetches 2024-11-20T18:39:35.240Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/run-task/fetch-content", line 612, in fetch_urls
[fetches 2024-11-20T18:39:35.240Z]     f.result()
[fetches 2024-11-20T18:39:35.240Z]   File "/usr/lib/python3.11/concurrent/futures/_base.py", line 456, in result
[fetches 2024-11-20T18:39:35.240Z]     return self.__get_result()
[fetches 2024-11-20T18:39:35.240Z]            ^^^^^^^^^^^^^^^^^^^
[fetches 2024-11-20T18:39:35.240Z]   File "/usr/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
[fetches 2024-11-20T18:39:35.240Z]     raise self._exception
[fetches 2024-11-20T18:39:35.240Z]   File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
[fetches 2024-11-20T18:39:35.240Z]     result = self.fn(*self.args, **self.kwargs)
[fetches 2024-11-20T18:39:35.240Z]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[fetches 2024-11-20T18:39:35.240Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/run-task/fetch-content", line 590, in fetch_and_extract
[fetches 2024-11-20T18:39:35.240Z]     download_to_path(url, dest_path, sha256=sha256, size=size)
[fetches 2024-11-20T18:39:35.240Z]   File "/builds/worker/checkouts/gecko/third_party/python/taskcluster_taskgraph/taskgraph/run-task/fetch-content", line 279, in download_to_path
[fetches 2024-11-20T18:39:35.240Z]     raise Exception("Download failed, no more retries!")
[fetches 2024-11-20T18:39:35.240Z] Exception: Download failed, no more retries!
[taskcluster 2024-11-20 18:39:35.563Z] === Task Finished ===
[taskcluster 2024-11-20 18:39:35.568Z] Artifact "public/logs" not found at "/builds/worker/logs/": (HTTP code 404) no such container - Could not find the file /builds/worker/logs/ in container c4b219191ea54b82f25e5c659d109eb91fa455585c9507413eaceef40c799489 
[taskcluster 2024-11-20 18:39:35.571Z] Artifact "public/build" not found at "/builds/worker/artifacts/": (HTTP code 404) no such container - Could not find the file /builds/worker/artifacts/ in container c4b219191ea54b82f25e5c659d109eb91fa455585c9507413eaceef40c799489 
[taskcluster 2024-11-20 18:39:35.573Z] Artifact "public/cidata/target.crashreporter-symbols-full.tar.zst" not found at "/builds/worker/cidata/target.crashreporter-symbols-full.tar.zst": (HTTP code 404) no such container - Could not find the file /builds/worker/cidata/target.crashreporter-symbols-full.tar.zst in container c4b219191ea54b82f25e5c659d109eb91fa455585c9507413eaceef40c799489 
[taskcluster 2024-11-20 18:39:35.575Z] Artifact "public/cidata/sccache.log" not found at "/builds/worker/cidata/sccache.log": (HTTP code 404) no such container - Could not find the file /builds/worker/cidata/sccache.log in container c4b219191ea54b82f25e5c659d109eb91fa455585c9507413eaceef40c799489 
[taskcluster 2024-11-20 18:39:35.577Z] Artifact "public/cidata/sccache-stats.json" not found at "/builds/worker/cidata/sccache-stats.json": (HTTP code 404) no such container - Could not find the file /builds/worker/cidata/sccache-stats.json in container c4b219191ea54b82f25e5c659d109eb91fa455585c9507413eaceef40c799489 
[taskcluster 2024-11-20 18:39:35.664Z] Unsuccessful task run with exit code: 1 completed in 499.877 seconds

Comment 1

[[e-mail] (:bhearsum)]

As far as I can tell, the downloads come out with different sizes each time, which seems to point toward either some network wonkiness, or some queue service wonkiness. I'm not aware of anything changing in either of those places, though. I'm able to get the correct sizes downloading locally.

Comment 2

[Cosmin Sabou [:CosminS]]

With having green reruns of the affected builds we decided to reopen the trees, it seems we're in the clear now.

Comment 3

[[e-mail] (:bhearsum)]

We dug through some cloud cdn logs with jbuck and found some interesting requests where we're getting 200s back with much smaller sizes than expected:

[{
"logName": "projects/moz-fx-taskcluster-prod-4b87/logs/requests",
"resource": {
"type": "http_load_balancer",
"labels": {
"url_map_name": "taskcluster-firefoxcitc-artifacts-gcs-cdn",
"forwarding_rule_name": "taskcluster-firefoxcitc-artifacts-gcs-cdn-https-ipv4",
"backend_service_name": "",
"target_proxy_name": "taskcluster-firefoxcitc-artifacts-gcs-cdn",
"zone": "global",
"project_id": "moz-fx-taskcluster-prod-4b87"
}
},
"textPayload": null,
"jsonpayload_type_loadbalancerlogentry": {
"statusdetails": "cache_lookup_failed_after_partial_response",
"_type": "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry",
"remoteip": "34.31.2.242",
"cacheid": "CBF",
"parentinsertid": null,
"cachedecision": ["RESPONSE_HAS_CACHE_CONTROL", "RESPONSE_CACHE_CONTROL_PUBLIC", "RESPONSE_HAS_ETAG", "RESPONSE_HAS_LAST_MODIFIED", "RESPONSE_HAS_EXPIRES", "RESPONSE_HAS_CONTENT_TYPE", "CACHE_MODE_FORCE_CACHE_ALL"],
"backendtargetprojectnumber": "projects/90111867433"
},
"timestamp": "2024-11-20 19:24:07.573867 UTC",
"receiveTimestamp": "2024-11-20 19:24:08.344779 UTC",
"severity": "INFO",
"insertId": "1bglqb1f9ryq0i",
"httpRequest": {
"requestMethod": "GET",
"requestUrl": "https://firefoxci.taskcluster-artifacts.net/LcGMJ5QdT8OUk0kqVu9GXg/0/public/build/clang.tar.zst",
"requestSize": "190",
"status": "200",
"responseSize": "6292192",
"userAgent": "Python-urllib/3.11",
"remoteIp": "34.31.2.242",
"serverIp": null,
"referer": null,
"cacheLookup": "true",
"cacheHit": "true",
"cacheValidatedWithOriginServer": null,
"cacheFillBytes": null,
"protocol": null,
"latency": "0.030741"
},
"operation": null,
"trace": "projects/moz-fx-taskcluster-prod-4b87/traces/846f339d822b94b814ac1962f85ae003",
"spanId": "2a5ea9aaa5cb394d",
"traceSampled": null,
"sourceLocation": null,
"split": null
}

jbuck suggests we file a GCP support ticket for this.

Comment 4

[Intermittent Failures Robot]

27 failures in 429 pushes (0.063 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 13
• mozilla-central: 11
• try: 3

Platform and build breakdown:

• linux64: 6
  • debug: 5
  • asan: 1
• android-5-0-armv7: 2
  • opt: 2
• linux32-shippable: 1
  • opt: 1
• windows2012-32-shippable: 3
  • opt: 3
• osx-nightlyasrelease: 1
  • opt: 1
• android-5-0-x86_64-shippable: 4
  • opt: 4
• osx-cross-aarch64: 1
  • debug: 1
• osx-shippable: 2
  • opt: 2
• android-5-0-x86: 1
  • opt: 1
• win64-nightlyasrelease: 1
  • opt: 1
• android-5-0-x86_64: 2
  • opt: 2
• windows2012-64-shippable: 1
  • opt: 1
• android-5-0-aarch64: 1
  • opt: 1
• fenix-android-all: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1932466&startday=2024-11-20&endday=2024-11-20&tree=all

Comment 5

[Intermittent Failures Robot]

157 failures in 366 pushes (0.429 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 157

Platform and build breakdown:

• linux2204-64-wayland: 70
  • opt: 70
• linux1804-64-qr: 87
  • opt: 87

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1932466&startday=2024-11-22&endday=2024-11-22&tree=all

Comment 6

[Intermittent Failures Robot]

184 failures in 3002 pushes (0.061 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 170
• mozilla-central: 11
• try: 3

Platform and build breakdown:

• linux64: 6
  • debug: 5
  • asan: 1
• android-5-0-armv7: 2
  • opt: 2
• linux32-shippable: 1
  • opt: 1
• windows2012-32-shippable: 3
  • opt: 3
• osx-nightlyasrelease: 1
  • opt: 1
• android-5-0-x86_64-shippable: 4
  • opt: 4
• osx-cross-aarch64: 1
  • debug: 1
• osx-shippable: 2
  • opt: 2
• android-5-0-x86: 1
  • opt: 1
• win64-nightlyasrelease: 1
  • opt: 1
• android-5-0-x86_64: 2
  • opt: 2
• windows2012-64-shippable: 1
  • opt: 1
• android-5-0-aarch64: 1
  • opt: 1
• fenix-android-all: 1
  • opt: 1
• linux2204-64-wayland: 70
  • opt: 70
• linux1804-64-qr: 87
  • opt: 87

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1932466&startday=2024-11-18&endday=2024-11-24&tree=all