1843393 - KeyError: 'content-length' when fetching certain taskcluster builds

Status: RESOLVED FIXED

(Testing :: mozregression, defect, P2)

Description

[Zeid [:zeid]]

Some taskcluster artifacts no longer have the content-length header in the response, which is causing some errors when fetching builds (example below is a GVE build):

$ mozregression --app=gve --good 2023-07-12 --bad 2023-07-13
 0:00.20 INFO: Getting mozilla-central builds between 2023-07-12 and 2023-07-13
 0:00.70 INFO: Using 409794f954d9150f8da58e49d75898a655aca25d (pushed on 2023-07-12 09:49:40) for date 2023-07-12
 0:00.70 INFO: Using 211b29e869cab1e9d99577edd8b6fa2fed823f77 (pushed on 2023-07-13 09:17:48) for date 2023-07-13
 0:01.80 INFO: Testing good and bad builds to ensure that they are really good and bad...
 0:01.81 INFO: Downloading build from: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Afnmf0LBQl2ysepvUrfDZw/runs/0/artifacts/public%2Fbuild%2Fgeckoview_example.apk

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/bin/mozregression", line 8, in <module>
    sys.exit(main())
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/mozregression/main.py", line 347, in main
    sys.exit(method())
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/mozregression/main.py", line 155, in bisect_integration
    return self._bisect_integration(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/mozregression/main.py", line 169, in _bisect_integration
    result = self._do_bisect(handler, good_rev, bad_rev, expand=expand)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/mozregression/main.py", line 227, in _do_bisect
    return self.bisector.bisect(handler, good, bad, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/mozregression/bisector.py", line 614, in bisect
    return self._bisect(handler, build_range)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/mozregression/bisector.py", line 638, in _bisect
    if bisection.ensure_good_and_bad():
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/mozregression/bisector.py", line 519, in ensure_good_and_bad
    self.download_manager.focus_download(good)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/mozregression/download_manager.py", line 388, in focus_download
    dl.wait()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/mozregression/download_manager.py", line 109, in wait
    self.raise_if_error()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/mozregression/download_manager.py", line 124, in raise_if_error
    raise self.__error[1]
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/mozregression/download_manager.py", line 166, in _download
    total_size = int(response.headers["Content-length"].strip())
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/structures.py", line 52, in __getitem__
    return self._store[key.lower()][1]
KeyError: 'content-length'

$ curl -I  https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Afnmf0LBQl2ysepvUrfDZw/runs/0/artifacts/public%2Fbuild%2Fgeckoview_example.apk
HTTP/2 303 
server: openresty
date: Thu, 13 Jul 2023 18:07:22 GMT
content-type: application/json; charset=utf-8
content-length: 141
content-security-policy: report-uri /__cspreport__;default-src 'none';frame-ancestors 'none';
x-content-type-options: nosniff
x-for-trace-id: 44ae6a0d11f0e87ed670841d0ad1857c
x-for-request-id: 3da60e0c-350c-4d47-bc41-79c46158fe0b
access-control-allow-origin: *
access-control-max-age: 900
access-control-allow-methods: OPTIONS,GET,HEAD,POST,PUT,DELETE,TRACE,CONNECT
access-control-request-method: *
access-control-allow-headers: X-Requested-With,Content-Type,Authorization,Accept,Origin,Cache-Control
cache-control: no-store no-cache must-revalidate
x-taskcluster-artifact-storage-type: s3
location: https://firefoxci.taskcluster-artifacts.net/Afnmf0LBQl2ysepvUrfDZw/0/public/build/geckoview_example.apk
etag: W/"8d-utvS3fNNQDNSXNoLAF0INkg62Yg"
strict-transport-security: max-age=31536000
via: 1.1 google
alt-svc: clear

$ curl -I https://firefoxci.taskcluster-artifacts.net/Afnmf0LBQl2ysepvUrfDZw/0/public/build/geckoview_example.apk
HTTP/2 200 
x-guploader-uploadid: ADPycdtF1PeBiPrvxD0FeVTTKTBmtt2qfAzmGhOwaunr08IWTPCfNkaa78fQDJVWIzaPj9OmQRgfzemuLaZ_BNvE9F5bjPDoX6jP
date: Thu, 13 Jul 2023 18:07:40 GMT
cache-control: public,max-age=604800
last-modified: Wed, 12 Jul 2023 17:27:01 GMT
etag: W/"e4d1e7510760a422b29076d551f21861"
x-goog-generation: 1689182821866447
x-goog-metageneration: 1
x-goog-stored-content-encoding: gzip
x-goog-stored-content-length: 69940196
x-goog-meta-x-goog-source-etag: "e4d1e7510760a422b29076d551f21861"
content-type: application/vnd.android.package-archive
x-goog-hash: crc32c=Aa0URg==
x-goog-hash: md5=5NHnUQdgpCKykHbVUfIYYQ==
x-goog-storage-class: STANDARD
access-control-allow-origin: *
access-control-expose-headers: *
x-guploader-response-body-transformations: gunzipped
warning: 214 UploadServer gunzipped
server: UploadServer
x-cache-status: miss
alt-svc: clear

See also: https://cloud.google.com/storage/docs/transcoding.

Comment 1

[BMO Github Automation]

Attachment: [mozilla/mozregression] download_manager: try alternate content-length header (bug 1843393) (#1380)