Closed Bug 1811729 Opened 1 year ago Closed 1 year ago

Intermittent requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://www.googleapis.com/download/storage/v1/b/chromium-browser-snapshots/o/Win%2F1095440%2Fchromedriver_win32.zip?alt=media

Categories

(Firefox Build System :: Task Configuration, defect, P5)

Product:
Firefox Build System
Component:
Task Configuration
Type:
defect

Tracking

(firefox111 fixed)

Status:
RESOLVED FIXED
Milestone:
111 Branch
Tracking Flags:
Tracking Status
firefox111 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: gbrown)

References

Details

(Keywords: intermittent-failure)

Attachments

(1 file)

Bug 1811729 - Retry failed network connections for a few more minutes
48 bytes, text/x-phabricator-request
Details | Review

Filed by: mlaza [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=403137418&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/EQy8wzAtQSaNfSlFXRtgGw/runs/0/artifacts/public/logs/live_backing.log

[task 2023-01-22T10:36:11.847Z] build-chromium: Downloading chromedriver from https://www.googleapis.com/download/storage/v1/b/chromium-browser-snapshots/o/Win%2F1095440%2Fchromedriver_win32.zip?alt=media
[task 2023-01-22T10:36:11.847Z] Traceback (most recent call last):
[task 2023-01-22T10:36:11.847Z]   File "/builds/worker/bin/fetch-chromium.py", line 235, in <module>
[task 2023-01-22T10:36:11.847Z]     build_chromium_archive(**args)
[task 2023-01-22T10:36:11.847Z]   File "/builds/worker/bin/fetch-chromium.py", line 199, in build_chromium_archive
[task 2023-01-22T10:36:11.847Z]     fetch_chromedriver(platform, revision, chromium_dir)
[task 2023-01-22T10:36:11.847Z]   File "/builds/worker/bin/fetch-chromium.py", line 131, in fetch_chromedriver
[task 2023-01-22T10:36:11.847Z]     fetch_file(download_url, tmpzip)
[task 2023-01-22T10:36:11.847Z]   File "/usr/local/lib/python3.9/dist-packages/redo/__init__.py", line 215, in _retriable_wrapper
[task 2023-01-22T10:36:11.847Z]     return retry(func, args=args, kwargs=kwargs, *retry_args, **retry_kwargs)
[task 2023-01-22T10:36:11.847Z]   File "/usr/local/lib/python3.9/dist-packages/redo/__init__.py", line 170, in retry
[task 2023-01-22T10:36:11.847Z]     return action(*args, **kwargs)
[task 2023-01-22T10:36:11.847Z]   File "/builds/worker/bin/fetch-chromium.py", line 74, in fetch_file
[task 2023-01-22T10:36:11.847Z]     r.raise_for_status()
[task 2023-01-22T10:36:11.847Z]   File "/usr/lib/python3/dist-packages/requests/models.py", line 943, in raise_for_status
[task 2023-01-22T10:36:11.847Z]     raise HTTPError(http_error_msg, response=self)
[task 2023-01-22T10:36:11.847Z] requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://www.googleapis.com/download/storage/v1/b/chromium-browser-snapshots/o/Win%2F1095440%2Fchromedriver_win32.zip?alt=media
[taskcluster 2023-01-22 10:36:12.113Z] === Task Finished ===
[taskcluster 2023-01-22 10:36:12.159Z] Unsuccessful task run with exit code: 1 completed in 90.959 seconds

Hi Geoff! Can you please take a look at this? I've needinfo? you because I saw that you landed a patch in Bug 1756129 that fixed the issue.
Thank you!

Flags: needinfo?(gbrown)

It looks like fetch_chromium_revision can return a revision for which all the files we want aren't available?

Flags: needinfo?(gbrown)

(In reply to Julien Cristau [:jcristau] from comment #3)

It looks like fetch_chromium_revision can return a revision for which all the files we want aren't available?

...are temporarily not available -- this succeeded on task retry, and that url is okay now.

I think the retries added in bug 1756129 are working as expected, but maybe we could do better with more retries?

Currently it is 5 retries:

  • fetch
  • wait 5 seconds
  • fetch
  • wait 10 seconds
  • fetch
  • wait 20 seconds
  • fetch
  • wait 40 seconds
  • fetch

...waiting up to 75 seconds if necessary.

I see the task retries succeeded about 4 minutes after the failure.

Assignee: nobody → gbrown

Bump the number of retries from 5 (5+10+20+40=75 seconds wait worst case)
to 7 (5+10+20+40+80+160=315 seconds worst case).

Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/5ab951cd520e
Retry failed network connections for a few more minutes r=jcristau

https://hg.mozilla.org/mozilla-central/rev/5ab951cd520e

Status: NEW → RESOLVED
Closed: 1 year ago
status-firefox111: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 111 Branch
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: