Closed Bug 1756129 Opened 2 years ago Closed 1 year ago

High frequency requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://www.googleapis.com/download/storage/v1/b/chromium-browser-snapshots/o/Win%2F972876%2Fchromedriver_win32.zip?alt=media

Categories

(Firefox Build System :: Task Configuration, defect, P5)

defect

Tracking

(firefox109 fixed)

RESOLVED FIXED
109 Branch
Tracking Status
firefox109 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: gbrown)

References

Details

(Keywords: intermittent-failure)

Attachments

(1 file)

Filed by: smolnar [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=368342927&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/RpWCar_QQmW5ut2d1YCLtA/runs/0/artifacts/public/logs/live_backing.log


build-chromium: Downloading chromedriver from https://www.googleapis.com/download/storage/v1/b/chromium-browser-snapshots/o/Win%2F972876%2Fchromedriver_win32.zip?alt=media
[task 2022-02-18T10:36:54.327Z] Traceback (most recent call last):
[task 2022-02-18T10:36:54.327Z]   File "/builds/worker/bin/fetch-chromium.py", line 234, in <module>
[task 2022-02-18T10:36:54.327Z]     build_chromium_archive(**args)
[task 2022-02-18T10:36:54.328Z]   File "/builds/worker/bin/fetch-chromium.py", line 198, in build_chromium_archive
[task 2022-02-18T10:36:54.328Z]     fetch_chromedriver(platform, revision, chromium_dir)
[task 2022-02-18T10:36:54.328Z]   File "/builds/worker/bin/fetch-chromium.py", line 130, in fetch_chromedriver
[task 2022-02-18T10:36:54.328Z]     fetch_file(download_url, tmpzip)
[task 2022-02-18T10:36:54.328Z]   File "/builds/worker/bin/fetch-chromium.py", line 74, in fetch_file
[task 2022-02-18T10:36:54.328Z]     r.raise_for_status()
[task 2022-02-18T10:36:54.328Z]   File "/usr/lib/python3/dist-packages/requests/models.py", line 943, in raise_for_status
[task 2022-02-18T10:36:54.328Z]     raise HTTPError(http_error_msg, response=self)
[task 2022-02-18T10:36:54.328Z] requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://www.googleapis.com/download/storage/v1/b/chromium-browser-snapshots/o/Win%2F972876%2Fchromedriver_win32.zip?alt=media
[taskcluster 2022-02-18 10:36:54.592Z] === Task Finished ===
[taskcluster 2022-02-18 10:36:54.670Z] Unsuccessful task run with exit code: 1 completed in 8.801 seconds
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Flags: needinfo?(james)
Summary: Intermittent requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://www.googleapis.com/download/storage/v1/b/chromium-browser-snapshots/o/Win%2F972876%2Fchromedriver_win32.zip?alt=media → High frequency requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://www.googleapis.com/download/storage/v1/b/chromium-browser-snapshots/o/Win%2F972876%2Fchromedriver_win32.zip?alt=media

I don't think so; it appears to be using https://searchfox.org/mozilla-central/source/taskcluster/scripts/misc/fetch-chromium.py which doesn't use any of the logic from web-platform-tests. On the other hand, maybe we should check if our logic still matches what the Googlers implemented for fetching Chromium in wpt. It may also just be that there's an infra change on their end that is affecting our reliability here.

Flags: needinfo?(james) → needinfo?(gmierz2)

I don't see anything concerning here based on the logs. It looks like it was something on the chromium side that was giving us the incorrect latest revision.

Here's a failing run (revision 990937):

[task 2022-04-11T10:37:43.601Z] build-chromium: Creating upload directory in /builds/worker/artifacts...
[task 2022-04-11T10:37:43.601Z] build-chromium: Getting revision number for latest Win chromium build...
[task 2022-04-11T10:37:43.601Z] build-chromium: Downloading Win chromium build revision 990937...
[task 2022-04-11T10:37:43.601Z] build-chromium: Unpacking archive at: /tmp/tmps_bo9oj5/tmp-chromium.zip to: /tmp/tmps_bo9oj5
[task 2022-04-11T10:37:43.601Z] build-chromium: Downloading chromedriver from https://www.googleapis.com/download/storage/v1/b/chromium-browser-snapshots/o/Win%2F990937%2Fchromedriver_win32.zip?alt=media
[task 2022-04-11T10:37:43.601Z] Traceback (most recent call last):
[task 2022-04-11T10:37:43.601Z]   File "/builds/worker/bin/fetch-chromium.py", line 234, in <module>
[task 2022-04-11T10:37:43.601Z]     build_chromium_archive(**args)
[task 2022-04-11T10:37:43.601Z]   File "/builds/worker/bin/fetch-chromium.py", line 198, in build_chromium_archive
[task 2022-04-11T10:37:43.601Z]     fetch_chromedriver(platform, revision, chromium_dir)
[task 2022-04-11T10:37:43.601Z]   File "/builds/worker/bin/fetch-chromium.py", line 130, in fetch_chromedriver
[task 2022-04-11T10:37:43.601Z]     fetch_file(download_url, tmpzip)
[task 2022-04-11T10:37:43.601Z]   File "/builds/worker/bin/fetch-chromium.py", line 74, in fetch_file
[task 2022-04-11T10:37:43.601Z]     r.raise_for_status()
[task 2022-04-11T10:37:43.601Z]   File "/usr/lib/python3/dist-packages/requests/models.py", line 943, in raise_for_status
[task 2022-04-11T10:37:43.601Z]     raise HTTPError(http_error_msg, response=self)
[task 2022-04-11T10:37:43.601Z] requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://www.googleapis.com/download/storage/v1/b/chromium-browser-snapshots/o/Win%2F990937%2Fchromedriver_win32.zip?alt=media

Here's a passing run (revision 990369):

[task 2022-04-11T10:50:06.700Z] build-chromium: Creating upload directory in /builds/worker/artifacts...
[task 2022-04-11T10:50:06.700Z] build-chromium: Getting revision number for latest Win chromium build...
[task 2022-04-11T10:50:06.700Z] build-chromium: Downloading Win chromium build revision 990369...
[task 2022-04-11T10:50:06.700Z] build-chromium: Unpacking archive at: /tmp/tmp6rh8xofg/tmp-chromium.zip to: /tmp/tmp6rh8xofg
[task 2022-04-11T10:50:06.700Z] build-chromium: Downloading chromedriver from https://www.googleapis.com/download/storage/v1/b/chromium-browser-snapshots/o/Win%2F990369%2Fchromedriver_win32.zip?alt=media
[task 2022-04-11T10:50:06.700Z] build-chromium: Unpacking archive at: /tmp/tmpxqtq4au9/cd-tmp.zip to: /tmp/tmp65x4b4vx
[task 2022-04-11T10:50:06.700Z] build-chromium: Copying chromedriver from: /tmp/tmp65x4b4vx/chromedriver_win32/chromedriver.exe to: /tmp/tmp6rh8xofg/chrome-win
[task 2022-04-11T10:50:06.700Z] build-chromium: Added revision to /tmp/tmp6rh8xofg/chrome-win/.REVISION file.
[task 2022-04-11T10:50:06.700Z] build-chromium: Tarring with the command: ['tar', 'cjf', 'chromium-win32.tar.bz2', '-C', '/tmp/tmp6rh8xofg', 'chrome-win']
[task 2022-04-11T10:50:06.700Z] build-chromium: Moving chromium-win32.tar.bz2 to the upload directory...

Based on this (and that it only happened once/twice so far), I won't look into it any further. These came from this push: https://treeherder.mozilla.org/jobs?repo=mozilla-central&selectedTaskRun=B57pwAh2Tbu2h0QkiVBSUQ.2&searchStr=chromium%2Cfetch&revision=87b37ed2950dbb100301f9f83922c3a67b4b5a9d

Flags: needinfo?(gmierz2)
Status: REOPENED → RESOLVED
Closed: 2 years ago2 years ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Status: REOPENED → RESOLVED
Closed: 2 years ago2 years ago
Resolution: --- → INCOMPLETE
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---

Retry network requests to avoid those intermittent failures resulting from
temporary network or server outages.

Assignee: nobody → gbrown
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/3468828bb8a3
Retry network requests in fetch-chromium.py; r=sparky
Status: REOPENED → RESOLVED
Closed: 2 years ago1 year ago
Resolution: --- → FIXED
Target Milestone: --- → 109 Branch
See Also: → 1811729
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: