High frequency requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://www.googleapis.com/download/storage/v1/b/chromium-browser-snapshots/o/Win%2F972876%2Fchromedriver_win32.zip?alt=media
Categories
(Firefox Build System :: Task Configuration, defect, P5)
Tracking
(firefox109 fixed)
Tracking | Status | |
---|---|---|
firefox109 | --- | fixed |
People
(Reporter: intermittent-bug-filer, Assigned: gbrown)
References
Details
(Keywords: intermittent-failure)
Attachments
(1 file)
Filed by: smolnar [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=368342927&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/RpWCar_QQmW5ut2d1YCLtA/runs/0/artifacts/public/logs/live_backing.log
build-chromium: Downloading chromedriver from https://www.googleapis.com/download/storage/v1/b/chromium-browser-snapshots/o/Win%2F972876%2Fchromedriver_win32.zip?alt=media
[task 2022-02-18T10:36:54.327Z] Traceback (most recent call last):
[task 2022-02-18T10:36:54.327Z] File "/builds/worker/bin/fetch-chromium.py", line 234, in <module>
[task 2022-02-18T10:36:54.327Z] build_chromium_archive(**args)
[task 2022-02-18T10:36:54.328Z] File "/builds/worker/bin/fetch-chromium.py", line 198, in build_chromium_archive
[task 2022-02-18T10:36:54.328Z] fetch_chromedriver(platform, revision, chromium_dir)
[task 2022-02-18T10:36:54.328Z] File "/builds/worker/bin/fetch-chromium.py", line 130, in fetch_chromedriver
[task 2022-02-18T10:36:54.328Z] fetch_file(download_url, tmpzip)
[task 2022-02-18T10:36:54.328Z] File "/builds/worker/bin/fetch-chromium.py", line 74, in fetch_file
[task 2022-02-18T10:36:54.328Z] r.raise_for_status()
[task 2022-02-18T10:36:54.328Z] File "/usr/lib/python3/dist-packages/requests/models.py", line 943, in raise_for_status
[task 2022-02-18T10:36:54.328Z] raise HTTPError(http_error_msg, response=self)
[task 2022-02-18T10:36:54.328Z] requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://www.googleapis.com/download/storage/v1/b/chromium-browser-snapshots/o/Win%2F972876%2Fchromedriver_win32.zip?alt=media
[taskcluster 2022-02-18 10:36:54.592Z] === Task Finished ===
[taskcluster 2022-02-18 10:36:54.670Z] Unsuccessful task run with exit code: 1 completed in 8.801 seconds
Comment hidden (Intermittent Failures Robot) |
Comment 2•2 years ago
|
||
https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.
Reporter | ||
Comment 3•2 years ago
|
||
treeherder |
New failure instance: https://treeherder.mozilla.org/logviewer?job_id=372668648&repo=mozilla-central
Comment hidden (Intermittent Failures Robot) |
Comment 5•2 years ago
|
||
Hi James! Can you please take a look at this? Could the increase in failure frequency be caused by the recent changes from Bug 1751336?
https://treeherder.mozilla.org/jobs?repo=mozilla-central&group_state=expanded&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception%2Crunnable&searchStr=fetch%2Copt%2Cfetch%2Cand%2Cstore%2Ccontent%2Cfetch-win32-chromium%2Cwin32-chromium&revision=87b37ed2950dbb100301f9f83922c3a67b4b5a9d&selectedTaskRun=B57pwAh2Tbu2h0QkiVBSUQ.0
Thank you!
Comment 6•2 years ago
|
||
I don't think so; it appears to be using https://searchfox.org/mozilla-central/source/taskcluster/scripts/misc/fetch-chromium.py which doesn't use any of the logic from web-platform-tests. On the other hand, maybe we should check if our logic still matches what the Googlers implemented for fetching Chromium in wpt. It may also just be that there's an infra change on their end that is affecting our reliability here.
Comment 7•2 years ago
•
|
||
I don't see anything concerning here based on the logs. It looks like it was something on the chromium side that was giving us the incorrect latest revision.
Here's a failing run (revision 990937):
[task 2022-04-11T10:37:43.601Z] build-chromium: Creating upload directory in /builds/worker/artifacts...
[task 2022-04-11T10:37:43.601Z] build-chromium: Getting revision number for latest Win chromium build...
[task 2022-04-11T10:37:43.601Z] build-chromium: Downloading Win chromium build revision 990937...
[task 2022-04-11T10:37:43.601Z] build-chromium: Unpacking archive at: /tmp/tmps_bo9oj5/tmp-chromium.zip to: /tmp/tmps_bo9oj5
[task 2022-04-11T10:37:43.601Z] build-chromium: Downloading chromedriver from https://www.googleapis.com/download/storage/v1/b/chromium-browser-snapshots/o/Win%2F990937%2Fchromedriver_win32.zip?alt=media
[task 2022-04-11T10:37:43.601Z] Traceback (most recent call last):
[task 2022-04-11T10:37:43.601Z] File "/builds/worker/bin/fetch-chromium.py", line 234, in <module>
[task 2022-04-11T10:37:43.601Z] build_chromium_archive(**args)
[task 2022-04-11T10:37:43.601Z] File "/builds/worker/bin/fetch-chromium.py", line 198, in build_chromium_archive
[task 2022-04-11T10:37:43.601Z] fetch_chromedriver(platform, revision, chromium_dir)
[task 2022-04-11T10:37:43.601Z] File "/builds/worker/bin/fetch-chromium.py", line 130, in fetch_chromedriver
[task 2022-04-11T10:37:43.601Z] fetch_file(download_url, tmpzip)
[task 2022-04-11T10:37:43.601Z] File "/builds/worker/bin/fetch-chromium.py", line 74, in fetch_file
[task 2022-04-11T10:37:43.601Z] r.raise_for_status()
[task 2022-04-11T10:37:43.601Z] File "/usr/lib/python3/dist-packages/requests/models.py", line 943, in raise_for_status
[task 2022-04-11T10:37:43.601Z] raise HTTPError(http_error_msg, response=self)
[task 2022-04-11T10:37:43.601Z] requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://www.googleapis.com/download/storage/v1/b/chromium-browser-snapshots/o/Win%2F990937%2Fchromedriver_win32.zip?alt=media
Here's a passing run (revision 990369):
[task 2022-04-11T10:50:06.700Z] build-chromium: Creating upload directory in /builds/worker/artifacts...
[task 2022-04-11T10:50:06.700Z] build-chromium: Getting revision number for latest Win chromium build...
[task 2022-04-11T10:50:06.700Z] build-chromium: Downloading Win chromium build revision 990369...
[task 2022-04-11T10:50:06.700Z] build-chromium: Unpacking archive at: /tmp/tmp6rh8xofg/tmp-chromium.zip to: /tmp/tmp6rh8xofg
[task 2022-04-11T10:50:06.700Z] build-chromium: Downloading chromedriver from https://www.googleapis.com/download/storage/v1/b/chromium-browser-snapshots/o/Win%2F990369%2Fchromedriver_win32.zip?alt=media
[task 2022-04-11T10:50:06.700Z] build-chromium: Unpacking archive at: /tmp/tmpxqtq4au9/cd-tmp.zip to: /tmp/tmp65x4b4vx
[task 2022-04-11T10:50:06.700Z] build-chromium: Copying chromedriver from: /tmp/tmp65x4b4vx/chromedriver_win32/chromedriver.exe to: /tmp/tmp6rh8xofg/chrome-win
[task 2022-04-11T10:50:06.700Z] build-chromium: Added revision to /tmp/tmp6rh8xofg/chrome-win/.REVISION file.
[task 2022-04-11T10:50:06.700Z] build-chromium: Tarring with the command: ['tar', 'cjf', 'chromium-win32.tar.bz2', '-C', '/tmp/tmp6rh8xofg', 'chrome-win']
[task 2022-04-11T10:50:06.700Z] build-chromium: Moving chromium-win32.tar.bz2 to the upload directory...
Based on this (and that it only happened once/twice so far), I won't look into it any further. These came from this push: https://treeherder.mozilla.org/jobs?repo=mozilla-central&selectedTaskRun=B57pwAh2Tbu2h0QkiVBSUQ.2&searchStr=chromium%2Cfetch&revision=87b37ed2950dbb100301f9f83922c3a67b4b5a9d
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment 11•2 years ago
|
||
https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.
Reporter | ||
Comment 12•2 years ago
|
||
treeherder |
New failure instance: https://treeherder.mozilla.org/logviewer?job_id=382135842&repo=mozilla-central
Comment hidden (Intermittent Failures Robot) |
Comment 14•2 years ago
|
||
https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.
Reporter | ||
Comment 15•1 year ago
|
||
treeherder |
New failure instance: https://treeherder.mozilla.org/logviewer?job_id=398537788&repo=mozilla-central
Comment hidden (Intermittent Failures Robot) |
Assignee | ||
Comment 17•1 year ago
|
||
Retry network requests to avoid those intermittent failures resulting from
temporary network or server outages.
Updated•1 year ago
|
Comment 18•1 year ago
|
||
Pushed by gbrown@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/3468828bb8a3 Retry network requests in fetch-chromium.py; r=sparky
Comment 19•1 year ago
|
||
bugherder |
Comment hidden (Intermittent Failures Robot) |
Description
•