Closed Bug 1854191 Opened 2 years ago Closed 2 years ago

Intermittent googleapiclient.errors.HttpError

Categories

(Release Engineering :: General, defect, P5)

Product:
Release Engineering
Component:
General
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: intermittent-bug-filer, Unassigned)

Details

(Keywords: intermittent-failure)

Component: Release Automation: Bouncer → General
QA Contact: jcristau → jlorenzo

These failures have been frequent today, but tasks eventually succeeded on re-run.

I note that several apk uploads succeeded before one failed:

2023-09-20 16:30:40,424 - googleapiclient.discovery - DEBUG - URL being requested: POST https://androidpublisher.googleapis.com/upload/androidpublisher/v3/applications/org.mozilla.fenix/edits/16773465932317892413/apks?alt=json&uploadType=media
2023-09-20 16:30:52,973 - mozapkpublisher.common.store - INFO - "/app/workdir/cot/d160saotT9CqxQDPBLscsg/public/build/target.armeabi-v7a.apk" uploaded
2023-09-20 16:30:52,974 - mozapkpublisher.common.store - DEBUG - Upload response: {'versionCode': 2015975297, 'binary': {'sha1': '4f152be3ad0d0b429df7516395307690ccb2e7e0', 'sha256': '5e81d9931c8851e870eea884db69c2192bdf3a9b6f859a02858dbbfdcf1d6a5c'}}
2023-09-20 16:30:52,974 - mozapkpublisher.common.store - INFO - Uploading "/app/workdir/cot/d160saotT9CqxQDPBLscsg/public/build/target.x86.apk" ...
2023-09-20 16:30:52,975 - googleapiclient.discovery - WARNING - media_mime_type argument not specified: trying to auto-detect for /app/workdir/cot/d160saotT9CqxQDPBLscsg/public/build/target.x86.apk
2023-09-20 16:30:53,264 - googleapiclient.discovery - DEBUG - URL being requested: POST https://androidpublisher.googleapis.com/upload/androidpublisher/v3/applications/org.mozilla.fenix/edits/16773465932317892413/apks?alt=json&uploadType=media
Traceback (most recent call last):
  File "/app/bin/pushapkscript", line 33, in <module>
    sys.exit(load_entry_point('pushapkscript==5.0.0', 'console_scripts', 'pushapkscript')())
  File "/app/lib/python3.9/site-packages/pushapkscript/script.py", line 105, in main
    client.sync_main(async_main, config_path=config_path, default_config=get_default_config())
  File "/app/lib/python3.9/site-packages/scriptworker/client.py", line 166, in sync_main
    loop.run_until_complete(_handle_asyncio_loop(async_main, context))
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/app/lib/python3.9/site-packages/scriptworker/client.py", line 205, in _handle_asyncio_loop
    await async_main(context)
  File "/app/lib/python3.9/site-packages/pushapkscript/script.py", line 53, in async_main
    publish.publish(product_config, publish_config, files, contact_server)
  File "/app/lib/python3.9/site-packages/pushapkscript/publish.py", line 11, in publish
    push_apk(
  File "/app/lib/python3.9/site-packages/mozapkpublisher/push_apk.py", line 105, in push_apk
    edit.update_app(extracted_apks, **update_app_kwargs)
  File "/app/lib/python3.9/site-packages/mozapkpublisher/common/store.py", line 183, in update_app
    self.upload_apk(apk)
  File "/app/lib/python3.9/site-packages/mozapkpublisher/common/store.py", line 208, in upload_apk
    response = self._edit_resource.apks().upload(
  File "/app/lib/python3.9/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/app/lib/python3.9/site-packages/googleapiclient/http.py", line 938, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 401 when requesting https://androidpublisher.googleapis.com/upload/androidpublisher/v3/applications/org.mozilla.fenix/edits/16773465932317892413/apks?alt=json&uploadType=media returned "Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.". Details: "Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.">
exit code: 1

https://github.com/mozilla-releng/scriptworker-scripts/pull/843 included dependency updates, but nothing major. That has been on production since Sept 18 and nightly pushes have been okay until today.

(In reply to Geoff Brown [:gbrown] from comment #2)

...since Sept 18 and nightly pushes have been okay until today.

Actually, jcristau found the same failure in push tasks even before Sept 18:
https://firefox-ci-tc.services.mozilla.com/tasks/PKhetMvHScWxufKXXqdhYQ/runs/0
https://firefox-ci-tc.services.mozilla.com/tasks/OgZH01R6Rh2-YFod5tZ4Iw/runs/0

https://developers.google.com/identity/sign-in/web/sign-in has a deprecation warning banner at the top and links to https://developers.google.com/identity/sign-in/web/deprecation-and-sunset, which mentions a deprecation date of March 31, 2023 and a to-be-determined sunset date. Are we seeing deprecation brown-outs?

On the other hand, https://developers.google.com/identity/sign-in/web/deprecation-and-sunset says

The deprecation only affects the Google Sign-In JavaScript library for Web. If you are using ..., or OAuth 2.0 for authentication or authorization with Google services, this deprecation does not affect your app(s).

I'm not very familiar with our authentication, but I see:
https://github.com/mozilla-releng/mozapkpublisher/blob/master/mozapkpublisher/common/store.py#L8

from oauth2client.service_account import ServiceAccountCredentials

It seems like these errors have mostly gone away, with no changes on our side.

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.