1879549 - uploads to addons.allizom.org fail for staging pushes on Try: aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('https://addons.allizom.org/api/v5/addons/upload/')

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Uploads to addons.allizom.org fail for staging pushes on Try. Example:

2024-02-09 12:47:14,974 - addonscript.utils - DEBUG - Calling amo_post() with URL "https://addons.allizom.org/api/v5/addons/upload/"
2024-02-09 12:47:15,979 - addonscript.utils - DEBUG - amo_post() for URL "https://addons.allizom.org/api/v5/addons/upload/" returned HTTP status code: 403
2024-02-09 12:47:15,979 - scriptworker.utils - WARNING - retry_async: do_upload: too many retries!
Traceback (most recent call last):
  File "/app/bin/addonscript", line 33, in <module>
    sys.exit(load_entry_point('addonscript==1.0', 'console_scripts', 'addonscript')())
  File "/app/lib/python3.9/site-packages/addonscript/script.py", line 96, in main
    return scriptworker.client.sync_main(async_main, config_path=config_path, default_config=get_default_config())
  File "/app/lib/python3.9/site-packages/scriptworker/client.py", line 166, in sync_main
    loop.run_until_complete(_handle_asyncio_loop(async_main, context))
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/app/lib/python3.9/site-packages/scriptworker/client.py", line 205, in _handle_asyncio_loop
    await async_main(context)
  File "/app/lib/python3.9/site-packages/addonscript/script.py", line 92, in async_main
    await asyncio.gather(*tasks)
  File "/app/lib/python3.9/site-packages/addonscript/script.py", line 36, in sign_addon
    upload = await retry_async(do_upload, args=(context, locale), retry_exceptions=tuple([ClientError, asyncio.TimeoutError]))
  File "/app/lib/python3.9/site-packages/scriptworker/utils.py", line 273, in retry_async
    _check_number_of_attempts(attempt, attempts, func, "retry_async")
  File "/app/lib/python3.9/site-packages/scriptworker/utils.py", line 268, in retry_async
    return await func(*args, **kwargs)
  File "/app/lib/python3.9/site-packages/addonscript/api.py", line 80, in do_upload
    return await amo_post(context, url, data)
  File "/app/lib/python3.9/site-packages/addonscript/utils.py", line 96, in amo_post
    r.raise_for_status()
  File "/app/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 1060, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('https://addons.allizom.org/api/v5/addons/upload/')

The failed had been first observed for a push on Jan 18 from hneiva and succeeded on Jan 12 for jcristau

Comment 1

[Julien Cristau [:jcristau]]

As an aside, we probably shouldn't be retrying on 403...

Mat, any idea what might be going on here?

Comment 2

[Julien Cristau [:jcristau]]

Mat found that the developer agreement on AMO stage was updated on Jan 17. Logging in and accepting the new agreement at https://addons.allizom.org/en-US/developers/addon/submit/agreement seems to have unblocked things.

Another thing we should do here in addition to avoiding useless retries is print the http response body on errors to make debugging easier.

Comment 3

[Julien Cristau [:jcristau]]

Followups:

• Consider bypassing developer agreement check for langpacks upload (https://github.com/mozilla/addons-server/issues/21841)
• addonscript: don't retry on 4xx errors (https://github.com/mozilla-releng/scriptworker-scripts/issues/912)
• addonscript: log http error response body (https://github.com/mozilla-releng/scriptworker-scripts/issues/913)