Closed Bug 1837547 Opened 2 years ago Closed 2 years ago

Perma addonscript.exceptions.FatalSignatureError: XPI disabled on AMO

Categories

(Release Engineering :: Release Automation, defect, P1)

Product:
Release Engineering
Component:
Release Automation
Type:
defect

Tracking

(firefox114blocking fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox114 blocking fixed

People

(Reporter: intermittent-bug-filer, Unassigned)

References

Details

(Keywords: intermittent-failure)

Attachments

(1 file)

[mozilla-releng/relengdocs] langpacks: document bug 1837547 (#216)
53 bytes, text/x-github-pull-request
Details | Review

Filed by: chorotan [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=418646794&repo=mozilla-release
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/dZbQ55ukRBqcmdNgNqkqMA/runs/0/artifacts/public/logs/live_backing.log

2023-06-08 23:14:55,939 - scriptworker.client - DEBUG - Task is validated against this schema: {'title': 'Taskcluster addon submission task minimal schema', 'type': 'object', 'properties': {'dependencies': {'type': 'array', 'minItems': 1, 'uniqueItems': True, 'items': {'type': 'string'}}, 'scopes': {'type': 'array', 'minItems': 1, 'uniqueItems': True, 'items': {'type': 'string'}}, 'payload': {'type': 'object', 'properties': {'upstreamArtifacts': {'type': 'array', 'items': {'type': 'object', 'properties': {'taskType': {'type': 'string'}, 'taskId': {'type': 'string'}, 'paths': {'type': 'array', 'minItems': 1, 'uniqueItems': True, 'items': {'type': 'string'}}}, 'required': ['taskId', 'taskType', 'paths']}, 'minItems': 1, 'uniqueItems': True}, 'channel': {'type': 'string', 'enum': ['listed', 'unlisted']}}, 'required': ['upstreamArtifacts', 'channel']}}, 'required': ['scopes', 'payload']}
2023-06-08 23:14:55,955 - asyncio - DEBUG - Using selector: EpollSelector
2023-06-08 23:14:55,964 - addonscript.utils - DEBUG - Calling amo_put() with URL "https://addons.mozilla.org/api/v4/applications/firefox/114.0/"
2023-06-08 23:14:56,132 - addonscript.utils - DEBUG - amo_put() for URL "https://addons.mozilla.org/api/v4/applications/firefox/114.0/" returned HTTP status code: 202
2023-06-08 23:14:56,132 - addonscript.utils - DEBUG - amo_put() for URL "https://addons.mozilla.org/api/v4/applications/firefox/114.0/" returned: None
2023-06-08 23:14:56,133 - addonscript.utils - DEBUG - Calling amo_post() with URL "https://addons.mozilla.org/api/v5/addons/upload/"
2023-06-08 23:14:56,737 - addonscript.utils - DEBUG - amo_post() for URL "https://addons.mozilla.org/api/v5/addons/upload/" returned HTTP status code: 201
2023-06-08 23:14:56,742 - addonscript.utils - DEBUG - amo_post() for URL "https://addons.mozilla.org/api/v5/addons/upload/" returned: {'uuid': '45a45b2817b849ccac10a444c0ed19d7', 'channel': 'listed', 'processed': False, 'submitted': False, 'url': 'https://addons.mozilla.org/api/v5/addons/upload/45a45b2817b849ccac10a444c0ed19d7/', 'valid': False, 'validation': None, 'version': '114.0.20230608.214645'}
2023-06-08 23:14:56,743 - addonscript.utils - DEBUG - Calling amo_get() with URL "https://addons.mozilla.org/api/v5/addons/upload/45a45b2817b849ccac10a444c0ed19d7/"
2023-06-08 23:14:56,826 - addonscript.utils - DEBUG - amo_get() for URL "https://addons.mozilla.org/api/v5/addons/upload/45a45b2817b849ccac10a444c0ed19d7/" returned HTTP status code: 200
2023-06-08 23:14:56,828 - addonscript.utils - DEBUG - amo_get() for URL "https://addons.mozilla.org/api/v5/addons/upload/45a45b2817b849ccac10a444c0ed19d7/" returned: {'uuid': '45a45b2817b849ccac10a444c0ed19d7', 'channel': 'listed', 'processed': False, 'submitted': False, 'url': 'https://addons.mozilla.org/api/v5/addons/upload/45a45b2817b849ccac10a444c0ed19d7/', 'valid': False, 'validation': None, 'version': '114.0.20230608.214645'}
2023-06-08 23:14:56,829 - scriptworker.utils - DEBUG - retry_async: check_upload: sleeping 13.667788765987874 seconds before retry
2023-06-08 23:15:10,512 - addonscript.utils - DEBUG - Calling amo_get() with URL "https://addons.mozilla.org/api/v5/addons/upload/45a45b2817b849ccac10a444c0ed19d7/"
2023-06-08 23:15:10,618 - addonscript.utils - DEBUG - amo_get() for URL "https://addons.mozilla.org/api/v5/addons/upload/45a45b2817b849ccac10a444c0ed19d7/" returned HTTP status code: 200
2023-06-08 23:15:10,618 - addonscript.utils - DEBUG - amo_get() for URL "https://addons.mozilla.org/api/v5/addons/upload/45a45b2817b849ccac10a444c0ed19d7/" returned: {'uuid': '45a45b2817b849ccac10a444c0ed19d7', 'channel': 'listed', 'processed': True, 'submitted': False, 'url': 'https://addons.mozilla.org/api/v5/addons/upload/45a45b2817b849ccac10a444c0ed19d7/', 'valid': True, 'validation': {'errors': 0, 'warnings': 0, 'notices': 0, 'success': True, 'compatibility_summary': {'warnings': 0, 'errors': 0, 'notices': 0}, 'metadata': {'listed': True, 'identified_files': {}, 'id': 'langpack-en-US@firefox.mozilla.org', 'manifestVersion': 2, 'name': 'Language: English (US)', 'type': 1, 'version': '114.0.20230608.214645', 'firefoxMinVersion': '114.0', 'experimentApiPaths': {}, 'totalScannedFileSize': 1382817, 'emptyFiles': [], 'jsLibs': {}, 'unknownMinifiedFiles': []}, 'messages': [], 'message_tree': {}, 'ending_tier': 5}, 'version': '114.0.20230608.214645'}
2023-06-08 23:15:10,619 - addonscript.utils - DEBUG - Calling amo_put() with URL "https://addons.mozilla.org/api/v5/addons/addon/langpack-en-US@firefox.mozilla.org/"
2023-06-08 23:15:11,373 - addonscript.utils - DEBUG - amo_put() for URL "https://addons.mozilla.org/api/v5/addons/addon/langpack-en-US@firefox.mozilla.org/" returned HTTP status code: 200
2023-06-08 23:15:11,378 - addonscript.utils - DEBUG - amo_put() for URL "https://addons.mozilla.org/api/v5/addons/addon/langpack-en-US@firefox.mozilla.org/" returned: {'id': 407142, 'authors': [{'id': 4757633, 'name': 'Mozilla Firefox', 'url': 'https://addons.mozilla.org/en-US/firefox/user/4757633/', 'username': 'mozilla', 'picture_url': 'https://addons.mozilla.org/user-media/userpics/33/7633/4757633/4757633.png?modified=1583526273'}], 'average_daily_users': 1966673, 'categories': {'firefox': ['general']}, 'contributions_url': '', 'created': '2012-11-11T16:00:54Z', 'current_version': {'id': 5577089, 'approval_notes': '', 'compatibility': {'firefox': {'min': '115.0', 'max': '115.*'}}, 'edit_url': 'https://addons.mozilla.org/en-US/developers/addon/english-us-language-pack/versions/5577089', 'file': {'id': 4121431, 'created': '2023-06-06T21:14:11Z', 'hash': 'sha256:dfd7c429afa09e4037a0b94dc4fbbdee63d62512e9976cf462d89db1ca8bda81', 'is_mozilla_signed_extension': False, 'size': 523326, 'status': 'public', 'url': 'https://addons.mozilla.org/firefox/downloads/file/4121431/english_us_language_pack-115.0.20230606.180112.xpi', 'permissions': [], 'optional_permissions': [], 'host_permissions': []}, 'is_disabled': False, 'is_strict_compatibility_enabled': True, 'license': {'id': 3338, 'is_custom': False, 'name': {'en-US': 'Mozilla Public License 2.0'}, 'slug': 'MPL-2.0', 'url': 'http://www.mozilla.org/MPL/2.0/'}, 'release_notes': None, 'reviewed': '2023-06-06T21:18:51Z', 'version': '115.0.20230606.180112', 'source': None}, 'default_locale': 'en-US', 'description': None, 'developer_comments': None, 'edit_url': 'https://addons.mozilla.org/en-US/developers/addon/english-us-language-pack/edit', 'guid': 'langpack-en-US@firefox.mozilla.org', 'has_eula': False, 'has_privacy_policy': False, 'homepage': None, 'icon_url': 'https://addons.mozilla.org/static-server/img/addon-icons/default-64.d144b50f2bb8.png', 'icons': {'32': 'https://addons.mozilla.org/static-server/img/addon-icons/default-32.2b26add4b394.png', '64': 'https://addons.mozilla.org/static-server/img/addon-icons/default-64.d144b50f2bb8.png', '128': 'https://addons.mozilla.org/static-server/img/addon-icons/default-128.452018499894.png'}, 'is_disabled': False, 'is_experimental': False, 'last_updated': '2023-06-06T21:18:51Z', 'name': {'en-US': 'Language: English (US)'}, 'previews': [], 'promoted': None, 'ratings': {'average': 4.1672, 'bayesian_average': 4.159164715906709, 'count': 652, 'text_count': 123}, 'ratings_url': 'https://addons.mozilla.org/en-US/firefox/addon/english-us-language-pack/reviews/', 'requires_payment': False, 'review_url': 'https://addons.mozilla.org/en-US/reviewers/review/407142', 'slug': 'english-us-language-pack', 'status': 'public', 'summary': {'en-US': 'Firefox Language Pack for English (US) (en-US)'}, 'support_email': None, 'support_url': None, 'tags': [], 'type': 'language', 'url': 'https://addons.mozilla.org/en-US/firefox/addon/english-us-language-pack/', 'version': {'id': 5578125, 'approval_notes': '', 'channel': 'listed', 'compatibility': {'firefox': {'min': '114.0', 'max': '114.*'}}, 'edit_url': 'https://addons.mozilla.org/en-US/developers/addon/english-us-language-pack/versions/5578125', 'file': {'id': 4122467, 'created': '2023-06-08T23:15:10Z', 'hash': 'sha256:d2d393213fb8022974c2c7728c31abff11d0e479b9d07a0030227eeff0ae1361', 'is_mozilla_signed_extension': False, 'size': 481016, 'status': 'unreviewed', 'url': 'https://addons.mozilla.org/firefox/downloads/file/4122467/english_us_language_pack-114.0.20230608.214645.zip', 'permissions': [], 'optional_permissions': [], 'host_permissions': []}, 'is_disabled': False, 'is_strict_compatibility_enabled': True, 'license': {'id': 3338, 'is_custom': False, 'name': {'en-US': 'Mozilla Public License 2.0'}, 'slug': 'MPL-2.0', 'text': {'en-US': 'This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at <a rel="nofollow" href="https://prod.outgoing.prod.webservices.mozgcp.net/v1/cfa1520767b4bc45590e23d48de7bec652c237ee0ad206608dfa16f2a0e21c8d/http%3A//mozilla.org/MPL/2.0/">http://mozilla.org/MPL/2.0/</a>.'}, 'url': 'http://www.mozilla.org/MPL/2.0/'}, 'release_notes': None, 'reviewed': None, 'source': None, 'version': '114.0.20230608.214645'}, 'versions_url': 'https://addons.mozilla.org/en-US/firefox/addon/english-us-language-pack/versions/', 'weekly_downloads': 1675, 'latest_unlisted_version': {'id': 5573360, 'approval_notes': '', 'compatibility': {'firefox': {'min': '102.0', 'max': '102.*'}}, 'edit_url': 'https://addons.mozilla.org/en-US/developers/addon/english-us-language-pack/versions/5573360', 'file': {'id': 4117702, 'created': '2023-05-29T18:40:58Z', 'hash': 'sha256:5cb7aed9bb759b4fa06d1177a9cb65c1c438b31e7cb5027924e1e0c628e6dc09', 'is_mozilla_signed_extension': False, 'size': 482817, 'status': 'public', 'url': 'https://addons.mozilla.org/firefox/downloads/file/4117702/english_us_language_pack-102.12.0buildid20230529.170948.xpi', 'permissions': [], 'optional_permissions': [], 'host_permissions': []}, 'is_disabled': False, 'is_strict_compatibility_enabled': True, 'license': {'id': 3338, 'is_custom': False, 'name': {'en-US': 'Mozilla Public License 2.0'}, 'slug': 'MPL-2.0', 'url': 'http://www.mozilla.org/MPL/2.0/'}, 'release_notes': None, 'reviewed': '2023-05-29T18:45:43Z', 'version': '102.12.0buildid20230529.170948', 'source': None}}
2023-06-08 23:15:11,378 - addonscript.utils - DEBUG - Calling amo_get() with URL "https://addons.mozilla.org/api/v5/addons/addon/langpack-en-US@firefox.mozilla.org/versions/5578125/"
2023-06-08 23:15:11,480 - addonscript.utils - DEBUG - amo_get() for URL "https://addons.mozilla.org/api/v5/addons/addon/langpack-en-US@firefox.mozilla.org/versions/5578125/" returned HTTP status code: 200
2023-06-08 23:15:11,480 - addonscript.utils - DEBUG - amo_get() for URL "https://addons.mozilla.org/api/v5/addons/addon/langpack-en-US@firefox.mozilla.org/versions/5578125/" returned: {'id': 5578125, 'approval_notes': '', 'channel': 'listed', 'compatibility': {'firefox': {'min': '114.0', 'max': '114.*'}}, 'edit_url': 'https://addons.mozilla.org/en-US/developers/addon/english-us-language-pack/versions/5578125', 'file': {'id': 4122467, 'created': '2023-06-08T23:15:10Z', 'hash': 'sha256:d2d393213fb8022974c2c7728c31abff11d0e479b9d07a0030227eeff0ae1361', 'is_mozilla_signed_extension': False, 'size': 481016, 'status': 'unreviewed', 'url': 'https://addons.mozilla.org/firefox/downloads/file/4122467/english_us_language_pack-114.0.20230608.214645.zip', 'permissions': [], 'optional_permissions': [], 'host_permissions': []}, 'is_disabled': False, 'is_strict_compatibility_enabled': True, 'license': {'id': 3338, 'is_custom': False, 'name': {'en-US': 'Mozilla Public License 2.0'}, 'slug': 'MPL-2.0', 'text': {'en-US': 'This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at <a rel="nofollow" href="https://prod.outgoing.prod.webservices.mozgcp.net/v1/cfa1520767b4bc45590e23d48de7bec652c237ee0ad206608dfa16f2a0e21c8d/http%3A//mozilla.org/MPL/2.0/">http://mozilla.org/MPL/2.0/</a>.'}, 'url': 'http://www.mozilla.org/MPL/2.0/'}, 'release_notes': None, 'reviewed': None, 'source': None, 'version': '114.0.20230608.214645'}
2023-06-08 23:15:11,480 - scriptworker.utils - DEBUG - retry_async: get_signed_addon_info: sleeping 14.358905832346366 seconds before retry
2023-06-08 23:15:25,854 - addonscript.utils - DEBUG - Calling amo_get() with URL "https://addons.mozilla.org/api/v5/addons/addon/langpack-en-US@firefox.mozilla.org/versions/5578125/"
2023-06-08 23:15:25,978 - addonscript.utils - DEBUG - amo_get() for URL "https://addons.mozilla.org/api/v5/addons/addon/langpack-en-US@firefox.mozilla.org/versions/5578125/" returned HTTP status code: 200
2023-06-08 23:15:25,979 - addonscript.utils - DEBUG - amo_get() for URL "https://addons.mozilla.org/api/v5/addons/addon/langpack-en-US@firefox.mozilla.org/versions/5578125/" returned: {'id': 5578125, 'approval_notes': '', 'channel': 'listed', 'compatibility': {'firefox': {'min': '114.0', 'max': '114.*'}}, 'edit_url': 'https://addons.mozilla.org/en-US/developers/addon/english-us-language-pack/versions/5578125', 'file': {'id': 4122467, 'created': '2023-06-08T23:15:10Z', 'hash': 'sha256:d2d393213fb8022974c2c7728c31abff11d0e479b9d07a0030227eeff0ae1361', 'is_mozilla_signed_extension': False, 'size': 481016, 'status': 'disabled', 'url': 'https://addons.mozilla.org/firefox/downloads/file/4122467/english_us_language_pack-114.0.20230608.214645.zip', 'permissions': [], 'optional_permissions': [], 'host_permissions': []}, 'is_disabled': False, 'is_strict_compatibility_enabled': True, 'license': {'id': 3338, 'is_custom': False, 'name': {'en-US': 'Mozilla Public License 2.0'}, 'slug': 'MPL-2.0', 'text': {'en-US': 'This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at <a rel="nofollow" href="https://prod.outgoing.prod.webservices.mozgcp.net/v1/cfa1520767b4bc45590e23d48de7bec652c237ee0ad206608dfa16f2a0e21c8d/http%3A//mozilla.org/MPL/2.0/">http://mozilla.org/MPL/2.0/</a>.'}, 'url': 'http://www.mozilla.org/MPL/2.0/'}, 'release_notes': None, 'reviewed': None, 'source': None, 'version': '114.0.20230608.214645'}
2023-06-08 23:15:25,980 - scriptworker.client - ERROR - Failed to run async_main
Traceback (most recent call last):
  File "/app/lib/python3.9/site-packages/scriptworker/client.py", line 205, in _handle_asyncio_loop
    await async_main(context)
  File "/app/lib/python3.9/site-packages/addonscript/script.py", line 92, in async_main
    await asyncio.gather(*tasks)
  File "/app/lib/python3.9/site-packages/addonscript/script.py", line 52, in sign_addon
    signed_addon_info = await retry_async(
  File "/app/lib/python3.9/site-packages/scriptworker/utils.py", line 268, in retry_async
    return await func(*args, **kwargs)
  File "/app/lib/python3.9/site-packages/addonscript/api.py", line 137, in get_signed_addon_info
    raise FatalSignatureError("XPI disabled on AMO")
addonscript.exceptions.FatalSignatureError: XPI disabled on AMO
exit code: 1
Summary: Intermittent addonscript.exceptions.FatalSignatureError: XPI disabled on AMO → Perma addonscript.exceptions.FatalSignatureError: XPI disabled on AMO

The error message is coming from: https://github.com/mozilla-releng/scriptworker-scripts/blob/68e9b4994575343bace2f97715841509b0319cdb/addonscript/src/addonscript/api.py#L136-L137, but that doesn't explain why AMO sets the status to disabled in the first place.

Flags: needinfo?(mpillard)

Setting as blocker for today's release.

status-firefox114: --- → affected
tracking-firefox114: --- → blocking
Priority: P5 → P1

Moving NI request to Andrew

Flags: needinfo?(awilliamson)

AMO only allows a single version in each channel to be awaiting approval at one time - if a new version is uploaded before the previous one is approved&signed the previous version is skipped and disabled.

From the reviewer tools it looks like 114.0.20230608.214645 and 115.0.20230608.214116 were uploaded almost at the same time.

Flags: needinfo?(awilliamson)

In order to fix the langpack version stuck, we could hard delete it (with the django shell command below) and re-trigger the failing task.

Version.objects.get(pk=5578125).delete(hard=True)
Flags: needinfo?(wezhou)

(In reply to Andrew Williamson [:eviljeff] from comment #4)

AMO only allows a single version in each channel to be awaiting approval at one time - if a new version is uploaded before the previous one is approved&signed the previous version is skipped and disabled.

That sounds like a problem for us (having beta/release/esr builds in parallel is not uncommon); we should talk :)

(In reply to Julien Cristau [:jcristau] from comment #6)

(In reply to Andrew Williamson [:eviljeff] from comment #4)

AMO only allows a single version in each channel to be awaiting approval at one time - if a new version is uploaded before the previous one is approved&signed the previous version is skipped and disabled.

That sounds like a problem for us (having beta/release/esr builds in parallel is not uncommon); we should talk :)

Sure. I should mention it's a very old restriction - like from when the site was written 10+ years ago - that you've apparently not hit until now.

(In reply to William Durand [:willdurand] from comment #5)

In order to fix the langpack version stuck, we could hard delete it (with the django shell command below) and re-trigger the failing task.

Version.objects.get(pk=5578125).delete(hard=True)

Confirming that should work (and is the correct pk on AMO prod)

Fwiw, this will cascade delete every object that is foreign key connected to the Version/File in the database, and bypass some of the clean-up that happens when a version is normally (soft) deleted so it's really an emergency-last-resort action - and not without the risk of unintended consequences.

Flags: needinfo?(mpillard)

The version has been deleted.

Flags: needinfo?(wezhou)
See Also: → 1848872
Component: Release Automation: Signing → Release Automation
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: