1848872 - Perma addonscript.exceptions.FatalSignatureError: XPI disabled on AMO

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Signing, defect, P1)

Description

[Treeherder Bug Filer]

Filed by: nfay [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=426114853&repo=mozilla-release
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Ptb-Mc0cTumzK__lSgAQEQ/runs/0/artifacts/public/logs/live_backing.log

---

2023-08-15 21:23:37,082 - scriptworker.utils - DEBUG - retry_async: get_signed_addon_info: sleeping 120 seconds before retry
2023-08-15 21:24:58,674 - addonscript.utils - DEBUG - Calling amo_get() with URL "https://addons.mozilla.org/api/v5/addons/addon/[e-mail]/versions/5609404/"
2023-08-15 21:24:58,795 - addonscript.utils - DEBUG - amo_get() for URL "https://addons.mozilla.org/api/v5/addons/addon/[e-mail]/versions/5609404/" returned HTTP status code: 200
2023-08-15 21:24:58,795 - addonscript.utils - DEBUG - amo_get() for URL "https://addons.mozilla.org/api/v5/addons/addon/[e-mail]/versions/5609404/" returned: {'id': 5609404, 'approval_notes': '', 'channel': 'listed', 'compatibility': {'firefox': {'min': '116.0', 'max': '116.*'}}, 'edit_url': 'https://addons.mozilla.org/en-US/developers/addon/traditional-chinese-zh-tw-l/versions/5609404', 'file': {'id': 4153745, 'created': '2023-08-15T21:20:15Z', 'hash': 'sha256:2d47b58055a8068abf25b8a6b069aa1df99b803309ee3ecde53a1e9a62440075', 'is_mozilla_signed_extension': False, 'size': 551499, 'status': 'disabled', 'url': 'https://addons.mozilla.org/firefox/downloads/file/4153745/traditional_chinese_zh_tw_l-116.0.20230815.173142.zip', 'permissions': [], 'optional_permissions': [], 'host_permissions': []}, 'is_disabled': False, 'is_strict_compatibility_enabled': True, 'license': {'id': 3338, 'is_custom': False, 'name': {'en-US': 'Mozilla Public License 2.0'}, 'slug': 'MPL-2.0', 'text': {'en-US': 'This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at <a rel="nofollow" href="https://prod.outgoing.prod.webservices.mozgcp.net/v1/cfa1520767b4bc45590e23d48de7bec652c237ee0ad206608dfa16f2a0e21c8d/http%3A//mozilla.org/MPL/2.0/">http://mozilla.org/MPL/2.0/</a>.'}, 'url': 'http://www.mozilla.org/MPL/2.0/'}, 'release_notes': None, 'reviewed': None, 'source': None, 'version': '116.0.20230815.173142'}
2023-08-15 21:24:58,796 - scriptworker.client - ERROR - Failed to run async_main
Traceback (most recent call last):
  File "/app/lib/python3.9/site-packages/scriptworker/client.py", line 205, in _handle_asyncio_loop
    await async_main(context)
  File "/app/lib/python3.9/site-packages/addonscript/script.py", line 92, in async_main
    await asyncio.gather(*tasks)
  File "/app/lib/python3.9/site-packages/addonscript/script.py", line 52, in sign_addon
    signed_addon_info = await retry_async(
  File "/app/lib/python3.9/site-packages/scriptworker/utils.py", line 268, in retry_async
    return await func(*args, **kwargs)
  File "/app/lib/python3.9/site-packages/addonscript/api.py", line 137, in get_signed_addon_info
    raise FatalSignatureError("XPI disabled on AMO")
addonscript.exceptions.FatalSignatureError: XPI disabled on AMO
exit code: 1

Comment 1

[:wezhou]

@eviljeff @willdurand, can we get the correct command to run to fix this like last time (assuming it's the same cause)?

Thanks!

Comment 2

[William Durand [:willdurand]]

I am not sure what you mean. I'll let Mat reply.

Comment 3

[Andrew Williamson [:eviljeff]]

(In reply to William Durand [:willdurand] from comment #2)

I am not sure what you mean. I'll let Mat reply.

You provided the command last time in https://bugzilla.mozilla.org/show_bug.cgi?id=1837547#c5

This time it would be Version.objects.get(pk=5609404).delete(hard=True) but I'll reiterate that this is not a non-consequential "easy fix" and we shouldn't be doing it.

Comment 4

[Mathieu Pillard [:mat]]

We'll need to add https://github.com/mozilla/addons-server/issues/21096 to one of the next sprints - the behavior of preventing 2 listed versions from being awaiting review at the same time is extremely old and something we want to keep for the general add-on population, but for langpacks it looks like this is problematic.

Short-term, it's a terrible hack and something that shouldn't be done lightly, but I don't think we have a choice and we'll need to hard-delete that version to unblock the release.

Comment 5

[Dianna Smith [:diannaS]]

is there a time frame on this or a follow up on who can hard-delete that version? We are trying to rollout the desktop dot release as soon as possible.

Comment 6

[:wezhou]

I just ran the above command with the following output,

In [3]: Version.objects.get(pk=5609404).delete(hard=True)
{"Timestamp": 1692201375948792832, "Type": "z.versions", "Logger": "http_app_addons", "Hostname": "ip-172-31-33-49.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 1, "Fields": {"uid": "<anon>", "remoteAddressChain": "", "msg": "Version deleted: <Version: 116.0.20230815.173142> (5609404)"}, "severity": 200}
{"Timestamp": 1692201375966638336, "Type": "z.amo.activity", "Logger": "http_app_addons", "Hostname": "ip-172-31-33-49.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 4, "Pid": 1, "Fields": {"uid": "<anon>", "remoteAddressChain": "", "msg": "Activity log called with no user: 18"}, "severity": 400}
{"Timestamp": 1692201376035347200, "Type": "z.files", "Logger": "http_app_addons", "Hostname": "ip-172-31-33-49.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 1, "Fields": {"uid": "<anon>", "remoteAddressChain": "", "msg": "Removing filename: traditional_chinese_zh_tw_l-116.0.20230815.173142.zip for file: 4153745"}, "severity": 200}

There was a 400 level message, but I think the said version had been deleted.

Comment 7

[:wezhou]

(In reply to Mathieu Pillard [:mat] from comment #4)

Short-term, it's a terrible hack and something that shouldn't be done lightly, ...

This is True! Glad we have opened a ticket aiming to fix this.

Comment 8

[Dianna Smith [:diannaS]]

(rerun still failed) Since this had a few runs i think deleting pk=5609404 is not enough. These versions were also created by different runs:
5609402, 5609403, 5609405 and 5609406.

This should cover all the locales handled by the task.
do you think you can help us with that?

Comment 9

[:wezhou]

The listed 4 are deleted,

In [8]: for i in [5609402, 5609403, 5609405, 5609406]:
   ...:     Version.objects.get(pk=i).delete(hard=True)
   ...:
{"Timestamp": 1692212174065752832, "Type": "z.versions", "Logger": "http_app_addons", "Hostname": "ip-172-31-33-49.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 1, "Fields": {"uid": "<anon>", "remoteAddressChain": "", "msg": "Version deleted: <Version: 116.0.20230815.173142> (5609402)"}, "severity": 200}
{"Timestamp": 1692212174088127232, "Type": "z.amo.activity", "Logger": "http_app_addons", "Hostname": "ip-172-31-33-49.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 4, "Pid": 1, "Fields": {"uid": "<anon>", "remoteAddressChain": "", "msg": "Activity log called with no user: 18"}, "severity": 400}
{"Timestamp": 1692212174121345280, "Type": "z.files", "Logger": "http_app_addons", "Hostname": "ip-172-31-33-49.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 1, "Fields": {"uid": "<anon>", "remoteAddressChain": "", "msg": "Removing filename: uzbek_language_pack-116.0.20230815.173142.zip for file: 4153743"}, "severity": 200}
{"Timestamp": 1692212174323160064, "Type": "z.versions", "Logger": "http_app_addons", "Hostname": "ip-172-31-33-49.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 1, "Fields": {"uid": "<anon>", "remoteAddressChain": "", "msg": "Version deleted: <Version: 116.0.20230815.173142> (5609403)"}, "severity": 200}
{"Timestamp": 1692212174345498880, "Type": "z.amo.activity", "Logger": "http_app_addons", "Hostname": "ip-172-31-33-49.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 4, "Pid": 1, "Fields": {"uid": "<anon>", "remoteAddressChain": "", "msg": "Activity log called with no user: 18"}, "severity": 400}
{"Timestamp": 1692212174365661184, "Type": "z.files", "Logger": "http_app_addons", "Hostname": "ip-172-31-33-49.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 1, "Fields": {"uid": "<anon>", "remoteAddressChain": "", "msg": "Removing filename: vietnamese_language_pack-116.0.20230815.173142.zip for file: 4153744"}, "severity": 200}
{"Timestamp": 1692212174551376640, "Type": "z.versions", "Logger": "http_app_addons", "Hostname": "ip-172-31-33-49.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 1, "Fields": {"uid": "<anon>", "remoteAddressChain": "", "msg": "Version deleted: <Version: 116.0.20230815.173142> (5609405)"}, "severity": 200}
{"Timestamp": 1692212174571740416, "Type": "z.amo.activity", "Logger": "http_app_addons", "Hostname": "ip-172-31-33-49.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 4, "Pid": 1, "Fields": {"uid": "<anon>", "remoteAddressChain": "", "msg": "Activity log called with no user: 18"}, "severity": 400}
{"Timestamp": 1692212174591047936, "Type": "z.files", "Logger": "http_app_addons", "Hostname": "ip-172-31-33-49.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 1, "Fields": {"uid": "<anon>", "remoteAddressChain": "", "msg": "Removing filename: chinese_simplified_zh_cn_la-116.0.20230815.173142.zip for file: 4153746"}, "severity": 200}
{"Timestamp": 1692212174802134016, "Type": "z.versions", "Logger": "http_app_addons", "Hostname": "ip-172-31-33-49.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 1, "Fields": {"uid": "<anon>", "remoteAddressChain": "", "msg": "Version deleted: <Version: 116.0.20230815.173142> (5609406)"}, "severity": 200}
{"Timestamp": 1692212174824813056, "Type": "z.amo.activity", "Logger": "http_app_addons", "Hostname": "ip-172-31-33-49.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 4, "Pid": 1, "Fields": {"uid": "<anon>", "remoteAddressChain": "", "msg": "Activity log called with no user: 18"}, "severity": 400}
{"Timestamp": 1692212174848164352, "Type": "z.files", "Logger": "http_app_addons", "Hostname": "ip-172-31-33-49.us-west-2.compute.internal", "EnvVersion": "2.0", "Severity": 6, "Pid": 1, "Fields": {"uid": "<anon>", "remoteAddressChain": "", "msg": "Removing filename: xhosa_language_pack-116.0.20230815.173142.zip for file: 4153747"}, "severity": 200}

Comment 10

[Dianna Smith [:diannaS]]

THANK YOU! the task successfully ran now
https://firefox-ci-tc.services.mozilla.com/tasks/groups/PujSmkC6S8GCymEuQNlkEw

Comment 11

[Intermittent Failures Robot]

5 failures in 3076 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 5

Platform and build breakdown:

• linux64-shippable: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1848872&startday=2023-08-14&endday=2023-08-20&tree=all