Closed Bug 1548973 (armagadd-on-2.0) Opened 5 years ago Closed 5 years ago

[meta] All extensions disabled due to expiration of intermediate signing cert

Categories

(Toolkit :: Add-ons Manager, defect, P1)

Product:
Toolkit
Component:
Add-ons Manager
Version:
66 Branch
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
relnote-firefox --- 66+
firefox-esr60 --- fixed
firefox-esr68 --- fixed
firefox66 --- fixed
firefox67 --- fixed
firefox68 --- fixed

People

(Reporter: braiamp, Unassigned)

References

Details

(Keywords: dogfood, meta, Whiteboard: cert2019)

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0

Steps to reproduce:

Wait until it's past midnight on 2019-05-04 UTC.

Actual results:

All addons got disabled due not having valid signature.

Expected results:

If the signature was due to expire, it should have been renewed weeks ago. Not all extensions were disabled. Fakespot and Google Scholar Button were left in their disabled state.

Some reports on reddit 1 says that they had their clocks a day forward, but they may be just early canaries for the actual widespread issue.

Going backwards in time allows installation from AMO but do not remove the unsupported mark from the add ons already installed.

List of affected add-ons:

Activate Reader View	0.1.0	true	@activatereaderview
Netflix 1080p	1.8	true	{89d04aec-e93f-4f56-b77c-f2295051c13e}
Amazon Assistant for Firefox	10.1904.10.11834	false	abb@amazon.com
Amazon SMILE!	1.4.5	false	{1417a6e0-be73-4358-912c-5dce719b5791}
CanvasBlocker	0.5.8	false	CanvasBlocker@kkapsner.de
Check4Change	2.2.3	false	check4change-owner@mozdev.org
Facebook Container	1.6.5	false	@contain-facebook
Fakespot - Analyze Fake Amazon Reviews	0.3.1	false	contact@fakespot.com
Firefox Multi-Account Containers	6.1.0	false	@testpilot-containers
Ghostery – Privacy Ad Blocker	8.3.3	false	firefox@ghostery.com
Google Scholar Button	2.0	false	button@scholar.google.com
Greasemonkey	4.7	false	{e4a8a97b-f2ed-450b-b12d-ee082ba24781}
Honey	11.1.0	false	jid1-93CWPmRbVPjRQA@jetpack
HTTPS Everywhere	2019.5.2.1	false	https-everywhere@eff.org
InvisibleHand	6.6	false	canitbecheaper@trafficbroker.co.uk
Kee - Password Manager	3.1.21	false	keefox@chris.tomlinson
Laboratory	3.0.5	false	1b2383b324c8520974ee097e46301d5ca4e076de387c02886f1c6b1503671586@pokeinthe.io
MEGA	3.57.9	false	firefox@mega.co.nz
NflxMultiSubs (Netflix Multi. Subtitles)	1.6.7	false	{e7ca39ec-6668-455e-9768-db28c364e4d2}
NoScript	10.6.1	false	{73a6fe31-595d-460b-a920-fcc0f8843232}
ReviewMeta.com Review Analyzer	2.5	false	FirefoxExtension@ReviewMeta.com
Substital	2.1.0	false	jid1-Cn7LiNrWh4k6RA@jetpack
uBlock Origin	1.18.16	false	uBlock0@raymondhill.net
User-Agent Switcher	1.2.11	false	user-agent-switcher@ninetailed.ninja

Note, only Activate Reader View and Netflix 1080p were tested to check possible workarounds. I would leave those disabled for now. Also Firefox own Multi-Account Containers was blocked.

I can confirm

Severity: normal → blocker
Status: UNCONFIRMED → NEW
status-firefox66: --- → affected
status-firefox67: --- → affected
status-firefox68: --- → affected
status-firefox-esr60: --- → affected
Component: Untriaged → Add-ons Manager
Ever confirmed: true
Product: Firefox → Toolkit
Summary: Firefox marked addons due signing as unsupported, but doesn't allow re-downloads from AMO → All extensions disabled due to expiration of intermediate signing cert
See Also: → 1548975

TREES ARE CLOSED FOR THIS.

(In reply to Andreea Pavel [:apavel] from comment #4)

TREES ARE CLOSED FOR THIS.

To clarify, XPCShell signing tests are failing because of the expired cert.

See Also: 1548975 →

Should other bug reports be opened about the empty error message that the browser console shows and related symptoms to help people know what it's going on? Or should that be implemented in a post-morten?

Flags: needinfo?(ddurst)

In case it's not understood I'm seeing a rash of reports of this across mozilla and freenode IRC networks as well as reddit.

Many people are very angry and it seems to be growing.

We don't yet know how broadly affected the user base is.
This seems like an urgent matter we want to get fixed as quickly as possible, at a high cost if necessary.

Alias: armagadd-on-2.0
Flags: needinfo?(dveditz)
Flags: needinfo?(dkeeler)
See Also: → armagadd-on

(In reply to Caspy7 from comment #12)

We don't yet know how broadly affected the user base is.

We do. All users with add-ons and remotely accurate system clocks are affected, with the possible exception of nightly/dev edition users with signing disabled.

CloudOps is taking a look at this

Can somehow signing be disabled?

Confirming that add-ons were also disabled here on 66.0.3 (Win 10) at approx 9pm ET.

(In reply to Milos from comment #16)

Can somehow signing be disabled?

Only on dev, nightly versions as :kmag noted.

(In reply to Milos from comment #16)

Can somehow signing be disabled?

I don't think so if you are using Firefox 48+ on PC: https://wiki.mozilla.org/Add-ons/Extension_Signing#Timeline

However, on Firefox for Android (at least up to 66.0.2), you can set xpinstall.signatures.required to false and bypass this problem.

We have confirmed this issue. Extra comments about this being broken will not advance this bug to being fixed.

Restrict Comments: true
Blocks: 1549013

If you want to watch somewhere for user-facing updates on this issue, it looks like https://twitter.com/mozamo is the place to watch.

caitmuenster has also said this page will receive official updates/statuses on the issue:
https://discourse.mozilla.org/t/certificate-issue-causing-add-ons-to-be-disabled-or-fail-to-install/39047

Blocks: 1549017
Flags: needinfo?(dveditz)
Priority: -- → P1

The dates on the cert in question were:

Not Before: May 4 00:09:46 2017 GMT
Not After : May 4 00:09:46 2019 GMT

Summary: All extensions disabled due to expiration of intermediate signing cert → [work in progress] All extensions disabled due to expiration of intermediate signing cert

For all the CC folks: we are making progress

Update: We have rolled out a partial fix for this issue. We generated a new intermediate certificate with the same name/key but an updated validity window and pushed it out to users via Normandy (this should be most users). Users who have Normandy on should see their add-ons start working over the next few hours. We are continuing to work on packaging up the new certificate for users who have Normandy disabled.

Summary: [work in progress] All extensions disabled due to expiration of intermediate signing cert → [first mitigation completed, working on a second one] All extensions disabled due to expiration of intermediate signing cert

(In reply to Eric Rescorla (:ekr) from comment #57)

[...] Users who have Normandy on should see their add-ons start working over the next few hours. [...]

Will we need to take any action, e.g. quit and restart, or check for add-on updates, or will it just start working?

For everyone's info: we don't need to so anything if ""studies" is enabled (Firefox Preferences -> Privacy & Security -> Allow Firefox to install and run studies).

See https://discourse.mozilla.org/t/certificate-issue-causing-add-ons-to-be-disabled-or-fail-to-install/39047/12

Thanks Eddi for the tip.

Depends on: 1549061
Blocks: 1549078
Depends on: 1549116
No longer depends on: 1549116
Blocks: 1549117
No longer blocks: 1549117
Blocks: 1549121
No longer blocks: 1549121
Depends on: 1549129
See Also: → 1549134
See Also: 1549134 →
Depends on: 1549145
No longer depends on: 1549129
See Also: → 1549129
See Also: → 1549216
See Also: → 1549224
Depends on: 1549249
Blocks: 1549266
Depends on: 1549305
Depends on: 1549258

(When) can http://ftp.mozilla.org/pub/firefox/candidates/ be opened again?

Depends on: 1549400

Please document here the root cause of this issue and what is being done to ensure that it does not happen again, or if that has already been documented elsewhere, please post a link to that documentation here. I've reviewed all the comments on this bug as well as the other information sources to which it links, and I don't see an explanation of the root cause or of steps being taken to prevent recurrence.

Depends on: 1549441
relnote-firefox: --- → 66+
Whiteboard: [stockwell needswork:owner] → [stockwell needswork:owner] cert2019
Depends on: 1549604
Depends on: 1549905

is the part
"[first mitigation completed, working on a second one]"
in the bug title meanigful in any way?

No longer depends on: 1549305
Depends on: 1551321
Summary: [first mitigation completed, working on a second one] All extensions disabled due to expiration of intermediate signing cert → All extensions disabled due to expiration of intermediate signing cert
Depends on: 1552392
Flags: needinfo?(ddurst)
Whiteboard: [stockwell needswork:owner] cert2019 → cert2019
Keywords: meta
Summary: All extensions disabled due to expiration of intermediate signing cert → [meta] All extensions disabled due to expiration of intermediate signing cert
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
See Also: → 1549018
You need to log in before you can comment on or make changes to this bug.