1548973 - (armagadd-on-2.0) [meta] All extensions disabled due to expiration of intermediate signing cert

Status: RESOLVED FIXED

(Toolkit :: Add-ons Manager, defect, P1, blocker)

Description

[braiamp]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0

Steps to reproduce:

Wait until it's past midnight on 2019-05-04 UTC.

Actual results:

All addons got disabled due not having valid signature.

Expected results:

If the signature was due to expire, it should have been renewed weeks ago. Not all extensions were disabled. Fakespot and Google Scholar Button were left in their disabled state.

Some reports on reddit 1 says that they had their clocks a day forward, but they may be just early canaries for the actual widespread issue.

Going backwards in time allows installation from AMO but do not remove the unsupported mark from the add ons already installed.

Comment 1

[braiamp]

List of affected add-ons:

Activate Reader View	0.1.0	true	@activatereaderview
Netflix 1080p	1.8	true	{89d04aec-e93f-4f56-b77c-f2295051c13e}
Amazon Assistant for Firefox	10.1904.10.11834	false	[e-mail]
Amazon SMILE!	1.4.5	false	{1417a6e0-be73-4358-912c-5dce719b5791}
CanvasBlocker	0.5.8	false	[e-mail]
Check4Change	2.2.3	false	[e-mail]
Facebook Container	1.6.5	false	@contain-facebook
Fakespot - Analyze Fake Amazon Reviews	0.3.1	false	[e-mail]
Firefox Multi-Account Containers	6.1.0	false	@testpilot-containers
Ghostery – Privacy Ad Blocker	8.3.3	false	[e-mail]
Google Scholar Button	2.0	false	[e-mail]
Greasemonkey	4.7	false	{e4a8a97b-f2ed-450b-b12d-ee082ba24781}
Honey	11.1.0	false	jid1-93CWPmRbVPjRQA@jetpack
HTTPS Everywhere	2019.5.2.1	false	[e-mail]
InvisibleHand	6.6	false	[e-mail]
Kee - Password Manager	3.1.21	false	[e-mail]
Laboratory	3.0.5	false	[e-mail]
MEGA	3.57.9	false	[e-mail]
NflxMultiSubs (Netflix Multi. Subtitles)	1.6.7	false	{e7ca39ec-6668-455e-9768-db28c364e4d2}
NoScript	10.6.1	false	{73a6fe31-595d-460b-a920-fcc0f8843232}
ReviewMeta.com Review Analyzer	2.5	false	[e-mail]
Substital	2.1.0	false	jid1-Cn7LiNrWh4k6RA@jetpack
uBlock Origin	1.18.16	false	[e-mail]
User-Agent Switcher	1.2.11	false	[e-mail]

Note, only Activate Reader View and Netflix 1080p were tested to check possible workarounds. I would leave those disabled for now. Also Firefox own Multi-Account Containers was blocked.

Comment 2

[Asif Youssuff]

See also https://github.com/mozilla/addons/issues/978

Comment 3

[laurent]

I can confirm

Comment 4

[Andreea Pavel [:apavel]]

TREES ARE CLOSED FOR THIS.

Comment 9

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

(In reply to Andreea Pavel [:apavel] from comment #4)

TREES ARE CLOSED FOR THIS.

To clarify, XPCShell signing tests are failing because of the expired cert.

Comment 11

[braiamp]

Should other bug reports be opened about the empty error message that the browser console shows and related symptoms to help people know what it's going on? Or should that be implemented in a post-morten?

Comment 12

[Caspy7]

In case it's not understood I'm seeing a rash of reports of this across mozilla and freenode IRC networks as well as reddit.

Many people are very angry and it seems to be growing.

We don't yet know how broadly affected the user base is.
This seems like an urgent matter we want to get fixed as quickly as possible, at a high cost if necessary.

Comment 13

[Kris Maglione [:kmag]]

(In reply to Caspy7 from comment #12)

We don't yet know how broadly affected the user base is.

We do. All users with add-ons and remotely accurate system clocks are affected, with the possible exception of nightly/dev edition users with signing disabled.

Comment 15

[Habibullah Pagarkar]

CloudOps is taking a look at this

Comment 16

[Milos]

Can somehow signing be disabled?

Comment 17

[crea8ive49]

Confirming that add-ons were also disabled here on 66.0.3 (Win 10) at approx 9pm ET.

Comment 18

[braiamp]

(In reply to Milos from comment #16)

Can somehow signing be disabled?

Only on dev, nightly versions as :kmag noted.

Comment 19

[Outvi V]

(In reply to Milos from comment #16)

Can somehow signing be disabled?

I don't think so if you are using Firefox 48+ on PC: https://wiki.mozilla.org/Add-ons/Extension_Signing#Timeline

However, on Firefox for Android (at least up to 66.0.2), you can set xpinstall.signatures.required to false and bypass this problem.

Comment 20

[Alex]

Stable, Beta, Nightly and Dev got hit for me.

I took off from work early today but someone from work texted me earlier and said about 200+ dev stations (major media conglomerate most people have heard of at one point in their life or another) got fuzzed hard thanks to this expired certificate.

Really surprised Mozilla is this careless about renewals.

Comment 21

[braiamp]

(In reply to Alex from comment #20)

Stable, Beta, Nightly and Dev got hit for me.

I took off from work early today but someone from work texted me earlier and said about 200+ dev stations (major media conglomerate most people have heard of at one point in their life or another) got fuzzed hard thanks to this expired certificate.

Really surprised Mozilla is this careless about renewals.

The only difference between release and the others is that non-release versions allow you to disable the check. It comes enabled by default for obvious reasons, like how malicious actors convinced people to download Chrome Canary to sidestep the safeguards on their release versions.

Comment 22

[Kevin Brosnan [:kbrosnan]]

We have confirmed this issue. Extra comments about this being broken will not advance this bug to being fixed.

Comment 36

[Steve Fink [:sfink] [:s:]]

If you want to watch somewhere for user-facing updates on this issue, it looks like https://twitter.com/mozamo is the place to watch.

Comment 37

[Caspy7]

caitmuenster has also said this page will receive official updates/statuses on the issue:
https://discourse.mozilla.org/t/certificate-issue-causing-add-ons-to-be-disabled-or-fail-to-install/39047

Comment 47

[David Baron :dbaron: 🏴󠁵󠁳󠁣󠁡󠁿 ⌚UTC-8]

The dates on the cert in question were:

Not Before: May 4 00:09:46 2017 GMT
Not After : May 4 00:09:46 2019 GMT

Comment 53

[Sylvestre Ledru [:Sylvestre]]

For all the CC folks: we are making progress

Comment 57

[Eric Rescorla (:ekr)]

Update: We have rolled out a partial fix for this issue. We generated a new intermediate certificate with the same name/key but an updated validity window and pushed it out to users via Normandy (this should be most users). Users who have Normandy on should see their add-ons start working over the next few hours. We are continuing to work on packaging up the new certificate for users who have Normandy disabled.

Comment 58

[Guillaume Démésy [:magsout]]

See Also: -> https://webcompat.com/issues/30516, https://webcompat.com/issues/30483, https://webcompat.com/issues/30515, https://webcompat.com/issues/30511, https://webcompat.com/issues/30510, https://webcompat.com/issues/30507, https://webcompat.com/issues/30506, https://webcompat.com/issues/30500, https://webcompat.com/issues/30499, https://webcompat.com/issues/30487

Comment 59

[Russell Odom]

(In reply to Eric Rescorla (:ekr) from comment #57)

[...] Users who have Normandy on should see their add-ons start working over the next few hours. [...]

Will we need to take any action, e.g. quit and restart, or check for add-on updates, or will it just start working?

Comment 60

[Russell Odom]

For everyone's info: we don't need to so anything if ""studies" is enabled (Firefox Preferences -> Privacy & Security -> Allow Firefox to install and run studies).

See https://discourse.mozilla.org/t/certificate-issue-causing-add-ons-to-be-disabled-or-fail-to-install/39047/12

Thanks Eddi for the tip.

Comment 70

[Guillaume Démésy [:magsout]]

See Also: → https://webcompat.com/issues/30522, https://webcompat.com/issues/30529, https://webcompat.com/issues/30530, https://webcompat.com/issues/30534, https://webcompat.com/issues/30536, https://webcompat.com/issues/30539

Comment 72

[Emma Humphries, Bugmaster ☕️🎸🧞‍♀️✨ (she/her) [:emceeaich] (UTC-8) needinfo? me]

See https://blog.mozilla.org/addons/2019/05/04/update-regarding-add-ons-in-firefox/

Comment 78

[Guillaume Démésy [:magsout]]

See Also: → https://webcompat.com/issues/30556, https://webcompat.com/issues/30561, https://webcompat.com/issues/30562, https://webcompat.com/issues/30563, https://webcompat.com/issues/30564, https://webcompat.com/issues/30565, https://webcompat.com/issues/30566, https://webcompat.com/issues/30571, https://webcompat.com/issues/30572

Comment 79

[Intermittent Failures Robot]

124 failures in 56 pushes (2.214 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 124

Platform breakdown:

• linux64-qr: 16
• linux64: 36
• android-em-4-3-armv7-api16: 12
• linux32: 12
• osx-10-10: 1
• windows7-32: 2
• linux64-ccov: 5
• windows10-64: 2
• windows10-aarch64: 3
• osx-10-10-shippable: 1
• windows10-64-ccov: 8
• windows7-32-shippable: 1
• linux32-shippable: 8
• linux64-shippable-qr: 8
• linux64-shippable: 8
• windows10-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1548973&startday=2019-05-04&endday=2019-05-04&tree=all

Comment 89

[Guillaume Démésy [:magsout]]

See Also: → https://webcompat.com/issues/30576, https://webcompat.com/issues/30578, https://webcompat.com/issues/30584, https://webcompat.com/issues/30588, https://webcompat.com/issues/30600, https://webcompat.com/issues/30603, https://webcompat.com/issues/30637, https://webcompat.com/issues/30638, https://webcompat.com/issues/30643, https://webcompat.com/issues/30645

Comment 90

[Intermittent Failures Robot]

124 failures in 4131 pushes (0.03 failures/push) were associated with this bug in the last 7 days.

This is the #16 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 124

Platform breakdown:

• linux64-qr: 16
• linux64: 36
• android-em-4-3-armv7-api16: 12
• linux32: 12
• osx-10-10: 1
• windows7-32: 2
• linux64-ccov: 5
• windows10-64: 2
• windows10-aarch64: 3
• osx-10-10-shippable: 1
• windows10-64-ccov: 8
• windows7-32-shippable: 1
• linux32-shippable: 8
• linux64-shippable-qr: 8
• linux64-shippable: 8
• windows10-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1548973&startday=2019-04-29&endday=2019-05-05&tree=all

Comment 92

[Markus Popp]

(When) can http://ftp.mozilla.org/pub/firefox/candidates/ be opened again?

Comment 93

[Jonathan Kamens]

Please document here the root cause of this issue and what is being done to ensure that it does not happen again, or if that has already been documented elsewhere, please post a link to that documentation here. I've reviewed all the comments on this bug as well as the other information sources to which it links, and I don't see an explanation of the root cause or of steps being taken to prevent recurrence.

Comment 94

[sjw]

Please see https://support.mozilla.org/en-US/kb/add-ons-disabled-or-fail-to-install-firefox for current solutions and workarounds.
Status updates are posted here: https://blog.mozilla.org/addons/2019/05/04/update-regarding-add-ons-in-firefox/

Comment 97

[j.j.]

is the part
"[first mitigation completed, working on a second one]"
in the bug title meanigful in any way?