1549249 - 66.0.4 isn't applying the intermediate certificate for some users (throwing with SEC_ERROR_TOKEN_NOT_LOGGED_IN, possibly antivirus or master password related)

Status: VERIFIED FIXED

(Toolkit :: General, defect, P1)

Description

[Brian Grinstead [:bgrins]]

+++ This bug was initially created as a clone of Bug #1549061 +++

Starting around https://bugzilla.mozilla.org/show_bug.cgi?id=1549061#c19 we've been investigating why some users aren't seeing the cert installed (and the extensions.signer.hotfixed pref being set).

The error seen in Comment 26 in that bug is:

1557097728294 addons.xpi ERROR failed to add new intermediate certificate:: [Exception... "Component returned failure code: 0x805a1f65 [nsIX509CertDB.addCertFromBase64]" nsresult: "0x805a1f65 (<unknown>)" location: "JS frame :: resource://gre/modules/addons/XPIProvider.jsm :: addMissingIntermediateCertificate :: line 1896" data: no] Stack trace: addMissingIntermediateCertificate()@resource://gre/modules/addons/XPIProvider.jsm:1896

Which is causing an exception at https://hg.mozilla.org/releases/mozilla-release/rev/848b15028562c6757748070f637e0e4f0bbb5f65#l1.25 and thus not injecting the cert or setting the pref.

Some initial investigation indicates that this is from SEC_ERROR_TOKEN_NOT_LOGGED_IN and that it may be related to the user having a Master Password. Testing locally on Linux and OSX doesn't reproduce though. Some users also report having antivirus installed, so there may be some interaction between the two. Or SEC_ERROR_TOKEN_NOT_LOGGED_IN could be coming from something else.

Comment 1

[Brian Grinstead [:bgrins]]

Moving needinfos from Bug 1549061

Comment 5

[Brian Grinstead [:bgrins]]

For people being referred here from duplicate bugs or posts, we are still working to narrow down exactly what is causing this problem. If you are seeing the problem where your extensions haven't been re-enabled in 66.0.4 and you'd like to help, could you please answer the following questions:

1. What OS are you on?
2. Do you have antivirus software installed? If so, which one?
3. Do you have a Master Password? If so, could you check if the workaround in Comment 11 fixes the problem for you?
4. Could you check with a brand new profile on 66.0.4 and see if you can install addons from there?
5. At startup, if you open the browser console (Ctrl+Shift+J or Cmd+Shift+J on OSX), do you see the error "failed to add new intermediate certificate"? If so, do you see the code 0x805a1f65 next to it, or something else?

Comment 6

[sgtyrrell]

1. Windows 10, 64bit, 1809
2. F-secure AV, but I did disable it at various points with no noticeable change.
3. Yes, but I don't use FF for logins any more, so hadn't been prompted to enter it.
4. This allowed me to add extensions.
5. I see that code.

Entering the master password (by virtue of making a FF account which in turn prompted me for my MP at the next startup) re-enabled extensions on my default profile.

Comment 7

[Paul Theriault [:pauljt] (no longer reading bugmail)]

Ok, the issue seems to be if you created a master password with the OLD preferences UI (not the new in-content one). That's a guess but I have an STR which is supports it.

STR (its not pretty):

1. create a profile in Firefox 45
2. create a master password
3. close Firefox 45
4. Open profile in Firefox 66.0.4

Expected:
Cert is added and addons work.

Results:
Browser Console shows errors below indicating that both the dot release patch AND normandy fix fail (obviously, since they are both trying to access the certdb while it is still locked).

1557114647849	addons.xpi	ERROR	failed to add new intermediate certificate:: [Exception... "Component returned failure code: 0x805a1f65 [nsIX509CertDB.addCertFromBase64]"  nsresult: "0x805a1f65 (<unknown>)"  location: "JS frame :: resource://gre/modules/addons/XPIProvider.jsm :: addMissingIntermediateCertificate :: line 1896"  data: no] Stack trace: addMissingIntermediateCertificate()@resource://gre/modules/addons/XPIProvider.jsm:1896
startup()@resource://gre/modules/addons/XPIProvider.jsm:2144
callProvider()@resource://gre/modules/AddonManager.jsm:203
_startProvider()@resource://gre/modules/AddonManager.jsm:652
startup()@resource://gre/modules/AddonManager.jsm:805
startup()@resource://gre/modules/AddonManager.jsm:2775
observe()@jar:file:///C:/Program%20Files/Mozilla%20Firefox/omni.ja!/components/addonManager.js:66 Log.jsm:679
    append resource://gre/modules/Log.jsm:679
    log resource://gre/modules/Log.jsm:360
    error resource://gre/modules/Log.jsm:368
    addMissingIntermediateCertificate resource://gre/modules/addons/XPIProvider.jsm:1899
    startup resource://gre/modules/addons/XPIProvider.jsm:2144
    callProvider resource://gre/modules/AddonManager.jsm:203
    _startProvider resource://gre/modules/AddonManager.jsm:652
    startup resource://gre/modules/AddonManager.jsm:805
    startup resource://gre/modules/AddonManager.jsm:2775
    observe jar:file:///C:/Program Files/Mozilla Firefox/omni.ja!/components/addonManager.js:66
1557114647884	addons.xpi-utils	WARN	Could not find source bundle for add-on [e-mail]: [Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIFile.initWithPath]"  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: resource://gre/modules/addons/XPIDatabase.jsm :: parseDB/< :: line 1340"  data: no] Stack trace: parseDB/<()@resource://gre/modules/addons/XPIDatabase.jsm:1340
parseDB()@resource://gre/modules/addons/XPIDatabase.jsm:1338
asyncLoadDB/this._dbPromise<()@resource://gre/modules/addons/XPIDatabase.jsm:1409
awaitPromise()@resource://gre/modules/addons/XPIProvider.jsm:186
syncLoadDB()@resource://gre/modules/addons/XPIDatabase.jsm:1291
checkForChanges()@resource://gre/modules/addons/XPIProvider.jsm:2595
startup()@resource://gre/modules/addons/XPIProvider.jsm:2175
callProvider()@resource://gre/modules/AddonManager.jsm:203
_startProvider()@resource://gre/modules/AddonManager.jsm:652
startup()@resource://gre/modules/AddonManager.jsm:805
startup()@resource://gre/modules/AddonManager.jsm:2775
observe()@jar:file:///C:/Program%20Files/Mozilla%20Firefox/omni.ja!/components/addonManager.js:66
1557114647884	addons.xpi-utils	ERROR	Failed to load XPI JSON data from profile: Error: Expected passed argument to contain a path(resource://gre/modules/addons/XPIDatabase.jsm:262:15) JS Stack trace: [e-mail]:262:15
parseDB/<@XPIDatabase.jsm:1348:24
[e-mail]:1338:13
asyncLoadDB/this._dbPromise<@XPIDatabase.jsm:1409:15
[e-mail]:186:3
[e-mail]:1291:7
[e-mail]:2595:9
[e-mail]:2175:25
[e-mail]:203:12
[e-mail]:652:5
[e-mail]:805:9
[e-mail]:2775:5
[e-mail]:66:9 Log.jsm:679
1557114647884	addons.xpi-utils	WARN	Rebuilding add-ons database from installed extensions.
1557114648001	[e-mail]	WARN	Loading extension '[e-mail]': Reading manifest: Invalid extension permission: mozillaAddons
1557114648001	[e-mail]	WARN	Loading extension '[e-mail]': Reading manifest: Invalid extension permission: resource://pdf.js/
1557114648002	[e-mail]	WARN	Loading extension '[e-mail]': Reading manifest: Invalid extension permission: about:reader*
1557114648085	[e-mail]	WARN	Loading extension '[e-mail]': Reading manifest: Invalid extension permission: mozillaAddons
1557114648085	[e-mail]	WARN	Loading extension '[e-mail]': Reading manifest: Invalid extension permission: resource://pdf.js/
1557114648085	[e-mail]	WARN	Loading extension '[e-mail]': Reading manifest: Invalid extension permission: about:reader*
Key event not available on some keyboard layouts: key=“i” modifiers=“accel,alt,shift” id=“key_browserToolbox” browser.xul
Use of nsIFile in content process is deprecated.
NetUtil.jsm:259:12
Source map error: TypeError: NetworkError when attempting to fetch resource.
Resource URL: resource://activity-stream/css/activity-stream.css
Source Map URL: activity-stream-windows.css.map[Learn More]
WebExtensions: failed to add new intermediate certificate: 
Exception { name: "", message: "Component returned failure code: 0x805a1f65 [nsIX509CertDB.addCertFromBase64]", result: 2153389925, filename: "jar:file:///C:/Users/User/AppData/Roaming/Mozilla/Firefox/Profiles/kqy0emxk.FF45/extensions/[e-mail]!/experiments/skeleton/api.js", lineNumber: 14, columnNumber: 0, data: null, stack: "doTheThing@jar:file:///C:/Users/User/AppData/Roaming/Mozilla/Firefox/Profiles/kqy0emxk.FF45/extensions/[e-mail]!/experiments/skeleton/api.js:14:15\ncall/result</<@resource://gre/modules/ExtensionParent.jsm:950:49\nwithPendingBrowser@resource://gre/modules/ExtensionParent.jsm:604:26\ncall/result<@resource://gre/modules/ExtensionParent.jsm:949:16\nwithTiming@resource://gre/modules/ExtensionParent.jsm:916:14\ncall@resource://gre/modules/ExtensionParent.jsm:948:20\n", location: XPCWrappedNative_NoHelper }
api.js:17
WebExtensions: signatures re-verified api.js:23

Comment 8

[Paul Theriault [:pauljt] (no longer reading bugmail)]

NB: it probably doesn't have to be as old as FF45, i just wanted to make sure I got the old version of about:preferences.

Comment 9

[Paul Theriault [:pauljt] (no longer reading bugmail)]

Note: I have also confirmed that once you have entered your master password the certificate can be added via the hotfix/dot realase script , and addons work again.

Comment 10

[sgtyrrell]

(In reply to Paul Theriault [:pauljt] from comment #7)

Ok, the issue seems to be if you created a master password with the OLD preferences UI (not the new in-content one). That's a guess but I have an STR which is supports it.

STR (its not pretty):

1. create a profile in Firefox 45
2. create a master password
3. close Firefox 45
4. Open profile in Firefox 66.0.4 ...

I've kept my profile from prehistoric times, so this STR reflects my likely situation.

Comment 11

[Paul Theriault [:pauljt] (no longer reading bugmail)]

Workaround: change your master password (even to the same value) and restart your browser. For me, the patch landed in 66.0.4 worked after I "changed" my password.

To see how to change your password see: https://support.mozilla.org/en-US/kb/use-master-password-protect-stored-logins?#w_changing-the-master-password

Comment 12

[Paul Theriault [:pauljt] (no longer reading bugmail)]

(In reply to sgtyrrell from comment #10)

(In reply to Paul Theriault [:pauljt] from comment #7)

Ok, the issue seems to be if you created a master password with the OLD preferences UI (not the new in-content one). That's a guess but I have an STR which is supports it.

STR (its not pretty):

1. create a profile in Firefox 45
2. create a master password
3. close Firefox 45
4. Open profile in Firefox 66.0.4 ...

I've kept my profile from prehistoric times, so this STR reflects my likely situation.

sgtyrrell Can you try the workaround of changing your master password and see if it helps?

Comment 13

[sgtyrrell]

(In reply to Paul Theriault [:pauljt] from comment #12)

(In reply to sgtyrrell from comment #10)

(In reply to Paul Theriault [:pauljt] from comment #7)

Ok, the issue seems to be if you created a master password with the OLD preferences UI (not the new in-content one). That's a guess but I have an STR which is supports it.

STR (its not pretty):

1. create a profile in Firefox 45
2. create a master password
3. close Firefox 45
4. Open profile in Firefox 66.0.4 ...

I've kept my profile from prehistoric times, so this STR reflects my likely situation.

sgtyrrell Can you try the workaround of changing your master password and see if it helps?

I'm already up and running.

My work-around was to create a FF account (I did not have one). The next time I restarted it asked me for my master password, and that got everything working. The act of entering my existing master password did the trick for me - as I don't use FF for other password logins I'd not been asked for it for many many months.

I guess the conclusion is that there doesn't need to be a new MP, just that the user has to have authenticated the MP?

Comment 14

[Kris Maglione [:kmag]]

(In reply to sgtyrrell from comment #13)

I guess the conclusion is that there doesn't need to be a new MP, just that the user has to have authenticated the MP?

It's more complicated than that. Setting the master password in a recent Firefox version removes the issue of it being required in order to insert a certificate into the DB. Entering the master password before we try to inject the certificate, in the same session, works around it for that one particular injection.

[edit] It looks like just entering the password is indeed enough to cause the cert DB to migrate to the new format and fix the problem.

Comment 16

[Paul Theriault [:pauljt] (no longer reading bugmail)]

Actually there is an easier work around - you just have to

1. enter your master password
2. restart your browser

You are prompted to enter your master password in a couple of situations:
a) when you click "View Saved Logins" in the security section of about:preferences
b) if you create a sync account
c) if if you have a sync enabled you are prompted on login
d) when you try to save credentials from a website.

So if you are blocked the easier is probably (a) - ie try to view your saved logins and then you will have to enter your password.

Note that you DO have to restart after entering your password.

---

On a technical level, the theory being discussed in channel is that cert8 -> cert9 db migration is blocked on authenticating.

Comment 17

[Paul Theriault [:pauljt] (no longer reading bugmail)]

PS To view your saved logins, see instructions here: https://support.mozilla.org/sk/kb/password-manager-remember-delete-change-and-import#w_viewing-and-deleting-passwords

Comment 18

[Paul Theriault [:pauljt] (no longer reading bugmail)]

If you have forgotten your password, you can reset it using the instructions here: https://support.mozilla.org/sk/kb/password-manager-remember-delete-change-and-import#w_viewing-and-deleting-passwords

Comment 19

[Cameron McCormack (:heycam)]

Current thought is that we can add the certificate in C++ in AppTrustDomain, like we do for the addons public root. That will make the certificate be present without running into problems with adding to the certdb when a master password is enabled.

Comment 20

[Emma Mason]

From reddit:

https://www.reddit.com/r/firefox/comments/bl6x0f/why_is_firefox_suddenly_asking_for_my_master/emm71gv/

User has

Windows 7
Avast free version
master password enabled
add-ons are working
no error seen in console

updating the master password doesn't fix the issue.

Comment 21

[Matthew N. [:MattN]]

In reply to comment 20:

That sounds like bug 1541927 which isn't related to the add-ons issue.

Comment 22

[Mark Goodwin [:mgoodwin]]

(In reply to Cameron McCormack (:heycam) from comment #19)

Current thought is that we can add the certificate in C++ in AppTrustDomain, like we do for the addons public root. That will make the certificate be present without running into problems with adding to the certdb when a master password is enabled.

For more detail, see how other TrustDomains (e.g. CSTrustDomain) make data available in path building. E.g. the code around here https://searchfox.org/mozilla-central/rev/b2015fdd464f598d645342614593d4ebda922d95/security/manager/ssl/CSTrustDomain.cpp#122

Something analogous for the cert used in the hotfix could do the job here.

Comment 23

[skyhook]

Came from reddit, see Emma Mason's comment above.

My issue is that every time I launch Firefox, it asks for the Master Password.
This began during the missing extensions bug and continues now with ver. 66.0.4.
I do not use sync.
I do not use Pocket.
New page, new tab and home page are set to Blank Page (meaning Firefox's homepage is not a website that requires a password.)
I have installed 66.0.4. My addons are back and are working fine.

Windows 7
Avast Free
Yes, I have a Master Password.
I have entered the Master Password. This did not fix my issue.
I have changed the Master Password. This did not fix my issue.
I have no problem installing addons.
I do not find any error in the console.

Comment 24

[skyhook]

Apparently, I am in the wrong thread. Apologies.

Comment 25

[Cameron McCormack (:heycam)]

Apparently, I am in the wrong thread. Apologies.

You might be looking for bug 1541927.

Comment 26

[Steve]

I've installed the 66.0.4, everything was fixed after a restart.
Now it's back to all blocked again. (version is fine)

What OS are you on?
win7 x64

Do you have antivirus software installed?
Nope

Do you have a Master Password?
Nope

At startup, if you open the browser console (Ctrl+Shift+J or Cmd+Shift+J on OSX), do you see the error...
This is the contents of the console: (I've got two pinned tabs, one to aws, one to a local myAdmin)

17:01:42.113 Content Security Policy: Ignoring “'self'” within script-src: ‘strict-dynamic’ specified
17:01:42.113 Content Security Policy: Ignoring “'unsafe-inline'” within script-src: ‘strict-dynamic’ specified
17:01:42.113 Content Security Policy: Ignoring “'self'” within script-src: ‘strict-dynamic’ specified
17:01:42.113 Content Security Policy: Ignoring “'unsafe-inline'” within script-src: ‘strict-dynamic’ specified
17:01:42.205 Content Security Policy: Ignoring “'self'” within script-src: ‘strict-dynamic’ specified
17:01:42.205 Content Security Policy: Ignoring “'unsafe-inline'” within script-src: ‘strict-dynamic’ specified
17:01:42.660 The script from “https://phd.aws.amazon.com/phd/auth?state=hashArgs” was loaded even though its MIME type (“text/plain”) is not a valid JavaScript MIME type.[Learn More] home
17:01:43.021 The ‘content’ attribute of Window objects is deprecated. Please use ‘window.top’ instead. home:313:23
17:01:49.252 This site appears to use a scroll-linked positioning effect. This may not work well with asynchronous panning; see https://developer.mozilla.org/docs/Mozilla/Performance/ScrollLinkedEffects for further details and to join the discussion on related tools and features! home
17:02:11.233 Key event not available on some keyboard layouts: key=“i” modifiers=“accel,alt,shift” id=“key_browserToolbox” browser.xul

Comment 27

[Cameron McCormack (:heycam)]

Paul's workaround is now summarised at https://support.mozilla.org/en-US/kb/add-ons-disabled-or-fail-to-install-firefox#w_master-password

Comment 28

[sjw]

Could this be added to the blog post and release notes?

Comment 29

[Julien Cristau [:jcristau]]

Added as a known issue to the 66.0.4 and 60.6.2 release notes.

Comment 30

[Cameron McCormack (:heycam)]

Kev, should this be mentioned in the blog post?

Comment 31

[ypetrov]

I have the same problem.

What OS are you on?

MacOS 10.12.6

Do you have antivirus software installed? If so, which one?

No.

Do you have a Master Password? If so, could you check if the workaround in Comment 11 fixes the problem for you?

No. And I can't set it because error is shown:
https://i.imgur.com/lkjzCfk.png

Could you check with a brand new profile on 66.0.4 and see if you can install addons from there?

Yes. Addons are working in a new profile.

At startup, if you open the browser console (Ctrl+Shift+J or Cmd+Shift+J on OSX), do you see the error "failed to add new intermediate certificate"? If so, do you see the code 0x805a1f65 next to it, or something else?

I see different error with same text and code 0x805a1fe8.

Comment 33

[fu.ruisi]

(In reply to skyhook from comment #23)

I reset my Master Password, and all add-on is back.

for reset master password, I used the guide below.
https://support.mozilla.org/sw/kb/reset-your-master-password-if-you-forgot-it

Came from reddit, see Emma Mason's comment above.

My issue is that every time I launch Firefox, it asks for the Master Password.
This began during the missing extensions bug and continues now with ver. 66.0.4.
I do not use sync.
I do not use Pocket.
New page, new tab and home page are set to Blank Page (meaning Firefox's homepage is not a website that requires a password.)
I have installed 66.0.4. My addons are back and are working fine.

Windows 7
Avast Free
Yes, I have a Master Password.
I have entered the Master Password. This did not fix my issue.
I have changed the Master Password. This did not fix my issue.
I have no problem installing addons.
I do not find any error in the console.

Comment 34

[cgemici]

Just commenting to add more possibly related context. On my Nightly 68.0a1 on Ubuntu, my addons worked after the fix but for some reason all three themes (default/light/dark) were showing as disabled and I wasn't able to do anything to change them from the themes or customize window. I didn't have a master password during or after the breakage of the add-ons. (But have had one before) Setting a master password fixed that as well.

Comment 35

[ypetrov]

Fixed it by changing ownership of files in the profile folder. For some reason cert9.db, key4.db and pkcs11.txt were owned by root. Now all addons are enabled.

Comment 36

[Brian Grinstead [:bgrins]]

(In reply to ypetrov from comment #35)

Fixed it by changing ownership of files in the profile folder. For some reason cert9.db, key4.db and pkcs11.txt were owned by root. Now all addons are enabled.

Great! Sounds like fixing file permissions is a workaround for the error code in Comment 31, then (0x805a1fe8).

Comment 37

[Daniel Cicas [:dcicas], Release QA]

Hello,

QA looked into this issue and we tested on the following platforms Windows 10 & Windows 7 (both x64) using a Master Password the following Antiviruses:
*F-Secure Total
*Avast Free Antivirus
*Kaspersky Internet Security
*ESET Smart Security Premium
*McAfee Total Protection

We managed to reproduce this issue on every configuration, and we also verified that the workaround works.

If you want to see in more detail what we tested here is a link to the document https://tinyurl.com/y2ocwush

Comment 38

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

As discussed elsewhere, I think a good solution for this would be to compile in the new intermediate like we do the root. I can work on a patch for this.

Comment 40

[Brian Grinstead [:bgrins]]

From https://bugzilla.mozilla.org/show_bug.cgi?id=1549441#c3, there was a user who was seeing the 0x805a1fe8 error referenced in Comment 31, but the file was already writable: https://discourse.mozilla.org/t/fix-66-04-does-notwork/39824/17?u=freaktechnik. They reported that backing up and deleting those three files (cert9.db, key4.db and pkcs11.txt) resolved the problem for them: https://discourse.mozilla.org/t/fix-66-04-does-notwork/39824/18?u=freaktechnik).

Is this an OK thing to suggest as a workaround for others seeing this problem?

Comment 41

[Matthew N. [:MattN]]

(In reply to Brian Grinstead [:bgrins] from comment #40)

Is this an OK thing to suggest as a workaround for others seeing this problem?

That will cause the user to lose access to all their saved logins, including their FxA/Sync one, and any saved certificates so it should have a large/obvious warning like https://support.mozilla.org/en-US/kb/add-ons-disabled-or-fail-to-install-firefox#w_master-password does for the reset path.

Comment 42

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Attachment: bug 1549249 - hard-code new add-on signing intermediate so it's always available r?jcj,kmag

Our previous approach to making this intermediate available relied on being able
to add it to the user's NSS cert DB. This does work in the majority of cases,
but there are some situations where it doesn't work (e.g. if the user's DB is
set to read only, if they've configured Firefox to run in "nocertdb" mode, if
they have a master password but forgot it, and so on). This patch compiles the
intermediate in to Firefox in the same way we incorporate the root, so it should
always be available.

At the same time, this patch reverts the changes from
023dd959512e2cfa685187616560f91efa91183c and
1d35f8d88bdd007e01d42c4ff76c6d10d7c01a98 (the patches that implemented the
original approach) because they should no longer be necessary.

Comment 43

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Hi Dana, can you confirm whether ESR60 is affected or not by this? My assumption is yet but would like to get a confirmation. Thanks!

Comment 44

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/c52835481c08
hard-code new add-on signing intermediate so it's always available r=jcj,kmag a=ryanvm

Comment 45

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

We want this on all supported branches.

Comment 46

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Marking ESR60 as affected.

Comment 47

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Attachment: bug 1549249 - hard-code new add-on signing intermediate so it's always available (esr60 version) r=jcj,kmag

Our previous approach to making this intermediate available relied on being able
to add it to the user's NSS cert DB. This does work in the majority of cases,
but there are some situations where it doesn't work (e.g. if the user's DB is
set to read only, if they've configured Firefox to run in "nocertdb" mode, if
they have a master password but forgot it, and so on). This patch compiles the
intermediate in to Firefox in the same way we incorporate the root, so it should
always be available.

At the same time, this patch reverts the changes from
537700ea54aaceda64e1e5395085e536e1c9d3e3 (the patch that implemented the
original approach) because it should no longer be necessary.

Comment 48

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Attachment: bug 1549249 - follow-up to bump add-on db schema so users pick up the changes faster r?kmag

Comment 49

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/d5a6ca2d0e9d
follow-up to bump add-on db schema so users pick up the changes faster r=kmag a=ryanvm

Comment 50

[Liz Henry (:lizzard) (relman/hg->git project)]

Dana can you request uplift for other branches?

Comment 51

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Attachment: bug 1549249 - hard-code new add-on signing intermediate so it's always available (beta version) r=jcj,kmag

Our previous approach to making this intermediate available relied on being able
to add it to the user's NSS cert DB. This does work in the majority of cases,
but there are some situations where it doesn't work (e.g. if the user's DB is
set to read only, if they've configured Firefox to run in "nocertdb" mode, if
they have a master password but forgot it, and so on). This patch compiles the
intermediate in to Firefox in the same way we incorporate the root, so it should
always be available.

At the same time, this patch reverts the changes from
be8cd9575508ce1a95b971ccbfe3a7ceec59bc0b (the patch that implemented the
original approach) because it should no longer be necessary.

This also bumps the add-on DB schema to trigger add-on revalidation.

Comment 52

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Attachment: bug 1549249 - hard-code new add-on signing intermediate so it's always available (release version) r=jcj,kmag

Our previous approach to making this intermediate available relied on being able
to add it to the user's NSS cert DB. This does work in the majority of cases,
but there are some situations where it doesn't work (e.g. if the user's DB is
set to read only, if they've configured Firefox to run in "nocertdb" mode, if
they have a master password but forgot it, and so on). This patch compiles the
intermediate in to Firefox in the same way we incorporate the root, so it should
always be available.

At the same time, this patch reverts the changes from
848b15028562c6757748070f637e0e4f0bbb5f65 (the patch that implemented the
original approach) because it should no longer be necessary.

This also bumps the add-on DB schema to trigger add-on revalidation.

Comment 53

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Comment on attachment 9063045 [details]
bug 1549249 - hard-code new add-on signing intermediate so it's always available (esr60 version) r=jcj,kmag

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: more robust fix for cert2019 (aka "all add-ons got disabled")
• User impact if declined: Some users will still have all their add-ons disabled
• Fix Landed on Version: 68
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This is analogous to how we hard-code the add-on signing root, but with an intermediate - we just have to make it available to the system for it to pick it up.
• String or UUID changes made by this patch: none

Comment 54

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Comment on attachment 9063100 [details]
bug 1549249 - hard-code new add-on signing intermediate so it's always available (beta version) r=jcj,kmag

Beta/Release Uplift Approval Request

• User impact if declined: cert2019
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: see qa doc
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): see other uplift request
• String changes made/needed: none

Comment 55

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Comment on attachment 9063101 [details]
bug 1549249 - hard-code new add-on signing intermediate so it's always available (release version) r=jcj,kmag

Beta/Release Uplift Approval Request

• User impact if declined: cert2019
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: see qa doc
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): see other uplift request
• String changes made/needed: none

Comment 57

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Attachment: bug 1549249 - build bustage follow-up r?tomprince

SECItem uses an unsigned int to indicate its length. We need to cast a size_t
down to the appropriate size. This is safe because what we're casting will
always fit in an unsigned int on the platforms we're using (it's just the size
of the intermediate certificate we added).

Comment 58

[Liz Henry (:lizzard) (relman/hg->git project)]

(patch above is to address esr build failures at https://treeherder.mozilla.org/#/jobs?repo=mozilla-esr60&revision=37b8b055918e5b9cbbd2ebbfed4125047181d2a1&selectedJob=244994526)

Comment 59

[Ryan VanderMeulen [:RyanVM]]

This was pushed to m-c in comment 44 and comment 49.

Also pushed to Beta:
https://hg.mozilla.org/releases/mozilla-beta/rev/8f9fdaef9fc06cd0dda8edcb6e58aa37efbf2ddf (default)
https://hg.mozilla.org/releases/mozilla-beta/rev/fe6ddfaa325f8e9b57b134cd5ff48753726b8aff (FIREFOX_ESR_67b18_RELBRANCH)

And to Release:
https://hg.mozilla.org/releases/mozilla-release/rev/5b264ffa56e752df9a66c8a781e06fde51ada9e8

And to ESR60:
https://hg.mozilla.org/releases/mozilla-esr60/rev/5749f5b42cbf5a972bc8c398ed377977da35dbd2 (default)
https://hg.mozilla.org/releases/mozilla-esr60/rev/37b8b055918e5b9cbbd2ebbfed4125047181d2a1 (FIREFOX_ESR_60_6_X_RELBRANCH)
https://hg.mozilla.org/releases/mozilla-esr60/rev/24bb6566385fc566f1a6b98ea24cad7d0af7e3a3 (FIREFOX_ESR_60_6_X_RELBRANCH)

Comment 60

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/ab7db57ad2ac
build bustage follow-up r=mt,dveditz a=tomprince

Comment 61

[Emma Humphries ☕️🎸🧞‍♀️✨ (she/they) [:emceeaich] (Pacific Time) use needinfo]

Does this fix also resolve bug 1549624, and bug 1549627?

Comment 62

[Kris Maglione [:kmag]]

(In reply to Emma Humphries, Bugmaster ☕️🎸🧞‍♀️✨ (she/her) [:emceeaich] (UTC-8) needinfo? me from comment #61)

Does this fix also resolve bug 1549624, and bug 1549627?

Not really. Those only apply to the hotfix add-on, which their MitM proxies prevent the installation of. This only applies to the dot release fix, which users of those AVs should hopefully still get.

Comment 63

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/ab7db57ad2ac

Comment 64

[[:estemal]]

I have the same problem as ypetrov, but fixing permissions did not help me, the were OK. Also I have 2 addons that work OK now - HTTPS Everywhere and WebRTC Protect. All other addons are disabled and one has even been completely deleted. And I cannot install new addons.

1557244076860	addons.xpi	ERROR	failed to add new intermediate certificate:: [Exception... "Component returned failure code: 0x805a1fe8 [nsIX509CertDB.addCertFromBase64]"  nsresult: "0x805a1fe8 (<unknown>)"  location: "JS frame :: resource://gre/modules/addons/XPIProvider.jsm :: addMissingIntermediateCertificate :: line 1896"  data: no] Stack trace: addMissingIntermediateCertificate()@resource://gre/modules/addons/XPIProvider.jsm:1896
startup()@resource://gre/modules/addons/XPIProvider.jsm:2144
callProvider()@resource://gre/modules/AddonManager.jsm:203
_startProvider()@resource://gre/modules/AddonManager.jsm:652
startup()@resource://gre/modules/AddonManager.jsm:805
startup()@resource://gre/modules/AddonManager.jsm:2775
observe()@jar:file:///usr/lib64/firefox/omni.ja!/components/addonManager.js:66

Comment 65

[Bogdan Maris, Desktop QA]

QA has finished testing the fixes in all branches across multiple OSs, finding only bug 1549718 as new. Please see the testplan for more details: https://docs.google.com/document/d/17Td7VOlzlWoH-z7xTCrS8Yx4y2oS1YJn5uMqD7dt9bw/edit#

Comment 67

[Liz Henry (:lizzard) (relman/hg->git project)]

Adding to release notes for 66.0.5 (and 60.6.3esr) as, Fix for users who have master passwords set, to re-enable web extensions that had been disabled

If you have alternate wording let me know.

Comment 68

[Brian Grinstead [:bgrins]]

(In reply to Eternal Sorrow from comment #64)

I have the same problem as ypetrov, but fixing permissions did not help me, the were OK. Also I have 2 addons that work OK now - HTTPS Everywhere and WebRTC Protect. All other addons are disabled and one has even been completely deleted. And I cannot install new addons.

Could you try updating to 66.0.5? It's now released with a fix which should resolve that error. For any addons that got deleted or disabled, re-installing or re-enabling them should restore any addon-specific data.

Comment 69

[Kris Maglione [:kmag]]

(In reply to Brian Grinstead [:bgrins] from comment #68)

(In reply to Eternal Sorrow from comment #64)

I have the same problem as ypetrov, but fixing permissions did not help me, the were OK. Also I have 2 addons that work OK now - HTTPS Everywhere and WebRTC Protect. All other addons are disabled and one has even been completely deleted. And I cannot install new addons.

Could you try updating to 66.0.5? It's now released with a fix which should resolve that error. For any addons that got deleted or disabled, re-installing or re-enabling them should restore any addon-specific data.

Note that you'll continue seeing these errors until we unship the hotfix add-on, but they should no longer prevent add-ons from working.

Comment 70

[sjw]

bug 1549766 shows also the 0x805a1f65 error in console, but without a master password. AVG free antivirus is installed which has not been tested in Comment 37 (I can't access the document).
I'm asking the reporter to upgrade to 66.0.5.

Comment 71

[Alex J]

@kmag you're planning on unshipping the hotfix? What should I do if I'm on an OS that won't allow upgrade to 66.0.5?

Comment 72

[Kris Maglione [:kmag]]

(In reply to Alex J from comment #71)

@kmag you're planning on unshipping the hotfix?

Yes. It was only meant as a quick stopgap until we could ship a release.

What should I do if I'm on an OS that won't allow upgrade to 66.0.5?

See bug 1549604.

Comment 74

[Alex J]

Thanks for clearing that up @kmag!

Comment 75

[Firefox Bug Husbandry Bot]

Please specify a root cause for this bug. See :tmaity for more information.

Comment 76

[Mark Banner (:standard8)]

Dropping obsolete NI request.