Password sync fails with "Services.logins is undefined"

NEW
Unassigned

Status

()

Firefox
Sync
P3
normal
a year ago
a month ago

People

(Reporter: mrstegeman, Unassigned)

Tracking

(Depends on: 1 bug, Blocks: 1 bug)

48 Branch
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [sync:passwords][sync:rigor])

Attachments

(5 attachments)

(Reporter)

Description

a year ago
Created attachment 8781100 [details]
error-sync-1471265149150.txt

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:48.0) Gecko/20100101 Firefox/48.0
Build ID: 20160726073904

Steps to reproduce:

Clean install of Firefox 48 (64-bit) on a clean install of Windows 10. Entered credentials for Firefox sync.


Actual results:

Everything syncs except for my passwords/logins. This actually started happening on Firefox 47 on Windows 7, same machine. I wiped everything and started fresh, but the same problem occurred.

At the same time my sync issues started, I started having TLS issues on my corporate network. The certificates are never recognized as valid. I can still get to the page if I click through the exception dialogs, but it also will not let me add a permanent exception. If that box is checked, I cannot click the "Confirm Security Exception" button.


Expected results:

Sync should have finished successfully.
(Reporter)

Comment 1

a year ago
The actual error in the log I attached is here:

1471265148820	Sync.ErrorHandler	DEBUG	passwords failed: TypeError: Services.logins is undefined (resource://gre/modules/services-sync/engines/passwords.js:164:9) JS Stack trace: PasswordStore.prototype.getAllIDs@passwords.js:164:9 < SyncEngine.prototype._syncStartup@engines.js:931:22 < SyncEngine.prototype._sync@engines.js:1553:7 < WrappedNotify@util.js:146:21 < Engine.prototype.sync@engines.js:670:5 < _syncEngine@enginesync.js:213:7 < sync@enginesync.js:163:15 < onNotify@service.js:1262:7 < WrappedNotify@util.js:146:21 < WrappedLock@util.js:101:16 < _lockedSync@service.js:1252:12 < sync/<@service.js:1244:14 < WrappedCatch@util.js:75:16 < sync@service.js:1232:5

Updated

a year ago
Component: Untriaged → Sync
Services.logins being undefined is going to cause a number of problems. I suspect an addon might be interfering - can you please try to reproduce in safe mode? See https://support.mozilla.org/en-US/kb/troubleshoot-firefox-issues-using-safe-mode for details.
Flags: needinfo?(mrstegeman)
(Reporter)

Comment 3

a year ago
Created attachment 8781536 [details]
error-sync-1471350890614.txt

Same issue in safe mode. I attached a new error log for good measure.
Flags: needinfo?(mrstegeman)
Priority: -- → P3

Updated

11 months ago
Summary: Password sync fails with error → Password sync fails with "Services.logins is undefined"

Updated

11 months ago
Duplicate of this bug: 1309333

Comment 5

11 months ago
Matt, the logs from these syncs report "TypeError: Services.logins is undefined" and since we landed Sync telemetry we are seeing many hundreds of failures with this same issue. Are you aware of any issues which might cause this (eg, invalid logins.json or similar) or any way for these users to diagnose the issue which doesn't involve them disclosing all their passwords?
Status: UNCONFIRMED → NEW
Ever confirmed: true
Flags: needinfo?(MattN+bmo)
Please enable password manager debug logging and attach the output from the Browser Console after a restart followed by a manual sync (which I assume triggers this error).

https://wiki.mozilla.org/Firefox:Password_Manager_Debugging
Flags: needinfo?(MattN+bmo) → needinfo?(mrstegeman)
(Reporter)

Comment 7

11 months ago
Created attachment 8800212 [details]
password manager debug output

I've attached the browser console log with a couple failed syncs, both before and after I was asked to reconnect to the sync server with my credentials (which also happens often).
Flags: needinfo?(mrstegeman)

Comment 8

10 months ago
Thanks for the log. The relevant lines seem to be:

> : Component returned failure code: 0x805a1f9b [nsIPK11Token.initPassword]crypto-SDR.js:66
> Login crypto:Initializing key3.db with default blank password.crypto-SDR.js:65
> Error: Initialization failedstorage-json.js:90:13

To try and reproduce this I truncated key3.db at some random size and started the browser. On a recent Nightly I got the same messages as the log, but Services.logins still existed. However, trying the same thing on the release channel did make more noise, including an error regarding Services.logins being undefined.

I had a quick look at the hg logs but couldn't find an obvious candidate for us handling such an error. Matt, is there some other bug you are aware of that might have fixed this, or any other clues as to what is going on?
Flags: needinfo?(MattN+bmo)

Comment 9

10 months ago
Oops - I spoke too soon - while I didn't get that error on startup using Nightly, I did get it a little later, so I suspect it hasn't actually been fixed, just that the code has changed so the symptoms are slightly different. Also, bug 1124553 is another example of key3.db being corrupt causing problems, and trying to save my Sync credentials when in this state also causes an exception with "Services.logins in undefined".

Updated

10 months ago
Duplicate of this bug: 1276166
Depends on: 1313985
I don't have other ideas as it sounds like an issue with key3.db. I think we should pursue your suggestion I quoted in bug 1124553 comment 13.
Flags: needinfo?(MattN+bmo)
(Reporter)

Comment 12

9 months ago
Issue is still present in Firefox 50.

Updated

9 months ago
Duplicate of this bug: 1318310
I can reproduce consistently after using Refresh Firefox on my Ubuntu VM. Unfortunately, I can't repro on macOS; haven't tried Windows yet.

The issue is `sftk_getKeyDB` (http://searchfox.org/mozilla-central/rev/8562d3859b89ac89f46690b9ed2c473e0728d6c0/security/nss/lib/softoken/sftkdb.c#2404-2416) returns NULL. This causes `NSC_InitPIN` to fail (http://searchfox.org/mozilla-central/rev/8562d3859b89ac89f46690b9ed2c473e0728d6c0/security/nss/lib/softoken/pkcs11.c#3651-3655), and the error propagates through `PK11_InitPin` (http://searchfox.org/mozilla-central/rev/8562d3859b89ac89f46690b9ed2c473e0728d6c0/security/nss/lib/pk11wrap/pk11auth.c#456-461), `nsPK11Token::InitPassword` (http://searchfox.org/mozilla-central/rev/8562d3859b89ac89f46690b9ed2c473e0728d6c0/security/manager/ssl/nsPK11TokenDB.cpp#299-300), and `LoginManagerCrypto_SDR#init` (http://searchfox.org/mozilla-central/rev/8562d3859b89ac89f46690b9ed2c473e0728d6c0/toolkit/components/passwordmgr/crypto-SDR.js#66).

That, in turn, breaks login manager for the remainder of the session. Interestingly, this only happens in the session immediately after refreshing. If I refresh Firefox, then manually restart, the issue goes away; login manager can initialize successfully.

FWIW, Firefox Refresh copies over key3.db from the old profile (http://searchfox.org/mozilla-central/rev/8562d3859b89ac89f46690b9ed2c473e0728d6c0/browser/components/migration/FirefoxProfileMigrator.js#133). Also, the VM has very slow disk I/O, while the Mac has a speedy SSD. I wonder if, in this case, there's a race where NSS initializes before we copy the DB over, and that causes havoc with some in-memory state. But I'm way out of my depth here. :-)

David, do you have insights into this?
Flags: needinfo?(dkeeler)
(In reply to Kit Cambridge [:kitcambridge] from comment #14)
...
> I wonder if, in this case, there's a race where
> NSS initializes before we copy the DB over, and that causes havoc with some
> in-memory state.

That could certainly cause this. Something that might give some insight is running with MOZ_LOG=pipnss:4 and MOZ_LOG_FILE=logfile.txt or similar. I'm betting it fails to initialize NSS in the profile directory the first run (at that point the implementation falls back to memory-only mode, which causes all password-saving machinery to not work).
Flags: needinfo?(dkeeler)
See Also: → bug 1124553

Updated

7 months ago
Blocks: 1337275
Created attachment 8838399 [details] [diff] [review]
delete-key3.patch

I've been playing a little with this, and as with everything NSS related, "it's complicated".

The level of corruption in key3.db causes 2 main symptoms that I've seen:

* As captured in bug 1124553 and bug 621846 - the DB initializes, but all attempts to decrypt anything fails in ways that look alot like the master password has been reset.

* Failure to initialize the DB:

> : Component returned failure code: 0x805a1f9b [nsIPK11Token.initPassword]crypto-SDR.js:66
> Login crypto:Initializing key3.db with default blank password.crypto-SDR.js:65
> Error: Initialization failedstorage-json.js:90:13

Let's just look at the second here, as that is more clearly "massive failure".

A quick play shows that if we catch the above exception we can delete key3.db, but I can't work out how to re-initialize the library. So while that doesn't help the current browser session, it *does* get handled next time the browser starts - obviously all existing logins are now unavailable, but the login manager (noisily) handles this OK - it looks just like a master-password reset, but new logins can be saved.

This patch is quite naive and doesn't have tests (and I'm not sure if testing this sanely would be possible), but is a patch like this worth persevering with, or should we just wait for bug 1271851 (and/or Godot)?
Attachment #8838399 - Flags: feedback?(dkeeler)
Attachment #8838399 - Flags: feedback?(MattN+bmo)
Comment on attachment 8838399 [details] [diff] [review]
delete-key3.patch

Review of attachment 8838399 [details] [diff] [review]:
-----------------------------------------------------------------

My input would be that maybe deleting key3.db isn't the most user-friendly approach - maybe it should be renamed (e.g. to "key3.db.backup.<some random hex>" or similar) instead?
Also, on android the file would be key4.db (not sure if this is happening on android).

(In reply to Mark Hammond [:markh] from comment #16)
...
> A quick play shows that if we catch the above exception we can delete
> key3.db, but I can't work out how to re-initialize the library.

Bad news/good news/bad news: you basically can't safely re-initialize NSS with the way the code is set up now. However, we're currently working on a project that could enable us to do exactly this (unload and reload the user's certificates/keys/etc.). See https://wiki.mozilla.org/Security/CryptoEngineering/Platform_Use_of_NSS . Unfortunately, there are some unanswered questions and significant code re-working that means we're still a ways out from implementing this, so it doesn't really help you right now.

Here's another thought, though - does calling token.reset() and then token.initPassword("") fix it? (again, this would clobber the user's password encryption key as well as any private keys (which they hopefully have backed up elsewhere, but still).
Attachment #8838399 - Flags: feedback?(dkeeler)
(In reply to David Keeler [:keeler] (use needinfo?) from comment #17)
> My input would be that maybe deleting key3.db isn't the most user-friendly
> approach - maybe it should be renamed (e.g. to "key3.db.backup.<some random
> hex>" or similar) instead?

Yeah, renaming would be better.

> Also, on android the file would be key4.db (not sure if this is happening on
> android).

I've not seen reports of it there, but I guess there's no reason it couldn't also happen there too, so wrapping that code in an AppConstants check also probably makes sense.

> Here's another thought, though - does calling token.reset() and then
> token.initPassword("") fix it? (again, this would clobber the user's
> password encryption key as well as any private keys (which they hopefully
> have backed up elsewhere, but still).

No, that's the first thing I tried and the second initPassword fails in the same way :(

I'm happy to update the patch to rename the file, but I'm still not clear if it is something we should go ahead with in general. I guess I'll wait and see what Matt says...

Updated

6 months ago
Duplicate of this bug: 1338410
Comment on attachment 8838399 [details] [diff] [review]
delete-key3.patch

Review of attachment 8838399 [details] [diff] [review]:
-----------------------------------------------------------------

Hi Mark,

I'm fine with this approach if you can show me data that the issue this is trying to solve isn't a transient issue i.e. that deleting the file isn't worse for the user than leaving it there in cases where the issue is some kind of race or file-system issue.

For example, if you have telemetry from sync ping, can you show data that if a user gets this issue that they continue to get this error every time until it gets fixed?

::: toolkit/components/passwordmgr/crypto-SDR.js
@@ +67,5 @@
> +        this.log("Failed to initialize key3.db with a new password - clobbering it for next time!", ex);
> +        // use main-thread IO here as converting this to handle async is alot
> +        // of work and we should only end up here in the rarest of cases.
> +        let path = OS.Path.join(OS.Constants.Path.profileDir, "key3.db")
> +        let file = Cc['@mozilla.org/file/local;1'].createInstance(Ci.nsILocalFile);

Nit: double quotes

@@ +69,5 @@
> +        // of work and we should only end up here in the rarest of cases.
> +        let path = OS.Path.join(OS.Constants.Path.profileDir, "key3.db")
> +        let file = Cc['@mozilla.org/file/local;1'].createInstance(Ci.nsILocalFile);
> +        file.initWithPath(path);
> +        file.remove(false);

I agree that renaming with a unique name would be friendlier.
Attachment #8838399 - Flags: feedback?(MattN+bmo) → feedback+

Comment 21

6 months ago
One more observation here. The profile is not creating key3.db and logins.json files. So there is no point in taking a backup of those files.
(Reporter)

Comment 22

6 months ago
Created attachment 8844066 [details]
Browser console errors with fresh profile

After upgrading to Firefox 52, I set up a clean profile, with no data, no add-ons, didn't even put in my sync info. With this clean slate, there are already a ton of errors in the browser console. I can reliably reproduce this.
(In reply to mrstegeman from comment #22)
> After upgrading to Firefox 52, I set up a clean profile, with no data, no
> add-ons, didn't even put in my sync info. With this clean slate, there are
> already a ton of errors in the browser console. I can reliably reproduce
> this.

That's really odd - what platform are you on? This issue probably doesn't belong in this bug, but there may already be one on file - Matt, have you ever seen a clean profile fail to init the login manager?
Flags: needinfo?(MattN+bmo)
(Reporter)

Comment 24

6 months ago
Windows 10, 64-bit. It happened on Windows 7 for me as well, before I upgraded my machine.

This bug is originally mine, so my issue has probably always been the same, but it does appear that lots of others have similar symptoms, at least.
(Reporter)

Comment 25

4 months ago
My specific issue, as referenced above, has been resolved with an updated WebSense (Triton AP-Endpoint DLP) version, and Firefox Developer Edition 54.0a2 (2017-04-18).
(In reply to Mark Hammond [:markh] from comment #23)
> Matt, have you ever seen a clean profile fail to init the login manager?

No, I don't think so.
Flags: needinfo?(MattN+bmo)

Comment 27

3 months ago
i have this symptom as well when the plugin "GNOME keyring password integration" is enabled. and only then - without i have no error messages on startup.

when the plugin is enabled i can't store any passwords anymore (and previously stored passwords do not show up).

i use 45.9.0 on linux. i can't remember the exact version that worked (and stored PWDs in the keyring).
(In reply to github from comment #27)
> i have this symptom as well when the plugin "GNOME keyring password
> integration" is enabled. and only then - without i have no error messages on
> startup.
> 
> when the plugin is enabled i can't store any passwords anymore (and
> previously stored passwords do not show up).

Bug 1254321 tracks one issue with the gnome keyring, but this problem sounds more serious.

> i use 45.9.0 on linux. i can't remember the exact version that worked (and
> stored PWDs in the keyring).

Is there anything reported in the browser console? Regardless, you should report this to the addon author - in this bug we are trying to track this symptom in Firefox without any extensions installed or involved.

Updated

a month ago
Whiteboard: [sync:passwords][sync:rigor]
See Also: → bug 1255992
You need to log in before you can comment on or make changes to this bug.