1384620 - Cross-device passwords data loss

Status: RESOLVED DUPLICATE of bug 727574

(Firefox :: Sync, defect, P3)

Description

[Richard Newman [:rnewman]]

https://twitter.com/e_elkhoury/status/890233854367326208

The most plausible cause, I think, is:

- Somehow a desktop device decided to track all of its logins -- e.g., after a node reassignment.
- During upload, either data loss occurred (so every login appears to be deleted), or the login manager was unavailable in such a way that the sync proceeded but every record was apparently missing (and thus was uploaded as deleted).

Mark, any other diagnosis you think is likely?

Comment 1

[Mark Hammond [:markh] [:mhammond]]

(In reply to Richard Newman [:rnewman] from comment #0)
> https://twitter.com/e_elkhoury/status/890233854367326208
> 
> The most plausible cause, I think, is:
> 
> - Somehow a desktop device decided to track all of its logins -- e.g., after
> a node reassignment.
> - During upload, either data loss occurred (so every login appears to be
> deleted), or the login manager was unavailable in such a way that the sync
> proceeded but every record was apparently missing (and thus was uploaded as
> deleted).
> 
> Mark, any other diagnosis you think is likely?

The passwords manager doesn't hold tombstones - however, the engine will create them if it attempts to create a record with a GUID that can't be loaded from the manager - so as Richard says, this could happen if somehow these items were added to the tracker.

The most likely answer to this "the passwords were actually deleted using the passwords API" - this will send notifications and we will upload tombstones. The user obviously isn't aware how that happened, but it still might have somehow.

The least likely answer is a confluence of bad luck - eg, node reassignment added all the items to the tracker, but then, say, key3.db became corrupt (or master password had a hard-reset) meaning the items are unable to be decrypted. In that case, the password manager silently declines to return the passwords, so we'd upload tombstones for them all. This seems unlikely, but possible.

Bug 621846 and bug 1313985 are just 2 of the bugs we have on file related to a corrupt key3.db.

If the user installed the aboutsync addon, we could determine if there are tombstones on the server, but even if we knew that, it would be tricky to know exactly what happened. Getting his logins.json and key3.db might also help to rule in-or-out this theory. I'm not sure twitter is the right medium to walk the user through that process though - maybe we could invite him to this bug?

Comment 2

[Richard Newman [:rnewman]]

I pointed the user to the bug, so we'll see!

Comment 3

[Nazar Mokrynskyi]

Today between 16:15 and 16:30 (my local time) I've got `logins.json.corrupt` file instead of `logins.json` file in my profile (I know this, because I'm doing snapshots and backups of the whole system each 15 minutes). And then empty `logins.json` was created later.

Interestingly, I was signed out from sync. When I've signed in again and did repeated re-sync on my desktop and mobile device a few times, no passwords were synced back to Desktop. At the same time nothing was deleted on mobile - I can see all of the passwords there. On both devices passwords synchronization is enabled.

Thankfully I've restored working `logins.json` from snapshot created on 16:15 earlier today, but not everyone is doing this kind of backups so often.

There is some serious issue with passwords synchronization. I think this is related to upgrading from 56 Nightly to 57.

Comment 4

[Richard Newman [:rnewman]]

(In reply to Nazar Mokrynskyi from comment #3)
> When I've signed in again and did repeated re-sync on my desktop
> and mobile device a few times, no passwords were synced back to Desktop.

As far as I know, there is no linkage between login manager storage's corruption recovery and Sync. Sync doesn't know that all of your logins disappeared, so it doesn't know to download them again.

Is that right, Mark? If we blow away logins.json, Sync won't do anything special?

That's not related to this bug, though.

Nazar, what's the .corrupt file like? Is it truncated, totally corrupted, partially overwritten, or does it look OK?

Also not relevant to this bug, but it would be valuable to capture that knowledge.

Comment 5

[Nazar Mokrynskyi]

Truncated, 0 bytes.

Comment 6

[Mark Hammond [:markh] [:mhammond]]

(In reply to Richard Newman [:rnewman] from comment #4)
> (In reply to Nazar Mokrynskyi from comment #3)
> > When I've signed in again and did repeated re-sync on my desktop
> > and mobile device a few times, no passwords were synced back to Desktop.
> 
> As far as I know, there is no linkage between login manager storage's
> corruption recovery and Sync. Sync doesn't know that all of your logins
> disappeared, so it doesn't know to download them again.
> 
> Is that right, Mark? If we blow away logins.json, Sync won't do anything
> special?

Sadly yes, that's correct. Bug 1377100 tracks this. FWIW, if you use about:config to reset the preference services.sync.passwords.lastSync to zero, you should find it repopulates from the server.

I also can't see how a zero-byte logins.json would be created (which appears to be the root cause here).

I'm going to resolve this as a dupe of bug 727574 (which describes the original symptoms in this bug), and hopefully we will get to bug 1377100 some time soon.

Comment 7

[Richard Newman [:rnewman]]

> I also can't see how a zero-byte logins.json would be created

My wild speculation is a writeAtomic bug or a bad interaction with a filesystem.

One more reason why "just flush it to a JSON file!" is my least favorite kind of storage.