1505038 - Passwords wiped out updating from 52.9.1 to version 60.3.0, possible workaround: comment #25, delete "pkcs11.txt" from profile

Status: RESOLVED DUPLICATE of bug 1510212

(Thunderbird :: Security, defect)

Description

[Norm Swaebe]

Attachment: Version 60.3.0.JPG

Notified of available update to version 60.3.0.
Allowed update to install, after installation my passwords and security information  was gone.

As I don't usually have a problem with Thunderbird updates I was going to run it with out first doing an email backup with Mozbackup, GLAD I DIDN'T!

Re-installed version 52.9.1 and restored Thunderbird from my backup.

Comment 1

[Wayne Mery (:wsmwk)]

Gone as in you couldn't find the password files?

Comment 3

[Wayne Mery (:wsmwk)]

> Wayne, any more reports like this one?

It's not an area I tend to follow in support fora.

Comment 4

[Jorg K (CEST = GMT+2)]

Sadly this seems to be a real problem. As a developer I have long stopped using TB 52.x, but if I create a new profile with that version, enter a password and then visit the profile with TB 60, I get prompted again. If I click cancel and visit the saved passwords, they are all gone. Returning back to TB 52, they are there again and I'm not prompted again.

I think I've noticed password prompts when switching between 52 and 60+ on test profiles for testing purposes, but I didn't pay much attention to it. I should have :-(

I think there were changes to the password storage, I just have to find the M-C person to ask ... which it turning out to be difficult. Let's try Marco (toolkit) and Dana (security).

Marco and Dana, what has changed in the way passwords were saved between 52 and 60? As you know, TB uses the toolkit password manager to store e-mail passwords. Was there a migration? It doesn't look like it works for us.

Comment 5

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Bug 1475775 was uplifted to ESR60 (60.2 I guess?) so potentially bug 1496736 could be what's going on if this is on a redhat-based system (and assuming thunderbird is based on esr?).

Comment 6

[Jorg K (CEST = GMT+2)]

Hmm, I'm on plain Windows 10 and I don't have a master password. Yes, TB 60.3 is is based on mozilla-esr60 at 60.3.

Comment 7

[Jorg K (CEST = GMT+2)]

Actually, I upgraded my wife's laptop from TB 52.7 to TB 60.3 without *any* problem and I didn't have to re-enter a single password. So there must be some special circumstances under which this happens.

Comment 8

[Norm Swaebe]

(In reply to Jorg K (GMT+2) from comment #7)
> Actually, I upgraded my wife's laptop from TB 52.7 to TB 60.3 without *any*
> problem and I didn't have to re-enter a single password. So there must be
> some special circumstances under which this happens.

Possible Windows ver? Just asking?

Comment 9

[Wayne Mery (:wsmwk)]

Can someone else reproduce this?  (copying some of our giant testers)

My key question is whether only 52>60.3.0 is affected?  Or is 60.2.1 also affected?
And as Jorg asks, under what circumstances?

Calling this dataloss until we know better. 
I think THIS issue is unlikely to be OS specific. (The original report is Windows.)  
However, there may be other issues. And cannot rule out OS specific. 
Casting a wide net for potential support postings - some of these sound similar to this bug report, others not so much:

https://support.mozilla.org/en-US/questions/1237460  10/17 linux, downgrade from 60.2.1 to 52.x and passwords work
https://support.mozilla.org/en-US/questions/1237698  10/20 linux also
https://support.mozilla.org/en-US/questions/1237790  10/21 Mac (unknown if user updated)
https://support.mozilla.org/en-US/questions/1238103  10/24 Mac (user had NOT updated to 60.x)
https://support.mozilla.org/en-US/questions/1237531  10/18 Win (constant issue for user, not just updates)
https://support.mozilla.org/en-US/questions/1237482  10/17 Win (unknown if user updated)
https://support.mozilla.org/en-US/questions/1237811  10/21 linux, user updated to 60

Comment 10

[Wayne Mery (:wsmwk)]

No recent bug reports that match this issue, at least not filed in Thunderbird product.  Here's a selection from https://mzl.la/2JKnohE (some are filed against TB52)
Bug 1503829 - impossible to send messages: password refused
Bug 1448769 - Thunderbird suddenly forgot mail server password, now not remembering passwords
Bug 1503829 - impossible to send messages: password refused

Also what I didn't state in the prior comment, is password issues are so prevalent that 
a) often the support posts don't all get looked out - so we can easily miss things
b) it's near impossible to distinguish non-bugs from bugs, and new bugs from old - so we can easily miss things

In short, we should get our act together to make the entire password area more robust, perhaps like was done with Firefox sessionstore.

Comment 11

[Jorg K (CEST = GMT+2)]

OK, here's some testing:
1) Created profile with TB 52, setup account, saved password.
   Checked saved passwords in PW manager. IMAP and SMTP there. All good.
2) Visited profile with TB 60.3. I did NOT get prompted to enter the password again,
   I could visit my IMAP folders, the PW manager showed the saved passwords. All good.

So a regular user just moving from TB 52 to TB 60 will NOT have any problem. That is my experience from upgrading two people (friend/family).

3) Went back to the profile with TB 52. All still good. Passwords visible in PW manager.
   IMAP working.
4) Went back with TB 60.3. All still working.

5) I went in with TB 64 beta. All still working. Then 52 and 60 again. No problem.

So on a new profile, I don't see any problem whatsoever.

I wasn't drunk when I wrote comment #4, but I didn't tell the full truth. The failing profile wasn't new, it had been created with TB 65 Daily.

If I create a profile with TB 64 beta (or TB 65 Daily), then the passwords are not accessible in TB 52. If I enter them again, then they become available in 52, 60 and 64.

Comment 12

[Marco Bonardo [:mak]]

I'm not the right person to tell you about passwords, maybe MattN?

Comment 13

[Onno Ekker [:nONoNonO UTC+1]]

I'm not sure when, but NSS changed from legacy NSS database files cert8.db and key3.db to newer SQLite database files cert9.db and key4.db files. Maybe that is causing problems in some cases? I recently answered a support question which might be related, although the user specified he was fiddling around on two computers: https://support.mozilla.org/en-US/questions/1238886?utm_campaign=questions-reply&utm_medium=email&utm_source=notification#answer-1168728

Comment 14

[Albert Scheiner [:alberts]]

Happened for me as well - tested on macOS / German

1) downloaded latest 52.x
2) added test account
3) saved password
4) checked that password is saved under Preferences > ...

meanwhile TB downloaded latest version 60.x

5) restarted
6) TB asked for PW
7) checked Preferences and PW was gone

Comment 15

[Norm Swaebe]

I am not a programmer so this may not help, since I reported this bug 1505038 and re-installed 52.9.1 I looked at the "troubleshooting report" and found I had a crash of version 60.3.0 521 seconds of "uptime" from install.
I am including the link here if it helps let me know or should I just stay back and watch!

LINK https://crash-stats.mozilla.com/report/index/b2d1bdf2-5912-4e43-9ec8-3fa910181106#tab-bugzilla

Norm

Comment 16

[Wayne Mery (:wsmwk)]

In https://support.mozilla.org/en-US/questions/1239638 the user shares the profile between OS

Comment 17

[Magnus Melin [:mkmelin]]

I tried 52.9.1 -> 60.3.0 upgrade on linux. Passwords were kept.

Comment 19

[:aceman]

Wayne, bug 1505630 seems to be the other way round (about downgrading from 60.3). Were the passwords created in 60.3 that got lost? As it appears to me the reports are that if you upgrade from 52 to 60 and then downgrade back, you get the passwords back (as the original file is used again).

Comment 20

[Jakobine Jónasdóttir]

(In reply to :aceman from comment #19)
> Wayne, bug 1505630 seems to be the other way round (about downgrading from
> 60.3). Were the passwords created in 60.3 that got lost? As it appears to me
> the reports are that if you upgrade from 52 to 60 and then downgrade back,
> you get the passwords back (as the original file is used again).

OP of bug 1505630 here. All my login data was created/saved in 52.9.1 or earlier. Upgraded to 60.3, which asked me to define a new master password (I didn't comply), then downgraded, then noticed the loss of my login data.

Comment 21

[:aceman]

In the past the passwords were stored in the file signons.sqlite in your profile. But I think that was long before TB 52. Now we use the file logins.json.
There should have been an automatic migration from one file to the other, keeping the passwords.

Can you see if you have those files and what their date of modification is?

Comment 22

[Jorg K (CEST = GMT+2)]

Hi Matt, according to Marco's comment #12 you might be the person to ask about stored passwords.

We have contradicting reports of problems when switching from TB 52 to TB 60 (and back). I had no problems with the former (comment #11), but others report problems (comment #14).

The bug was originally filed by someone who lost his passwords after the 52->60 upgrade, but there are other reports where this caused no problem (comment #7, comment #11, comment #17).

So the whole thing is a bit of a mystery :-(

Comment 23

[Matthew N. [:MattN]]

(In reply to Wayne Mery (:wsmwk) from comment #16)
> In https://support.mozilla.org/en-US/questions/1239638 the user shares the
> profile between OS

That sounds like bug 1453372.

(In reply to Jorg K (GMT+1) from comment #22)
> We have contradicting reports of problems when switching from TB 52 to TB 60
> (and back). I had no problems with the former (comment #11), but others
> report problems (comment #14).

It's expected that logins saved in a new profile created after bug 783994 (defaulting to key4.db) will not be able to be read in an older build expecting key3.db. i.e. if you have no key3.db, only key4.db, then logins can only be decrypted in TB60+. It didn't make sense to support downgrades on a profile that never previously used the old version.

> The bug was originally filed by someone who lost his passwords after the
> 52->60 upgrade, but there are other reports where this caused no problem
> (comment #7, comment #11, comment #17).

There have been no changes to password manager storage that I can think of. The only changes are related to the encryption that Dana mentioned in comment 5. You should read the two bugs mentioned there and look at the timing of bug 783994 to help solve this.

NFS and the environment variable NSS_DEFAULT_DB_TYPE could also be relevant to your investigation. Dana knows much more about the NSS/PSM side.

Comment 24

[Matt]

On a personal note, in the TB52 >60 migration I did almost daily for months,  I had issues with oauth passwords and caldav passwords being requested at odd times.  But standard POP/MAP/Chat accounts never needed to be re-entered.  I had assumed that the constantly changing user agent was causing providers to challenge the passwords, but perhaps it was a side effect of this bug.

Comment 25

[Matt]

https://support.mozilla.org/en-US/questions/1239779

This topic suggests passwords were missing and fixed by deleting "pkcs11.txt" from the profile or starting in safe mode.

Comment 26

[Norm Swaebe]

(In reply to Matt from comment #25)
> https://support.mozilla.org/en-US/questions/1239779
> 
> This topic suggests passwords were missing and fixed by deleting
> "pkcs11.txt" from the profile or starting in safe mode.

Trying to understand this completely. I have had a lot of headaches around the update from 52.9.1 to 60.3.0
Are you saying after the upgrade I should "delete pkcs.txt" from the profile and this will re-instate my passwords?

I have reinstalled 52.9.1 numerous times and finally deleted all reference to ThunderBird went to a clean install of 52.9.1 reinstalled my email settings from a Mozbackup file being careful to be disconnected from the internet as the default is to immediately load the 60.3.0 update!

Now running 52.9.1 email back up and running and "AUTO UPDATE" turned OFF!

Comment 27

[fettucini]

I guess, this Problem is much older; may I suggest having a look at my bug report:
https://bugzilla.mozilla.org/show_bug.cgi?id=1209803

Also, more recently, same Problem with 52.x -> 60.x

Comment 28

[fettucini]

@Matt: Thanks for pointing me into the right direction with "pkcs11.txt" - the problem seems to be resolved!

@mozdev: Changes between old and new file:

--- <unbenannt>
+++ <unbenannt>
@@ -1,8 +1,5 @@
-library=libnsssysinit.so
+library=
 name=NSS Internal PKCS #11 Module
-parameters=configdir='sql:/home/USER/.pki/nssdb' certPrefix='' keyPrefix='' secmod='secmod.db' flags= updatedir='' updateCertPrefix='' updateKeyPrefix='' updateid='' updateTokenDescription='' 
-NSS=Flags=moduleDBOnly,internal,critical trustOrder=75 cipherOrder=100 slotParams=(1={slotFlags=[RSA,DSA,DH,RC2,RC4,DES,RANDOM,SHA1,MD5,MD2,SSL,TLS,AES,Camellia,SEED,SHA256,SHA512] askpw=any timeout=30})
+parameters=configdir='sql:/USER/buddy/.thunderbird/0000000.default' certPrefix='' keyPrefix='' secmod='secmod.db' flags=optimizeSpace updatedir='' updateCertPrefix='' updateKeyPrefix='' updateid='' updateTokenDescription=''  manufacturerID='Mozilla.org' libraryDescription='PSM-interne Krypto-Dienste' cryptoTokenDescription='Allgemeine Krypto-Dienste' dbTokenDescription='das Software-Sicherheitsmodul' cryptoSlotDescription='PSM-interne Kryptographie-Dienste' dbSlotDescription='PSM private Schlüssel' FIPSSlotDescription='FIPS 140 Krypto-, Schlüssel- und Zertifikat-Dienste' FIPSTokenDescription='das Softw.-Sicherh.modul (FIPS)' minPS=0
+NSS=trustOrder=75 cipherOrder=100 slotParams={0x00000001=[slotFlags=RSA,RC4,RC2,DES,DH,SHA1,MD5,MD2,SSL,TLS,AES,SHA256,SHA512,Camellia,SEED,RANDOM askpw=any timeout=30 ] }  Flags=internal,critical
 
-library=/usr/lib/x86_64-linux-gnu/nss/libnssckbi.so
-name=Mozilla Root Certs
-NSS=trustOrder=100

Comment 32

[Wayne Mery (:wsmwk)]

Progress is being made on this issue in bug 1510212.  If you have concerns for yourself or for other users you should read the recent comments, and then comment there about any potential fixes or actions that might be taken.

Comment 35

[Marcus Burghardt]

A patch was created in BUG 1564284.
I believe this BUG has some relationship. Just updating... : )

Comment 36

[Kai Engert [:KaiE:]]

Marcus, have you seen that this bug is already marked as resolved, because it is a duplicate of bug 1510212 ?
Do you have any reason to believe it isn't a duplicate of bug 1510212 ?

Comment 37

[Marcus Burghardt]

Hi Kai.

This bug was in my list for some time. I realized it was resolved right after my comment. : )
Sorry and thanks!