1835786 - Unexpected inconsistency between files key4.db and encrypted-openpgp-passphrase.txt, OpenPGP secret keys no longer accessible (files with ending .corrupt seen in profile)

Status: RESOLVED FIXED

(MailNews Core :: Security: OpenPGP, defect, P1)

Description

[nani]

Attachment: Screenshot 2023-05-30 at 18.22.45.png

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/113.0

Steps to reproduce:

tried to open old encrypted emails

Actual results:

no longer able to see, error says that there is no digital signature but i could see these mails fine before, keys for sender and receivers all in my keychain. upon seeing this error also noticed i am no longer able to send encrypted mail even though my key pair reads in the key manager and has been verified.

Expected results:

read old encrypted mails, also be able to send encrypted mails

Comment 1

[Magnus Melin [:mkmelin]]

You'll need to provide much more details.
For receiving, check which keys the mail is encrypted to - and verify they key id is the one you have.
For sending - well what happens?

Comment 2

[nani]

thank you magnus for replying so quickly.

for previous mails in inbox no longer readable:

• get alert: "The secret key that is required to decrypt this message is not available" as well as when clicking the OpenPGP icon: "This message does not include the sender's digital signature" (the latter strange to me because i could read these mails all fine previously)
• when i open the Open PGP key manager: i have the keys for myself and all senders; noticed however that the sender's key was ticked "Yes, but i have not verified that it is the correct key", so i have now ticked "Yes, I've verified in person this key has the correct fingerprint" in case that was the issue (no change)
• for my own key: ticked the box "Yes, treat this key as a personal key"

for sending mail (tried to send a test to myself):

• simply get the error: "Sending of the message failed"

to note:

• i noticed there is a secring.gpg.corrupt file in my Profiles folder which was not there in backup versions of my harddrive, so guessing something must have blipped at some point and things stopped working? i actually replaced old secring.gpg and pubring.gpg files in the current Thunderbird in order to find my private key again, because when i first noticed this was happening, the key manager couldn't even find my key pair.
• i am a newbie to encrypted mail so not really sure how to find my keys properly again. sorry and thank you for your patience!

Comment 3

[Magnus Melin [:mkmelin]]

If I were to guess, sounds like there was corruption of the files, and you now somehow have just part of the key working. If you have backups, check an earlier back of the gpg files you mention. (Or real backup of they key, they way Thunderbird allows you to create key backup, if you had one.)

Comment 4

[nani]

actually i did that already, for the secring.gpg file and pubring.gpg file. the difference is that now i don't get the error that my key wasn't matched (in the key manager), but it's still the same problem with being unable to read the encrypted mails... are there other files i should also replace, or delete the corrupt files, or what do you think?

Comment 5

[Magnus Melin [:mkmelin]]

Can you try importing the keys into a new profile and read the message there?

Comment 6

[Kai Engert [:KaiE:]]

What you see really shouldn't have happened. I hope we can find out what caused this inconsistency. Broken hardware could explain it.

In past versions of Thunderbird (earlier than 102.3), we had a bug, which could cause an inconsistency, and later versions of Thunderbird attempt to automatically repair that inconsistency, when it is seen.

The inconsistency was triggered when using the "Tools / Import" functionality to copy over files from another profile into your current profile. But using versions 102.3 and later should no longer cause that problem.

If you want to try another repair attempt:

Ensure that Thunderbird is fully stopped when trying to restore files in the profile directory.

The following files have interdependencies:

• key4.db
• cert9.db
• encrypted-openpgp-passphrase.txt
• logins.json
• openpgp.sqlite
• pubring.gpg
• secring.gpg

If you have one, restore all of those files from a known-good backup.

Then start Thunderbird, and check if it works again.

Comment 7

[Kai Engert [:KaiE:]]

nani, did you use the profile import feature before the corruption occurred?

Comment 8

[Kai Engert [:KaiE:]]

I looked at the code, and I see we still have a scenario in which the old, broken code would be used (which is known to create the corruption).
That old code would be reached in a non-default configuration scenario, only, if setting mail.import.in_new_tab to false.

I just say this to try to understand what might have caused the corruption.
If you've never touched the setting mail.import.in_new_tab and it's set to true, I think it cannot have been the cause of your corruption.

Comment 9

[Kai Engert [:KaiE:]]

Low level background explanation:

key4.db contains the profile's primary symmetric key, which is used to encrypt other secrets.
(If a primary password is set, this symmetric key is encrypted with it. If no primary password is set, the symmetric key is available directly.)

File encrypted-openpgp-passphrase.txt contains a passphrase, it's encrypted (protected) with the symmetric key from file key4.db

The secret keys in file secring.gpg are encrypted (protected) using the passphrase from file encrypted-openpgp-passphrase.txt

Because you have files ending with .corrupt, which were automatically created by Thunderbird, it means the following had happened on your system:

• Thunderbird started
• Thunderbird tried to use the symmetric key from key4.db
  to decrypt the data in file encrypted-openpgp-passphrase.txt
  but that attempt failed.
• Thunderbird concluded we have profile corruption.
• In order to keep your settings in a usable state,
  Thunderbird moved the corrupt/inconsistent files to separate files with ending .corrupt

Comment 10

[Kai Engert [:KaiE:]]

I understand that Jan (CC'ed) had also experienced the same situation, and he told me, that he could still access all the saved passwords.

This means that key4.db wasn't corrupted. (Because the contents of file logins.json, having the saved passwords, can still be decrypted with the data from key4.db)

This means that it was file encrypted-openpgp-passphrase.txt that was corrupted.

I'd like to understand what went wrong with that file. Could you try to carefully look at the contents of the corrupted file encrypted-openpgp-passphrase.txt.corrupt ? Does it have similarly looking contents than the newer working file? (They are expected to be different, but they should look similar, e.g. both contain one long line of text with random characters, and approximately the same size). or does it look completely different? Or is it empty?

I'm wondering if something has modified that file. Maybe it was accidentally opened in a text editor the contents were changed?

Or could it be a file permission problem?

Comment 11

[nani]

hallo :KaiE:, thank you so much for all the advice!

luckily i could find the back-up from before the corrupt files appeared, and I tried replacing all the interdependent files you mentioned above, and it worked!

just to answer other questions in case it might help others:

• i didn't use the import feature at all before or after the corrupt files appeared, so not sure at all what might have happened?
• the files that i noticed had corrup versions were: encrypted-openpgp-passphrase.txt.corrupt, secring.gpg.corrupt, and secring.gpg.old.corrupt
• regarding the comparison between encrypted-openpgp-passphrase.txt and encrypted-openpgp-passphrase.txt.corrupt, yes they look similar, with the first half of the string being the same, but different later
• i did not modify the above files at all myself, nor were they accidentally opened in text editor

thank you again~

Comment 12

[Kai Engert [:KaiE:]]

(In reply to nani from comment #11)

luckily i could find the back-up from before the corrupt files appeared, and I tried replacing all the interdependent files you mentioned above, and it worked!

I'm very glad to hear that.

• i didn't use the import feature at all before or after the corrupt files appeared,

ok good to know

so not sure at all what might have happened?

That's the mistery, I don't have an answer either.

• regarding the comparison between encrypted-openpgp-passphrase.txt and encrypted-openpgp-passphrase.txt.corrupt, yes they look similar, with the first half of the string being the same, but different later

Then it's likely that this file got corrupted. It rather means key4.db had unexepctedly changed on your system, and no longer contained the key to decrypt this string.

Comment 13

[Kai Engert [:KaiE:]]

It would be good to turn this bug into a FAQ - if this scenario happens, try to repair by restoring all files mentioned in comment 6 from a backup.

Comment 14

[Jan Martin Keil]

I checked the version history of the relevant files in my weekly backup of the profile folder. I found the below file versions with date of backup creation (not file creation). I also compared the *.corrupted file version with the previous not corrupted version (using git diff --no-index --word-diff). Except of the *.old*.corrupt files, they had equal content. It looks like that valid files have falsely been flagged as corrupted. As the *.corrupted files exist, I would exclude a permission issue on file system level. Without knowing the code, I guess that the reason is a internal permission issue (maybe a timing/concurrency problem?) or a bug in the corruption detection.

2020-12-30 key4.db
2020-12-30 encrypted-openpgp-passphrase.txt
2020-12-30 secring.gpg
2021-05-01 secring.gpg
2021-05-05 secring.gpg.old
2021-05-22 secring.gpg
2021-05-22 secring.gpg.old
2022-12-25 encrypted-openpgp-passphrase.txt
2022-12-25 encrypted-openpgp-passphrase.txt.corrupt (= 2020-12-30 encrypted-openpgp-passphrase.txt)
2022-12-25 secring.gpg
2022-12-25 secring.gpg.corrupt (= 2021-05-22 secring.gpg)
2022-12-25 secring.gpg.old.corrupt
2023-04-08 secring.gpg
2023-04-21 encrypted-openpgp-passphrase.txt
2023-04-21 encrypted-openpgp-passphrase.txt-1.corrupt (= 2022-12-25 encrypted-openpgp-passphrase.txt)
2023-04-21 secring.gpg
2023-04-21 secring.gpg-1.corrupt (= 2023-04-08 secring.gpg)
2023-04-21 secring.gpg.old-1.corrupt
2023-05-13 secring.gpg
2023-05-13 secring.gpg.old
2023-05-22 secring.gpg.old

Comment 15

[Kai Engert [:KaiE:]]

If your theory was correct that the files are actually still good, then you could try to:

• quit thunderbird
• make a backup, at least of all the files I mentioned above
• remove the newer files encrypted-openpgp-passphrase.txt and secring.gpg
• rename encrypted-openpgp-passphrase.txt.corrupt to encrypted-openpgp-passphrase.txt
• rename secring.gpg.corrupt to secring.gpg
• start thunderbird

If those files are really good, then thunderbird should accept them (and not move them to .corrupt again)

Comment 16

[Jan Martin Keil]

Using the procedure you described, in both cases (encrypted-openpgp-passphrase.txt.corrupt + secring.gpg.corrupt and encrypted-openpgp-passphrase.txt-1.corrupt + secring.gpg-1.corrupt) Thunderbird accepted the files and I was able to decrypt mails. This confirms, that valid files have falsely been flagged as corrupted.

That's good news for people without backup and should ease writing an FAQ.

I hope, this also helps to find the bug and fix.

@nani: Does this work for you, too?

Comment 17

[Kai Engert [:KaiE:]]

That's very very helpful to know. Thanks a lot. I'll try to investigate how this could happen.

Comment 18

[Kai Engert [:KaiE:]]

Jan, do you have a primary password set?

Comment 19

[Jan Martin Keil]

(In reply to Kai Engert (:KaiE:) from comment #18)

Jan, do you have a primary password set?

Yes.

Comment 20

[Kai Engert [:KaiE:]]

I have identified the following plausible scenario, in which the code incorrectly marks the files as corrupt:

The application has received a command to exit, while an OpenPGP operation is still ongoing in parallel, and that OpenPGP operation is either the first OpenPGP operation in this session (no primary password is set), or (primary password is set) it's the first OpenPGP operation after the user has unlocked the primary password.

In this scenario, while the application is already shutting down, the code's attempt to obtain the global "secret decoder ring" service ("@mozilla.org/security/sdr;1") might fail.

This failure would be incorrectly treated as a failure to decrypt file encrypted-openpgp-passphrase.txt

There are additional scenarios which could potentially cause the files to be marked as corrupt, but those scenarios are less likely (if the files are indeed consistent and contain correct data):

• file "encrypted-openpgp-passphrase.txt" exists and contains correct data, buts IOUtils.readUTF8 fails with an exception
• IOUtils.readUTF8 returned the contents of the file (no exception), but nsISecretDecoderRing.decryptString fails,
  while either no primary password is set, or a primary password is set and the user has unlocked it already.

The fix is to check for this error scenarios separately.
Only the failure to decrypt should result in the repairing. Other failures should be treated differently

Comment 21

[Kai Engert [:KaiE:]]

Attachment: Bug 1835786 - Only the failure to decrypt the stored default OpenPGP passphrase should result in repairing the OpenPGP secret key storage. r=BenC

Comment 22

[Jan Martin Keil]

(In reply to Kai Engert (:KaiE:) from comment #20)

I have identified the following plausible scenario, in which the code incorrectly marks the files as corrupt:

The application has received a command to exit, while an OpenPGP operation is still ongoing in parallel, and that OpenPGP operation is either the first OpenPGP operation in this session (no primary password is set), or (primary password is set) it's the first OpenPGP operation after the user has unlocked the primary password.

In this scenario, while the application is already shutting down, the code's attempt to obtain the global "secret decoder ring" service ("@mozilla.org/security/sdr;1") might fail.

This failure would be incorrectly treated as a failure to decrypt file encrypted-openpgp-passphrase.txt

There are additional scenarios which could potentially cause the files to be marked as corrupt, but those scenarios are less likely (if the files are indeed consistent and contain correct data):

• file "encrypted-openpgp-passphrase.txt" exists and contains correct data, buts IOUtils.readUTF8 fails with an exception
• IOUtils.readUTF8 returned the contents of the file (no exception), but nsISecretDecoderRing.decryptString fails,
  while either no primary password is set, or a primary password is set and the user has unlocked it already.

The fix is to check for this error scenarios separately.
Only the failure to decrypt should result in the repairing. Other failures should be treated differently

That sound right. Chances are good that I once (or twice) killed Thunderbird after starting it by accident, as that is faster than entering the primary password and close it afterwards. (Unfortunately, Thunderbird can not be closed during the primary password dialog (bug?) and canceling primary password dialog only triggers the next primary password dialog many times, [I guess, depending on the number of services {several mailboxes, CardDavs, CalDavs and PGP keys} that require them].)

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/f800819b0a55
Only the failure to decrypt the stored default OpenPGP passphrase should result in repairing the OpenPGP secret key storage. r=BenC

Comment 24

[Wayne Mery (:wsmwk)]

Comment on attachment 9343108 [details]
Bug 1835786 - Only the failure to decrypt the stored default OpenPGP passphrase should result in repairing the OpenPGP secret key storage. r=BenC

[Triage Comment]
Approved for beta

Comment 25

[Daniel Darnell [:dandarnell]]

Thunderbird 116.0b4:
https://hg.mozilla.org/releases/comm-beta/rev/61136747dc4c

Comment 26

[Kai Engert [:KaiE:]]

Comment on attachment 9343108 [details]
Bug 1835786 - Only the failure to decrypt the stored default OpenPGP passphrase should result in repairing the OpenPGP secret key storage. r=BenC

I think it's time to uplift to 115 and 102.

Comment 27

[Wayne Mery (:wsmwk)]

Comment on attachment 9343108 [details]
Bug 1835786 - Only the failure to decrypt the stored default OpenPGP passphrase should result in repairing the OpenPGP secret key storage. r=BenC

[Triage Comment]
Approved for esr115

Comment 28

[Rob Lemley [:rjl]]

Thunderbird 115.1.1:
https://hg.mozilla.org/releases/comm-esr115/rev/552330c6683c

Comment 29

[Wayne Mery (:wsmwk)]

Comment on attachment 9343108 [details]
Bug 1835786 - Only the failure to decrypt the stored default OpenPGP passphrase should result in repairing the OpenPGP secret key storage. r=BenC

[Triage Comment]
Approved for esr102

Comment 30

[Rob Lemley [:rjl]]

Thunderbird 102.15.0:
https://hg.mozilla.org/releases/comm-esr102/rev/ac0b189e3bc0