Open Bug 1857477 Opened 1 year ago Updated 3 months ago

No longer able to digitally sign email with OpenPGP in TB 115, if only the public key is still valid, but the private key has expired.

Categories

(MailNews Core :: Security: OpenPGP, defect)

Product:
MailNews Core
Component:
Security: OpenPGP
Version:
Thunderbird 115
Type:
defect

Tracking

(thunderbird_esr115 affected)

Tracking Flags:
Tracking Status
thunderbird_esr115 --- affected

People

(Reporter: david.hebbeker, Unassigned)

References

Details

(Keywords: regression, Whiteboard: [RNP])

Attachments

(6 files, 2 obsolete files)

Troubleshooting Information generated from Thunderbird
79.29 KB, text/html
Details
public part of the key pair using for signing
7.87 KB, application/octet-stream
Details
Output of error console using Thunderbird v115.3.2 Build ID 20231010094303
6.14 KB, text/plain
Details
console-export-2023-10-16_15-26-48.txt
6.40 KB, text/plain
Details
console-export-2023-10-16_16-3-20.txt
7.77 KB, text/plain
Details
console output for using expired and extended private key executed with build ID 20231016092201
3.08 KB, text/plain
Details

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/118.0

Steps to reproduce:

Compose an e-mail with recipient, subject and message body. Digitally signing (not encryption) using OpenPGP is selected (by default). Attempt to send email (using button or CTRL+ENTER).

I attempted this with Troubleshoot mode on and off. It did not make a difference.

Actual results:

A dialog warns, that sending of the message failed.

The dialog is in German:

Senden der Nachricht fehlgeschlagen.

The message is not sent.

From the error console:

window.controllers/Controllers sollte nicht mehr verwendet werden. Verwenden Sie es nicht für die Browser-Erkennung. 2 blank
CryptoAPI.sync() failed result:  Error: rnp_op_sign_add_signature failed
    encryptAndOrSign chrome://openpgp/content/modules/RNP.jsm:3535
    sync chrome://openpgp/content/modules/cryptoAPI/interface.js:56
    encryptMessageStart chrome://openpgp/content/modules/encryption.jsm:446
    finishCryptoEncapsulation chrome://openpgp/content/modules/mimeEncrypt.jsm:516
    createMessageFile resource:///modules/MimeMessage.jsm:90
interface.js:46:17
Error: failure in finishCryptoEncapsulation, exitCode: -1
    finishCryptoEncapsulation chrome://openpgp/content/modules/mimeEncrypt.jsm:537
    createMessageFile resource:///modules/MimeMessage.jsm:90
mimeEncrypt.jsm:554:15
mimeEncrypt.js: caught exception: Error
Message: 'failure in finishCryptoEncapsulation, exitCode: -1'
File:    chrome://openpgp/content/modules/mimeEncrypt.jsm
Line:    537
Stack:   finishCryptoEncapsulation@chrome://openpgp/content/modules/mimeEncrypt.jsm:537:15
createMessageFile@resource:///modules/MimeMessage.jsm:90:27


Error: failure in finishCryptoEncapsulation, exitCode: -1 mimeEncrypt.jsm:537:15
    finishCryptoEncapsulation chrome://openpgp/content/modules/mimeEncrypt.jsm:537
    createMessageFile resource:///modules/MimeMessage.jsm:90
    InterpretGeneratorResume self-hosted:1455
    AsyncFunctionNext self-hosted:852
mailnews.send: NS_ERROR_XPC_JAVASCRIPT_ERROR_WITH_DETAILS: [JavaScript Error: "failure in finishCryptoEncapsulation, exitCode: -1" {file: "chrome://openpgp/content/modules/mimeEncrypt.jsm" line: 537}]'[JavaScript Error: "failure in finishCryptoEncapsulation, exitCode: -1" {file: "chrome://openpgp/content/modules/mimeEncrypt.jsm" line: 537}]' when calling method: [nsIMsgComposeSecure::finishCryptoEncapsulation]
    createMessageFile resource:///modules/MimeMessage.jsm:90
MessageSend.jsm:136:32
    createAndSendMessage resource:///modules/MessageSend.jsm:136
mailnews.send: Sending failed; , exitCode=2153185313, originalMsgURI= MessageSend.jsm:337:32
    fail resource:///modules/MessageSend.jsm:337
    createAndSendMessage resource:///modules/MessageSend.jsm:144
Error: rnp_op_sign_add_signature failed RNP.jsm:3535:19

Expected results:

The email should have been signed and sent.

What I also have tried:

  1. Export my private PGP key.
  2. Remove my private PGP key from Thunderbird.
  3. Import my private PGP key to Thunderbird and perform an online update of the key as it was expired.
  4. Set digitally signing using that key for new emails as default.

This did not change the actual results reported above.

This procedure was inspired by https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1043138

I forgot to mention: Sending digitally signed (using OpenPGP) emails worked before the automatic software update resulting in Thunderbird version 115.3.1 (64-bit). I am not sure which version I was using before. But with that update I was shown the Supernova release notes website.

Is your key using a valid hash algorithm?

status-thunderbird_esr115: --- → affected
Component: Untriaged → Security: OpenPGP
Keywords: regression
Product: Thunderbird → MailNews Core

I do not know if it is considered "valid". But the key I am using is RSA of length 4096.

I attached the public part of the key pair I am using for signing. I don't know if this is relevant.

Are you using the optional, external GnuPG configuration for secret keys?

Did you enable the new mechanism to have a separate passphrase for secret keys managed by Thunderbird?

(Please note, I don't suggest to make changes to your configuration. I'm just asking questions to better understand your configuration and possible causes.)

Do you use a specially prepared key, which contains secret keys only for some of your {primary key, subkeys} ?

Do you use separate passphrases for the secret keys?

(In reply to Magnus Melin [:mkmelin] from comment #3)

Is your key using a valid hash algorithm?

It's using SHA256 for internal signatures, it looks fine.

(In reply to Kai Engert (:KaiE:) from comment #6)

Are you using the optional, external GnuPG configuration for secret keys?

I am not sure if this is related to a Thunderbird setting or how to answer properly.

But

  • I am using a gpg client for signing with git using the same key
  • I have the Enigmail extension installed in Thunderbird (this is legacy I guess), but the error occurs independently of if the extension is enabled or not

(In reply to Kai Engert (:KaiE:) from comment #7)

Did you enable the new mechanism to have a separate passphrase for secret keys managed by Thunderbird?

Again, not sure how to answer. Is there any way I can look this up?

Anyway, I am using a primary password for Thunderbird. I only have to enter it when starting TB.

(In reply to Kai Engert (:KaiE:) from comment #8)

Do you use a specially prepared key, which contains secret keys only for some of your {primary key, subkeys} ?

Do you use separate passphrases for the secret keys?

No, or at least not to my knowledge. The TB OpenPGP Key Manager 'Structure' says I am using the primary key for signing and certifying.

IIRC the key was originally generated outside of Thunderbird or Enigmail in the year 2020. Either using gpg from "Free Software Foundation, Inc.", part of git bash or "g10 Code GmbH", part of Gpg4Win (I have both installed).

Later, when Thunderbird had their internal PGP implementation I (had to?) import my key to Thunderbird and strip it from the passphrase - IIRC.

Please let me know if there are any steps I can do to help.

Based on your responses I conclude the answer is "no" to all my questions.
(You apparently didn't do anything unusual.)

Are you running the "debian unstable" OS ?

No, I am running a different OS: Windows_NT 10.0 19045 "Windows 10 Enterprise"

Do you have any old encrypted emails, e.g. in your Sent folder, that were encrypted using this key?
If yes, can you check that you are still able to decrypt them?

I'm wondering if you have a corrupted profile with an unaccessible secret key.

Yes, I have an old email which I sent to a different recipient. That email was signed with my key and encrypted to the other recipient and myself.

The message headers says:

  • Good Digital Signature
  • Message is encrypted

Also I can still encrypt and send emails to myself, if I do not sign them.

(In reply to David H. from comment #18)

Also I can still encrypt and send emails to myself, if I do not sign them.

Encryption doesn't need the secret key.

Only signing or decrypting needs it.

So, if you send an encrypted email today (sending to yourself is fine), then go to your sent folder, and you are able to view the decrypted message, then you are able to access your secret decryption key.

But somehow you are unable to access your secret signing key, or there is a failure in the attempt to use it.

It's difficult to find out what's happening.

Here's one thing you can try, for additional debugging information from the RNP library:

If you're using Window, that might be a bit tricky to do.

You need to be comfortable using a terminal window, using the "cd" command to navigate to the directory that contains your installed Thunderbird, set the environment variable RNP_LOG_CONSOLE to value 1, and then start Thunderbird from inside the terminal window. Only that way you would be to able to see the debug output produced by RNP. However, that output might reveal why RNP is unable to use your secret key.

(In reply to Kai Engert (:KaiE:) from comment #19)

So, if you send an encrypted email today (sending to yourself is fine), then go to your sent folder, and you are able to view the decrypted message, then you are able to access your secret decryption key.

Yes, I can decrypt the sent message as well as the received message.

You need to be comfortable using a terminal window, using the "cd" command to navigate to the directory that contains your installed Thunderbird, set the environment variable RNP_LOG_CONSOLE to value 1, and then start Thunderbird from inside the terminal window. Only that way you would be to able to see the debug output produced by RNP. However, that output might reveal why RNP is unable to use your secret key.

OK, I did this:

Microsoft Windows [Version 10.0.19045.3324]
(c) Microsoft Corporation. Alle Rechte vorbehalten.

C:\Users\...>cd "C:\Program Files\Mozilla Thunderbird"

C:\Program Files\Mozilla Thunderbird>set RNP_LOG_CONSOLE=1

C:\Program Files\Mozilla Thunderbird>echo %RNP_LOG_CONSOLE%
1

C:\Program Files\Mozilla Thunderbird>thunderbird.exe

I then tried the procedure described above. But I did not get any output on stdout.

Sorry, I will have to continue further attempts on Monday.

OK, today I only have access to a different computer (also Windows 10). But this computer has the original copy of the Thunderbird profile I have been using on Friday.

First of all, the Thunderbird installation of that computer was still old:

Version 102.15.1
Build ID 20230912052514

So I did the following:

  1. Test sending digitally signed email: ✔ works
  2. Backup private key used for signing (which supposedly is the same I use on the other computer).
  3. Remove private key from Thunderbird (confirmed: can not decrypt any more)
  4. Import the private key which I exported from the other computer last week
  5. Test key usage: ✔ digitally signing and decryption works

So it seems to me, it works OK with the specific key and the old Thunderbird version.

Then I updated that Thunderbird installation. It now is:

Version 115.3.1
Build ID 20230928194049

I opened a CLI, set RNP_LOG_CONSOLE=1 and started thunderbird.exe -console.

Then I repeated the steps to reproduce from the original bug description. I got the exact same result as in the original bug description (also in the error console).

Using -console a separate shell opened for Thunderbird. But in that shell I did not get any output for the scenario above.

But when I opened an email encrypted to me the following was printed in the shell (but nothing in the error console):

[init_encrypted_src() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/librepgp/stream-parse.cpp:2156] failed to obtain decrypting key or password

Even though the email was decrypted correctly and Thunderbird shows check-marks for encryption and valid signature for that email. It seems to me this contradicts the error message (which would not have been visible without -console).

I conclude that

  • I was able to reproduce the error on a different computer with a very similar (or same) profile
  • it worked with an older version of Thunderbird, but not the newer one
Summary: Digitally signing email with PGP fails since update to Supernova → Digitally signing email with PGP fails since update to Supernova - [init_encrypted_src() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/librepgp/stream-parse.cpp:2156] failed to obtain decrypting key or password

I had the same error (old thunderbrid installation, recently updated from 102.15.1+build1-0ubuntu0.22.04.1 to 115.3.1+build1-0ubuntu0.22.04.2).
In my case this seems to be due to a public key in my keyring that I gathered on a signing party some years ago... The logs that helped me finding it:

Oct 09 14:58:15 hostname /usr/libexec/gdm-x-session[3550]: console.debug: "failed to extract email address from: XXXXXX xxx XXXXXX"
Oct 09 14:58:15 hostname /usr/libexec/gdm-x-session[3550]: console.log: "CryptoAPI.sync() failed result: %o" (new Error("rnp_op_encrypt_add_signature failed", "chrome://openpgp/content/modules/RNP.jsm", 3526))
Oct 09 14:58:15 hostname /usr/libexec/gdm-x-session[3550]: console.debug: (new Error("failure in finishCryptoEncapsulation, exitCode: -1", "chrome://openpgp/content/modules/mimeEncrypt.jsm", 537))
Oct 09 14:58:15 hostname /usr/libexec/gdm-x-session[3550]: console.error: mailnews.send: ({})
Oct 09 14:58:15 hostname /usr/libexec/gdm-x-session[3550]: JavaScript error: chrome://openpgp/content/modules/mimeEncrypt.jsm, line 537: Error: failure in finishCryptoEncapsulation, exitCode: -1
Oct 09 14:58:15 hostname /usr/libexec/gdm-x-session[3550]: console.error: mailnews.send: "Sending failed; , exitCode=2153185313, originalMsgURI=imap-message://<Mail account info omitted>/INBOX#11610"
Oct 09 14:58:15 hostname /usr/libexec/gdm-x-session[3550]: JavaScript error: chrome://openpgp/content/modules/RNP.jsm, line 3526: Error: rnp_op_encrypt_add_signature failed
[...]
Oct 09 15:17:23 hostname /usr/libexec/gdm-x-session[3550]: console.error: mailnews.ldap: ({get isTrusted() {
Oct 09 15:17:23 hostname /usr/libexec/gdm-x-session[3550]: [native code]
Oct 09 15:17:23 hostname /usr/libexec/gdm-x-session[3550]: }})
Oct 09 15:17:26 hostname /usr/libexec/gdm-x-session[3550]: JavaScript error: resource://gre/modules/LoginManagerChild.sys.mjs, line 149: NS_ERROR_FAILURE: Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIAutoCompleteInput.popup]
Oct 09 15:17:26 hostname /usr/libexec/gdm-x-session[3550]: console.debug: "failed to extract email address from: XXXXXX xxx XXXXXX"
Oct 09 15:17:30 hostname /usr/libexec/gdm-x-session[3550]: console.debug: "failed to extract email address from: XXXXXX xxx XXXXXX"
Oct 09 15:17:33 hostname /usr/libexec/gdm-x-session[3550]: console.debug: "failed to extract email address from: XXXXXX xxx XXXXXX"
Oct 09 15:17:37 hostname /usr/libexec/gdm-x-session[3550]: console.log: "CryptoAPI.sync() failed result: %o" (new Error("rnp_op_sign_add_signature failed", "chrome://openpgp/content/modules/RNP.jsm", 3535))
Oct 09 15:17:37 hostname /usr/libexec/gdm-x-session[3550]: console.debug: (new Error("failure in finishCryptoEncapsulation, exitCode: -1", "chrome://openpgp/content/modules/mimeEncrypt.jsm", 537))
Oct 09 15:17:37 hostname /usr/libexec/gdm-x-session[3550]: JavaScript error: chrome://openpgp/content/modules/mimeEncrypt.jsm, line 537: Error: failure in finishCryptoEncapsulation, exitCode: -1
Oct 09 15:17:37 hostname /usr/libexec/gdm-x-session[3550]: console.error: mailnews.send: ({})
Oct 09 15:17:37 hostname /usr/libexec/gdm-x-session[3550]: console.error: mailnews.send: "Sending failed; , exitCode=2153185313, originalMsgURI="
Oct 09 15:17:37 hostname /usr/libexec/gdm-x-session[3550]: JavaScript error: chrome://openpgp/content/modules/RNP.jsm, line 3535: Error: rnp_op_sign_add_signature failed

I managed to fix it by deleting the pubkey of "XXXXXX xxx XXXXXX" from my keyring, and restarted Thunderbird.
I can submit you the keyID in private (it's available on public keyservers) if you want to reproduce, but basically this is a key with 3 subIDs, two of which are a photo and one just a name without an e-mail address - maybe that's already enough to reproduce?

To be clear, that keyID wasn't in the recipients list nor has it been used in the last 4 years, it just happened to be in my local GPG keyring, which wasn't an issue for TB v102

HTH,
Felix

Oddly enough, I was able to send one signed mail, but am now stuck again with (partly) the same issue:

$ export RNP_LOG_CONSOLE=1 ; thunderbird -console
[ImapModuleLoader] Using nsImapService.cpp
ATTENTION: default value of option mesa_glthread overridden by environment.
ATTENTION: default value of option mesa_glthread overridden by environment.
ATTENTION: default value of option mesa_glthread overridden by environment.
Warning: unrecognized command line flag -console
console.error: "Assert failed: Calendar manager initialised calendars before loadCalendarComponent ran on the first 3pane window. This should not happen.\n "
console.debug: "Found 153 public keys and 4 secret keys (4 protected, 0 unprotected)"
console.debug: "Successfully loaded optional OpenPGP library libgpgme.so.11 from system's standard library locations"
console.debug: "gpgme version: 1.16.0"
console.warn: services.settings: Failed to load last_modified.json: TypeError: NetworkError when attempting to fetch resource.
console.debug: "Trying to load /usr/lib/thunderbird/libotr.so"
console.debug: "Trying to load libotr.so from system's standard library locations"
console.debug: "Trying to load libotr.so.5 from system's standard library locations"
console.debug: "Trying to load libotr.so from system's standard library locations"
console.log: (new Error("Cannot load required OTR library", "resource:///modules/OTRLib.sys.mjs", 110))
console.warn: Calendar: {"result":11,"message":"database disk image is malformed","ERROR":1,"INTERNAL":2,"PERM":3,"ABORT":4,"BUSY":5,"LOCKED":6,"NOMEM":7,"READONLY":8,"INTERRUPT":9,"IOERR":10,"CORRUPT":11,"FULL":13,"CANTOPEN":14,"EMPTY":16,"SCHEMA":17,"TOOBIG":18,"CONSTRAINT":19,"MISMATCH":20,"MISUSE":21,"NOLFS":22,"AUTH":23,"FORMAT":24,"RANGE":25,"NOTADB":26}
JavaScript error: resource:///modules/calendar/CalStorageDatabase.jsm, line 264: NS_ERROR_FAILURE: error executing async statement
JavaScript error: resource:///modules/calendar/CalStorageDatabase.jsm, line 264: NS_ERROR_FAILURE: error executing async statement
JavaScript error: resource:///modules/calendar/CalStorageDatabase.jsm, line 264: NS_ERROR_FAILURE: error executing async statement
console.error: Calendar:
Message: [Exception... "error executing async statement" nsresult: "0x80004005 (NS_ERROR_FAILURE)" location: "JS frame :: resource:///modules/calendar/CalStorageDatabase.jsm :: handleCompletion :: line 264" data: no]
Stack:
handleCompletion@resource:///modules/calendar/CalStorageDatabase.jsm:264:33

[init_encrypted_src() /build/thunderbird-YW7ibZ/thunderbird-115.3.1+build1/comm/third_party/rnp/src/librepgp/stream-parse.cpp:2156] failed to obtain decrypting key or password
console.log: "CryptoAPI.sync() failed result: %o" (new Error("rnp_op_encrypt_add_signature failed", "chrome://openpgp/content/modules/RNP.jsm", 3526))
console.debug: (new Error("failure in finishCryptoEncapsulation, exitCode: -1", "chrome://openpgp/content/modules/mimeEncrypt.jsm", 537))
JavaScript error: chrome://openpgp/content/modules/mimeEncrypt.jsm, line 537: Error: failure in finishCryptoEncapsulation, exitCode: -1
console.error: mailnews.send: ({})
console.error: mailnews.send: "Sending failed; , exitCode=2153185313, originalMsgURI=imap-message://<Mail account info omitted>/INBOX#11622"
JavaScript error: chrome://openpgp/content/modules/RNP.jsm, line 3526: Error: rnp_op_encrypt_add_signature failed
console.log: "CryptoAPI.sync() failed result: %o" (new Error("rnp_op_encrypt_add_signature failed", "chrome://openpgp/content/modules/RNP.jsm", 3526))
console.debug: (new Error("failure in finishCryptoEncapsulation, exitCode: -1", "chrome://openpgp/content/modules/mimeEncrypt.jsm", 537))
JavaScript error: chrome://openpgp/content/modules/mimeEncrypt.jsm, line 537: Error: failure in finishCryptoEncapsulation, exitCode: -1
console.error: mailnews.send: ({})
console.error: mailnews.send: "Sending failed; , exitCode=2153185313, originalMsgURI=imap-message://krohn%40puzzle.ch@mail.puzzle.ch/INBOX#11622"
JavaScript error: chrome://openpgp/content/modules/RNP.jsm, line 3526: Error: rnp_op_encrypt_add_signature failed

Those last 5 lines repeat everytime I try to send the message, where TB displays a "Send Message Error" with only "Sending of the message failed."

Thanks Felix, I think you're providing helpful information.

In your first log (comment 22), function rnp_op_sign_add_signature failed.
This is the same place that also failed for David.

The second log (comment 23) fails in function rnp_op_encrypt_add_signature.

So apparently, by deleting that one public key, you have partially fixed the issue.

Could you please send the key ID you had deleted to kaie@kuix.de ?

Maybe this will enable me to analyze the first part of the problem, and it might even provide hints what else might go wrong.

Nickolay, we don't know the details yet, but apparently RNP 0.17.0 is failing with some keys that worked with 0.16.3

Hopefully the information I'll receive from Felix will help us.

Magnus had changed the bug subject to an error message produced by RNP - but I think it's unlikely that error is a direct cause of this bug. It might be a side effect of some other data processing. But we'll see once we can reproduce.

Felix sent the key to me, but its presence doesn't cause problems for me.

(In reply to Kai Engert (:KaiE:) from comment #25)

Nickolay, we don't know the details yet, but apparently RNP 0.17.0 is failing with some keys that worked with 0.16.3

To clarify, it isn't yet clear whether it's indeed a regression in RNP.

Potentially, the bug could also be caused by the different strategy that Thunderbird 115 uses for unlocking secret keys.

In 78 to 102, unlocking was done on demand, through a prompt from the RNP callback. TB 115 no longer uses that callback, it requires that TB must have already unlocked the respective key.

In theory, we could have a code path, where TB needs to use a secret key for signing, but hasn't unlocked that secret key yet.

Status: UNCONFIRMED → NEW
Ever confirmed: true

David, Felix, I want to prepare a special version of Thunderbird, which logs additional information to the error console.

It will be a version based on the latest 115.x, which means you can use that version with your regular Thunderbird profile, the one that you currently use with Thunderbird 115.

I understand you either need the Windows 64bit or Linux 64bit versions.

If you would like to help by using it, you would:

  • download the archive from a link I will post here
  • completely quit your regular thunderbird
  • extract the archive to a separate, new folder
  • use a terminal to "cd" to that folder where you have extracted
  • use the following command to run Thunderbird, which will bring up a prompt to select the profile you wish to use:
    Linux:
    ./thunderbird -P
    Windows:
    thunderbird -P

After starting, you can open the error console with shortcut SHIFT+CTRL+J

Watch for lines that contain ###, which will tell you which key Thunderbird is trying to use, and for lines with "rnp_op_encrypt_add_signature failed" or "rnp_op_sign_add_signature failed", which will contain an additional error code, that will tell us why the call failed.

Hello Kai,

Thank you so much for digging into this!

Here's the output:

1C0E3B978055D16527864331AAC6F123CE81E5CE successfully unlocked with autoPassword RNP.jsm:276:15

console.trace() RNP.jsm:3534:23
encryptAndOrSign chrome://openpgp/content/modules/RNP.jsm:3534
sync chrome://openpgp/content/modules/cryptoAPI/interface.js:56
encryptMessageStart chrome://openpgp/content/modules/encryption.jsm:446
finishCryptoEncapsulation chrome://openpgp/content/modules/mimeEncrypt.jsm:516
createMessageFile resource:///modules/MimeMessage.jsm:90

1C0E3B978055D16527864331AAC6F123CE81E5CE locking secret key RNP.jsm:441:38

CryptoAPI.sync() failed result: Error: rnp_op_sign_add_signature failed: 301989894
encryptAndOrSign chrome://openpgp/content/modules/RNP.jsm:3534
sync chrome://openpgp/content/modules/cryptoAPI/interface.js:56
encryptMessageStart chrome://openpgp/content/modules/encryption.jsm:446
finishCryptoEncapsulation chrome://openpgp/content/modules/mimeEncrypt.jsm:516
createMessageFile resource:///modules/MimeMessage.jsm:90
interface.js:46:17
Error: failure in finishCryptoEncapsulation, exitCode: -1
finishCryptoEncapsulation chrome://openpgp/content/modules/mimeEncrypt.jsm:537
createMessageFile resource:///modules/MimeMessage.jsm:90
mimeEncrypt.jsm:554:15
mimeEncrypt.js: caught exception: Error
Message: 'failure in finishCryptoEncapsulation, exitCode: -1'
File: chrome://openpgp/content/modules/mimeEncrypt.jsm
Line: 537
Stack: finishCryptoEncapsulation@chrome://openpgp/content/modules/mimeEncrypt.jsm:537:15
createMessageFile@resource:///modules/MimeMessage.jsm:90:27

Error: failure in finishCryptoEncapsulation, exitCode: -1 mimeEncrypt.jsm:537:15
mailnews.send: NS_ERROR_XPC_JAVASCRIPT_ERROR_WITH_DETAILS: [JavaScript Error: "failure in finishCryptoEncapsulation, exitCode: -1" {file: "chrome://openpgp/content/modules/mimeEncrypt.jsm" line: 537}]'[JavaScript Error: "failure in finishCryptoEncapsulation, exitCode: -1" {file: "chrome://openpgp/content/modules/mimeEncrypt.jsm" line: 537}]' when calling method: [nsIMsgComposeSecure::finishCryptoEncapsulation]
createMessageFile resource:///modules/MimeMessage.jsm:90
MessageSend.jsm:136:32
mailnews.send: Sending failed; , exitCode=2153185313, originalMsgURI= MessageSend.jsm:337:32
Error: rnp_op_sign_add_signature failed: 301989894 RNP.jsm:3534:38

By the way, I can read messages sent to me encrypted, so the unlocking of the secret key must be successful somehow - just the signing seems to cause issues.

(In reply to Kai Engert (:KaiE:) from comment #29 and comment #30)

I used the Windows version of the special build (ID: 20231009212419) you provided to go through the steps of the original bug description. The special output lines containing ### appear twice in the error console. Find the full output attached.

(In reply to Felix Krohn from comment #32)

By the way, I can read messages sent to me encrypted, so the unlocking of the secret key must be successful somehow - just the signing seems to cause issues.

Same here.

(In reply to Kai Engert (:KaiE:) from comment #35)

I used the Windows version of the special build (ID: 20231010094303) you provided to go through the steps of the original bug description. Find the full output attached.

Attachment #9357573 - Attachment is obsolete: true

Thanks David.

It looks like Thunderbird itself does everything correctly.

I was briefly worried that Thunderbird might re-lock the key to early. However, at the time Thunderbird asks the RNP library to use that key for signing, the secret key is unlocked.

We don't get to the step where we ask RNP to create the signature on the data. It fails earlier. It fails already when we inform RNP which key should be used, and RNP rejects the key.

In the patch that you tried, the code checks that the secret key material is indeed available, and this test has passed.

I think this means the bug/regression is at the RNP library layer.

Nickolay:

rnp_op_sign_add_signature returns failure RNP_ERROR_NO_SUITABLE_KEY.

The key is unlocked, the key passed rnp_key_have_secret(),
and rnp_key_get_protection_type() is NOT "GPG-None", "GPG-Smartcard", "Unknown".

Do you have an idea why this might happen for a key in 0.17.0, but did not happen with 0.16.3 ?

Flags: needinfo?(o.nickolay)

Hi Kai,
There were changes in RNP code, used by rnp_op_sign_add_signature(), and related to the checks whether key is appropriate for the specific operation. So this partially or fully should be the reason. However, checking the keys attached it seems that those changes must work well.

Would it be possible to make a test build with some more logging, added to the RNP codebase, to further debug the issue, maybe I mislooking something?

Flags: needinfo?(o.nickolay)

Yes that's fine, thanks, please do.

Please change the function usable_for() in the src/lib/pgp-key.cpp to the following (only RNP_LOG() lines were added):

bool
pgp_key_t::usable_for(pgp_op_t op, bool if_secret) const
{
    RNP_LOG("usable_for: op: %d, if_secret: %d", (int) op, if_secret);
    switch (op) {
    case PGP_OP_ADD_SUBKEY:
        return is_primary() && can_sign() && (if_secret || has_secret());
    case PGP_OP_SIGN:
        RNP_LOG("usable_for: can_sign: %d, valid: %d, if_secret: %d, has_secret: %d",
                can_sign(),
                valid(),
                if_secret,
                has_secret());
        RNP_LOG("usable_for: is_secret: %d, format: %d, sec_len: %d, s2kuse: %d, s2kspec: %d",
                is_secret(),
                (int) format,
                (int) pkt_.sec_len,
                (int) pkt_.sec_protection.s2k.usage,
                (int) pkt_.sec_protection.s2k.specifier);
        return can_sign() && valid() && (if_secret || has_secret());
    case PGP_OP_CERTIFY:
        return can_certify() && valid() && (if_secret || has_secret());
    case PGP_OP_DECRYPT:
        return can_encrypt() && valid() && (if_secret || has_secret());
    case PGP_OP_UNLOCK:
    case PGP_OP_PROTECT:
    case PGP_OP_UNPROTECT:
        return has_secret();
    case PGP_OP_VERIFY:
        return can_sign() && valid();
    case PGP_OP_ADD_USERID:
        return is_primary() && can_sign() && (if_secret || has_secret());
    case PGP_OP_ENCRYPT:
        return can_encrypt() && valid();
    default:
        return false;
    }
}

David and Felix, I'm glad you were both able to get RNP log output using the -console parameter.

(I wouldn't worry about much about the cause of the output line in "stream-parse" mentioning "failed to obtain decrypting key or password".
I also get this message regularly, without any apparently failures on my side. I haven't yet analyzed what's causing it. I'm speculating it could be an attempt to decrypt with the wrong key material, while trying to find the correct key.)

Thanks for Nickolay. I will shortly provide updated builds for testing.

(In reply to Kai Engert (:KaiE:) from comment #42)

(I wouldn't worry about much about the cause of the output line in "stream-parse" mentioning "failed to obtain decrypting key or password".
I also get this message regularly, without any apparently failures on my side. I haven't yet analyzed what's causing it. I'm speculating it could be an attempt to decrypt with the wrong key material, while trying to find the correct key.)

Kai,
As far as I remember from other tickets or emails, you first try decryption without the keys to gather information about the recipients. That should invoke this error message.

Correct, that explains it, thanks!

Linux64:
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/ROut40pgRay525Q2iljzUg/runs/0/artifacts/public/build/target.tar.bz2

Hi Kai,
thanks for the build! I ran it and tried to send the same message from the same account to the same recipient again, and got the same error :) See the attached debug console output.

For completeness' sake here's the output on stdout/stderr:

console.log: "CryptoAPI.sync() failed result: %o" (new Error("rnp_op_sign_add_signature failed", "chrome://openpgp/content/modules/RNP.jsm", 3535))
console.debug: (new Error("failure in finishCryptoEncapsulation, exitCode: -1", "chrome://openpgp/content/modules/mimeEncrypt.jsm", 537))
JavaScript error: chrome://openpgp/content/modules/mimeEncrypt.jsm, line 537: Error: failure in finishCryptoEncapsulation, exitCode: -1
console.error: mailnews.send: ({})
console.error: mailnews.send: "Sending failed; , exitCode=2153185313, originalMsgURI=imap-message://<account info omitted>/Drafts#467"
JavaScript error: chrome://openpgp/content/modules/RNP.jsm, line 3535: Error: rnp_op_sign_add_signature failed

Cheers,
Felix

Felix, I don't see those log statements in your attachment. You'd still need to set the RNP_CONSOLE=1 environment variable to get them.

sorry, RNP_LOG_CONSOLE

Yeah, I did that...
$ RNP_LOG_CONSOLE=1 ; thunderbird -console

I just ran this again after a killall thunderbird just to exclude the possibility of any old instance lingering somewhere on this PC, and tried sending the mail again a couple times.
the output looks identical to me, see attachment.
Sorry I'm not able to interpret it myself :)

Felix

Thanks for trying again, but there is no relevant output on your console.
Everything you've showed looks like the output from the "Javascript" or "Developers error console".

What we need is the output from the terminal window.

For example, messages that look like

[init_encrypted_src() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/librepgp/stream-parse.cpp:2156] failed to obtain decrypting key

as seen in David's comment 21.

For example, if I run the latest code and sign an email, I get these log lines:

[usable_for() rnp/src/lib/pgp-key.cpp:1235] usable_for: op: 9, if_secret: 0
[usable_for() rnp/src/lib/pgp-key.cpp:1235] usable_for: op: 12, if_secret: 0
[usable_for() rnp/src/lib/pgp-key.cpp:1235] usable_for: op: 12, if_secret: 0
[usable_for() rnp/src/lib/pgp-key.cpp:1235] usable_for: op: 4, if_secret: 0
[usable_for() rnp/src/lib/pgp-key.cpp:1235] usable_for: op: 4, if_secret: 0
[usable_for() rnp/src/lib/pgp-key.cpp:1235] usable_for: op: 2, if_secret: 0
[usable_for() rnp/src/lib/pgp-key.cpp:1244] usable_for: can_sign: 1, valid: 1, if_secret: 0, has_secret: 1
[usable_for() rnp/src/lib/pgp-key.cpp:1250] usable_for: is_secret: 1, format: 1, sec_len: 988, s2kuse: 254, s2kspec: 3

Sorry, my bad... I forgot the ./ and started the version installed in $PATH again by error.
Here you go:

$ export RNP_LOG_CONSOLE=1 ; ./thunderbird
ATTENTION: default value of option mesa_glthread overridden by environment.
ATTENTION: default value of option mesa_glthread overridden by environment.
ATTENTION: default value of option mesa_glthread overridden by environment.
[init_encrypted_src() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/librepgp/stream-parse.cpp:2156] failed to obtain decrypting key or password
[usable_for() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/lib/pgp-key.cpp:1234] usable_for: op: 4, if_secret: 0
[usable_for() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/lib/pgp-key.cpp:1234] usable_for: op: 4, if_secret: 0
[usable_for() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/lib/pgp-key.cpp:1234] usable_for: op: 9, if_secret: 0
[usable_for() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/lib/pgp-key.cpp:1234] usable_for: op: 12, if_secret: 0
[usable_for() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/lib/pgp-key.cpp:1234] usable_for: op: 12, if_secret: 0
[usable_for() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/lib/pgp-key.cpp:1234] usable_for: op: 12, if_secret: 0
[usable_for() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/lib/pgp-key.cpp:1234] usable_for: op: 4, if_secret: 0
[usable_for() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/lib/pgp-key.cpp:1234] usable_for: op: 2, if_secret: 0
[usable_for() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/lib/pgp-key.cpp:1243] usable_for: can_sign: 1, valid: 0, if_secret: 0, has_secret: 1
[usable_for() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/lib/pgp-key.cpp:1249] usable_for: is_secret: 1, format: 1, sec_len: 1308, s2kuse: 254, s2kspec: 3
[usable_for() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/lib/pgp-key.cpp:1234] usable_for: op: 2, if_secret: 0
[usable_for() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/lib/pgp-key.cpp:1243] usable_for: can_sign: 0, valid: 0, if_secret: 0, has_secret: 1
[usable_for() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/lib/pgp-key.cpp:1249] usable_for: is_secret: 1, format: 1, sec_len: 1308, s2kuse: 254, s2kspec: 3

Thanks Felix.
Here is my interpretation:

RNP thinks your signing key is not valid.
Apparently RNP considers the use of another key for signing, but concludes that key cannot be used for signing (and isn't valid either).

I don't know which behavior has changed in RNP.

Did the signing keys expire?

And a stupid question, because I don't know if that's possible:
Could your secret key have a different expiration date set, and has already expired?

With your exporting and re-importing, could you have a copy of a private key from before expiration extension, and only your public key has an extended expiration?

Here is a (very) advanced test procedure that you could do.

First, determine the path of your profile directory. You could do so from within Thunderbird, burger menu (three lines upper right), help, troubleshooting information, search "about:profiles", click it. If you know the name of the profile you're using, you can find here the path using the "root directory" information.

Then, in the release archive that you have downloaded, there is also a binary called "rnpkeys" which we can use to gather some information. However, I advise that you do NOT run it directly on your profile directory. Rather, let's use a separate, temporary directory, and copy stuff to there.

Go to your terminal window, and use "cd" to change into directory where you downloaded and extracted the experimental Thunderbird build. That's the same directory from which you earlier ran Thunderbird (using ./thunderbird).

Once you are there, create two subdirectories:
$ mkdir sec-dir
$ mkdir pub-dir

Now let's copy only one file to each directory. Let's say your profile directory is PROFDIR. Below, replace PROFDIR with the actual path of your profile directory.

$ cp -i PROFDIR/secring.gpg ./sec-dir
$ cp -i PROFDIR/pubring.gpg ./pub-dir

Next, list the contents of the public directory, and to keep the output short, at the end of the command, add an identifier for your key (e.g. key id or your email address).

$ ./rnpkeys --homedir ./pub-dir --list-keys YOUR-KEY-IDENTIFIER

I assume it will list your with the regular expiration date, and probably shown as currently valid.

Now let's look at the secret keys:

$ ./rnpkeys --homedir ./sec-dir --list-keys YOUR-KEY-IDENTIFIER

Does it show the same expiration dates, or older dates?

(Note on my system, rnpkeys displays an additional [INVALID] next to the keys, but I guess that's simply because it cannot find the related public key, and shouldn't matter for this experiment.)

(In reply to Kai Engert (:KaiE:) from comment #53)

$ cp -i PROFDIR/secring.gpg ./sec-dir
$ cp -i PROFDIR/pubring.gpg ./pub-dir

On Windows, you probably have to use the "copy" command (don't give parameter -i):
$ copy PROFDIR/secring.gpg ./sec-dir
$ copy PROFDIR/pubring.gpg ./pub-dir

Kai,
Could it be the case that pubring and secring are out of sync, and secret keys are loaded to FFI separately? My guess is that not all signatures are attached to the secret key, giving it invalid (which could be expired as well). Output your asked for could give some clue (as well as alternative ./rnp --list-packets secring.gpg).

(In reply to Kai Engert (:KaiE:) from comment #45)

David, Felix, here are the new test builds that will produce additional logging output for Nickolay:

Win64:
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/RhxfuyMGRWKz5HE6FxhfUQ/runs/0/artifacts/public/build/target.zip

Here is the complete output for when I execute the steps from the original bug report.

(In reply to Kai Engert (:KaiE:) from comment #52)

And a stupid question, because I don't know if that's possible:
Could your secret key have a different expiration date set, and has already expired?

With your exporting and re-importing, could you have a copy of a private key from before expiration extension, and only your public key has an extended expiration?

This is probably the case with my key.

(In reply to David H. from comment #1)

What I also have tried:

  1. Export my private PGP key.
    ...
  2. Import my private PGP key to Thunderbird and perform an online update of the key as it was expired.

So I originally exported my private key using Thunderbird. I did just check it using this in my git bash:

cat secret_old.asc | gpg --show-keys --with-sig-list

And yes, the private key is expired.

sec# rsa4096 2020-02-05 [SC] [expired: 2021-01-31]

This explains why I had to update my key (see step 3 above) for Thunderbird to allow to use it. But I only updated the public key. And Thunderbird accepted this. Maybe to hasty?

So I did the following:

  1. Set the expiration date for my private key using gpg to a date in the future.
  2. Exported my private key to file using gpg.
  3. Removed my private key from Thunderbird (build ID 20231016092201).
  4. Imported the file with my extended private key.
  5. Test to send a digitally signed (using the above key) email: Success ✔

So I assume that the new Thunderbird version can not use expired private keys for signing any more. The check in "Account Settings", "End-To-End Encryption", "OpenPGP" shows that the key is valid as soon as the public part of the key pair is valid. 😕

Attachment #9358938 - Attachment is obsolete: true

(In reply to Kai Engert (:KaiE:) from comment #53)

Here is a (very) advanced test procedure that you could do.

...

$ ./rnpkeys --homedir ./pub-dir --list-keys YOUR-KEY-IDENTIFIER

So this does not work on my system 🤷‍♀️

If I execute it in the Windows shell it outputs

Home directory '' does not exist or is not writable!
fatal: cannot set keystore info

I did test different arguments for --homedir, without success.

If I execute it in git bash it outputs:

Keyring directory 'C:\Users\▓▓▓▓▓▓▓▓.ST\.rnp' is empty.
Use "rnpkeys" command to generate a new key or import existing keys from the file or GnuPG keyrings.
rnp 0.17.0+PR2073.MZLA.115.3.3
https://bugzilla.mozilla.org/enter_bug.cgi?product=Thunderbird
Backend: Botan
Backend version: 2.18.2
Supported algorithms:
Public key:  RSA, ELGAMAL, DSA, ECDH, ECDSA, EDDSA
Encryption:  IDEA, TRIPLEDES, CAST5, BLOWFISH, AES128, AES192, AES256, TWOFISH, CAMELLIA128, CAMELLIA192, CAMELLIA256
AEAD:  None, EAX, OCB
Key protection:  CFB
Hash:  MD5, SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224, SHA3-256, SHA3-512
Compression:  Uncompressed, ZIP, ZLIB, BZip2
Curves:  NIST P-256, NIST P-384, NIST P-521, Ed25519, Curve25519, brainpoolP256r1, brainpoolP384r1, brainpoolP512r1, secp256k1
Please report security issues at (https://www.rnpgp.org/feedback) and
general bugs at https://github.com/rnpgp/rnp/issues.
Manipulate OpenPGP keys and keyrings.
Usage: rnpkeys --command [options] [files]
Commands:
  -h, --help             This help message.
  -V, --version          Print RNP version information.
...

The usage information (beginning in line 3) is fully printed for each argument I gave to rnpkeys. I did put one of the key files into the path given in the warning (line 1). This had only the effect that the warning (line 1 and 2) was not printed any more. But it did still print the usage information as often as the number of arguments I provided. No key information was printed.

But maybe the relevant information was already provided in comment #57? Can I provide any more information or run tests?

So yes, my key expired earlier this year, and I renewed them and re-imported in TB (I believe).
The secret key is indeed expired... I guess you're on to something here :-)

$ ../thunderbird/rnpkeys --homedir ./pub-dir --list-keys AAC6F123CE81E5CE 
2 keys found

pub   4096/RSA aac6f123ce81e5ce 2018-08-02 [SCA] [EXPIRES 2028-08-22]
      1c0e3b978055d16527864331aac6f123ce81e5ce
uid           Felix Krohn <MY_MAIL_ADDRESS>
sub   4096/RSA 728bbe01cd8dcc56 2018-08-02 [E] [EXPIRES 2028-08-22]
      bd9fa21e6a92caf2868d8010728bbe01cd8dcc56
$ ../thunderbird/rnpkeys --homedir ./sec-dir --list-keys AAC6F123CE81E5CE
2 keys found

pub   4096/RSA aac6f123ce81e5ce 2018-08-02 [SCA] [EXPIRED 2023-08-01]
      1c0e3b978055d16527864331aac6f123ce81e5ce
uid           Felix Krohn <MY_MAIL_ADDRESS>
sub   4096/RSA 728bbe01cd8dcc56 2018-08-02 [E] [EXPIRED 2023-08-01]
      bd9fa21e6a92caf2868d8010728bbe01cd8dcc56

I closed TB, removed the secret keyring:

$ cp secring.gpg{,.bak}
$ cp pubring.gpg{,.bak}
$ rm secring.gpg
$ gpg --export-secret-keys MY_MAIL_ADDRESS -a > krohn@puzzle.ch.secret.gpg

and then re-imported my private keys using the "OpenPGP Key Manager".
Result: everything seems to be working now.

The console log of the last test build you posted now looks like this:

$ export RNP_LOG_CONSOLE=1 ; ./thunderbird
ATTENTION: default value of option mesa_glthread overridden by environment.
ATTENTION: default value of option mesa_glthread overridden by environment.
ATTENTION: default value of option mesa_glthread overridden by environment.
[init_encrypted_src() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/librepgp/stream-parse.cpp:2156] failed to obtain decrypting key or password
[usable_for() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/lib/pgp-key.cpp:1234] usable_for: op: 4, if_secret: 0
[usable_for() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/lib/pgp-key.cpp:1234] usable_for: op: 4, if_secret: 0
[usable_for() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/lib/pgp-key.cpp:1234] usable_for: op: 9, if_secret: 0
[usable_for() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/lib/pgp-key.cpp:1234] usable_for: op: 12, if_secret: 0
[usable_for() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/lib/pgp-key.cpp:1234] usable_for: op: 12, if_secret: 0
[usable_for() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/lib/pgp-key.cpp:1234] usable_for: op: 12, if_secret: 0
[usable_for() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/lib/pgp-key.cpp:1234] usable_for: op: 4, if_secret: 0
[usable_for() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/lib/pgp-key.cpp:1234] usable_for: op: 2, if_secret: 0
[usable_for() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/lib/pgp-key.cpp:1243] usable_for: can_sign: 1, valid: 1, if_secret: 0, has_secret: 1
[usable_for() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/lib/pgp-key.cpp:1249] usable_for: is_secret: 1, format: 1, sec_len: 1308, s2kuse: 254, s2kspec: 3
[usable_for() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/lib/pgp-key.cpp:1234] usable_for: op: 12, if_secret: 0
[usable_for() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/lib/pgp-key.cpp:1234] usable_for: op: 12, if_secret: 0
[usable_for() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/lib/pgp-key.cpp:1234] usable_for: op: 12, if_secret: 0
[init_encrypted_src() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/librepgp/stream-parse.cpp:2156] failed to obtain decrypting key or password
[usable_for() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/lib/pgp-key.cpp:1234] usable_for: op: 4, if_secret: 0
[usable_for() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/lib/pgp-key.cpp:1234] usable_for: op: 4, if_secret: 0

David, yes your information from comment 57 was already sufficient.

Sorry to see my instructions didn't work on Windows. (I had provided them based on my best guess, but I had not confirmed they actually work like this.)

I'm glad the instructions worked for Felix on Linux.

So the issue is confirmed, you have expired secret keys, and Thunderbird/RNP reject to use them, regardless of the respective public key having a longer expiration.

Nickolay, do you think this is the correct behavior of RNP, and Thunderbird should try to detect this scenario, and inform the user?

Or would you consider that a bug in RNP and want to fix it?

I can accept both options. I can understand if you want to keep the new behavior of 0.17.0

For future reference: To sum up the procedure Felix and me were using to work around the issue (comment #57, comment #60). We did this by

  1. exporting the private key from Thunderbird
  2. removing the expired private key from Thunderbird (Felix removed secring.gpg, I used "OpenGPG Key Manager")
  3. extending the private key (using gpg)
  4. importing the private key back to Thunderbird

Thank you for your support Kai and Nickolay! 🙏💐

I wonder if it's really necessary to remove the private key from Thunderbird.
Importing the newer private key might be sufficient.

In other words, it might be possible to drop step 2 from David's comment 63.
(Would be nice to get this confirmed. I'll probably experiment with this myself when working on a way to signal this inconsistency to users.)

(In reply to Nickolay Olshevsky from comment #55)

Could it be the case that pubring and secring are out of sync, and secret keys are loaded to FFI separately?

I assume that whenever we import a secret key, the related public is also imported?

I think the following can happen.
Because Thunderbird doesn't offer integrated key management for complex keys
(keys with different expiration setting for primary key and subkeys),
users who want to extend the validity of their key might decide to change the validity in gnupg and re-import only the public key.

In other words:

  • user has a secret key with matching public key in Thunderbird
  • user uses gnupg to extend the validity (set a different expiration date)
  • user exports public key from gnupg
  • user imports refreshed public key, only, into Thunderbird

As of today, in this situation, Thunderbird doesn't perform any consistency checks with regards to an existing private key, when importing a public key.

Summary: Digitally signing email with PGP fails since update to Supernova - [init_encrypted_src() /builds/worker/checkouts/gecko/comm/third_party/rnp/src/librepgp/stream-parse.cpp:2156] failed to obtain decrypting key or password → No longer able to digitally sign email with OpenPGP in TB 115, if only the public key is still valid, but the private key has expired.

(In reply to Kai Engert (:KaiE:) from comment #64)

I wonder if it's really necessary to remove the private key from Thunderbird.

I can confirm: It's not necessary. As I am maintaining two setups I just tried it with the other one today: I just imported the extended private key and was then able to sign and send a composed email (did not need to close the TB or the composer).

(In reply to Kai Engert (:KaiE:) from comment #62)

Nickolay, do you think this is the correct behavior of RNP, and Thunderbird should try to detect this scenario, and inform the user?

Or would you consider that a bug in RNP and want to fix it?

This check was added to disallow user to use his secret key when it is expired, revoked or is invalid in any other way. All this validity information is stored in signatures, which are attached to the key or it's userids. Updating key expiration by GnuPG would append new signature to the key.

As I understand, within Thunderbird situation is as following:

  • you maintain secret and public keyrings separately, loading them via different FFI objects (am I wrong with this step?).
  • when you load public key, with updated expiration, you add it (and all new signatures) only to the public keyring.
  • so, new signature absent in the secret keyring, and corresponding key is considered as expired.
  • the same would happen if you have a secret key, and then import corresponding public key with new userids: those would be visible for the public key, but unusable for the secret key, confusing the user.

The solution for this would be to check whether imported public key has corresponding key in the secret keyring, and import it to the secret keyring as well. Internally RNP will keep secret key part and attach all new signatures/userids/whatever else from the public key to it.

A single FFI is used for both secret and public keys. Both files secring.gpg and pubring.gpg are loaded into the same FFI.

Hm, let me check the code and do some more tests. I believed (and it was the intension during the implementation) that RNP should append everything to both public and secret keys during the import and key load, keeping both in sync.

Okay, confirmed: rnp_import_keys() transfers all the stuff from secret key to the public keyring, but not in the reverse direction, and this is the root cause of this issue. It worked in v0.16.2 and stopped to work with v0.17.0 since we added secret key validity check before the signing. Will be fixed on RNP side.

Thanks a lot Nickolay for identifying the cause and for working on the fix.

As I understand it, after your fix, the secret key will get the updated expiration information, even after importing an updated public key, only.
This means, in the future, importing an extended public key would be sufficient again.

What should be done about users who have already run into this situation?
Can we repair them?

Could RNP automatically repair, whenever encountering this inconsistency?
Or should Thunderbird detect and trigger a repair?

You don't have to answer my questions now, I'll wait for your fix.

I'm now guessing that Thunderbird cannot do much, because Thunderbird uses a single FFI for everything, and that means cannot easily inspect secret and public keys separately. Maybe RNP can internally just perform checks in both directions, and allow secret keys to be treated as non-expired, even if only the public key storage has the extended validity. But I'll now stop guessing and wait for you :-)

Hi Kai, sorry for a delay with reply.

(In reply to Kai Engert (:KaiE:) from comment #71)

As I understand it, after your fix, the secret key will get the updated expiration information, even after importing an updated public key, only.
This means, in the future, importing an extended public key would be sufficient again.

Yes, all new signatures/userids will be added to both public and secret keys, having those in sync. The only thing is that I'm not sure about the internal implementation, however this would not change external FFI interface in any way.

What should be done about users who have already run into this situation?
Can we repair them?

Depending on the internal implementation of the fix there could be two scenarios:

  • (easier) loading keyrings will fix the things in memory, and saving it back would fix the things on the disk.
  • more cumbersome: Thunderbird would need to import public keyring back and then save things to the disk. I'm more stick to the first way, as it would allow to make things easier, but that would require more internal changes.

Could RNP automatically repair, whenever encountering this inconsistency?
Or should Thunderbird detect and trigger a repair?

I don't think that RNP as a 'tool' should make automatic changes on drive, that should be controlled by the FFI user (i.e. Thunderbird).

(In reply to Nickolay Olshevsky from comment #73)

  • (easier) loading keyrings will fix the things in memory, and saving it back would fix the things on the disk.

This sounds like a very good solution.

I think it's fine to delay fixing the on-disk storage until Thunderbird needs to write out new files.

(If Thunderbird doesn't have a need to write the files in a session, we rely on RNP doing the in-memory repair again in the next session.)

Nickolay, are you tracking this issue in your github space?

Kai, created issue here: https://github.com/rnpgp/rnp/issues/2157

Can somebody confirm: At the moment I (still) have to import the extended private key to Thunderbird to avoid this issue?

I am used to extend my key on one device, publish it and just import it on all the other devices via public channels. I guess this does not work with the current state of Thunderbird any more.

(In reply to David H. from comment #77)

Can somebody confirm: At the moment I (still) have to import the extended private key to Thunderbird to avoid this issue?

Confirmed.

While we're still waiting for a fix from RNP, there's other good news:

Thunderbird 128 has support for managing the expiration date of any subkey, even if it was considered a "too complex" key in past versions, see bug 1666507.

Whiteboard: [RNP]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: