1371243 - Signing certificates become unsusable losing their dbkey pref on the first use.

Status: RESOLVED INVALID

(Thunderbird :: Security, defect)

Description

[Martin Pecka]

Attachment: One of the certificates I'm having problem with.

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0
Build ID: 20170524175422

Steps to reproduce:

I have a few identities set with signing keys located on a smartcard.

I want to send an email and sign it.


Actual results:

The mail window shows an error that the certificate can't be used. When I open the identity properties, Security tab, there are two cases on what's shown in the Electronic signature field:

1) OKsmart 3.0 MiniOs:[e-mail] [1C:AF:98:F4:44:11:55:F7:07:09:DA:6D:79:06:59:49]

2) (no nickname) [1C:AF:98:F4:44:11:55:F7:07:09:DA:6D:79:06:59:49]

The first is shown when I set the signing cert. Then after a few minutes of using TB and sending a few emails, option 1) suddenly changes to 2) and signing emails with this key is no longer possible. 

When this happens, any try to set back the correct certificate results in the Identity->Security->Choose dialog to tell me there are no usable certs on the card. In this case, restarting TB and then resetting the cert is the only way I found to temporarily fix this.

This used to work correctly before I updated to TB 52 on Ubuntu. My TB 52 on Windows doesn't suffer from this bug (in both computers, I use the same smartcard). 

The smartcard used is OKSmart 3.0 smartcard with PKCS11 library.

Comment 1

[Martin Pecka]

This is the prefs.js part defining the signing certificate:

user_pref("mail.identity.id1.signing_cert_name", "OKsmart 3.0 MiniOs:9085F81FA42F1DBA077D2E3F259464FD");

Comment 2

[Jorg K (CEST = GMT+2)]

(In reply to Martin Pecka from comment #0)
> This used to work correctly before I updated to TB 52 on Ubuntu. My TB 52 on
> Windows doesn't suffer from this bug.
That will make it hard to track down. Are there any system components that handle those certificates (differently) on Linux ans Windows?

Comment 3

[Martin Pecka]

I don't observe any differences in the card's behavior in other applications. Of course there can be differences in the PKSC11 driver implementations on Windows and Linux, but as I've said, this buggy behavior did not occur in older TBs.

Actually, Windows is also partially affected - after some time (longer than on Linux), TB just can't read any cert from the smartcard, and only restarting TB helps (reinserting the card does not). But the "(no nickname)" thing is only happening on Linux (and if it happens, some certs can be read, while other can't).

Comment 4

[Nicolas T]

I think I have the same problem in v52.9.1 and v60.0 too (both Windows clients).

I use another smart card, an IDProtect Athena token. Drivers are correctly installed, the PKCS module is loaded and the certificate is selected in security tab.

If I config everything and try to sign an email, it works. But, if I close the client, open it, and test it once again... it fails.

It fails because Thunderbird doesn't bring up the autentication dialog, that's why the signing process fails. I think that in previous versions Thunderbird asked for the smart card password on opening, but now it doesn't.

In v52.9.1 you can go to options -> advanced -> certificates -> security devices. In Device Manager, you select the Smart Card, and in the right corner you have the option to "Log In". That brings the "Password Required" dialog. But in v60.0 it's grayed out. In that case, you can go to Options -> Account Settings -> Security -> Digital Signing -> Select.

I really expect someone can fix this, because I work for a large government organization with more than five thousand computers and we have the requirement to be able to sign emails.

Comment 5

[Martin Pecka]

You're right the Log in button is disabled. Thank you for this input, since it led me to discover a bit more about this bug.

If you open the Secrurity devices dialog and click the smartcard entry, this error shows up in the error console:

TypeError: Services.policies is undefined device_manager.js:166:11
	enableButtons chrome://pippki/content/device_manager.js:166:11
	onselect chrome://pippki/content/device_manager.xul:1:1
	onxblmousedown chrome://global/content/bindings/tree.xml:1156:14
	openOptionsDialog chrome://messenger/content/mailCore.js:446:7
	oncommand chrome://messenger/content/messenger.xul:1:1

And the problematic lines are:

if (!Services.policies.isAllowed("createMasterPassword") &&
          selected_token.isInternalKeyToken &&
          !selected_token.hasPassword) {
        pw_toggle = "true";
      }

Which looks exactly like the code that should enable the Log in button!

Seems like somebody removed Services.policies and the tests did not cover the email signing functionality (I can imagine automated testing of a PKCS11 module can be quite difficult, if not impossible, so I wouldn't be surprised if the tests werent' there).

Comment 6

[Martin Pecka]

Unfortunately, the error console shows no further hints about why are the references to the signing keys malformed...

Comment 7

[Martin Pecka]

And the error continues to be there even in Nightly...

Comment 8

[Nicolas T]

(In reply to Martin Pecka from comment #5)
> You're right the Log in button is disabled. Thank you for this input, since
> it led me to discover a bit more about this bug.
> 
> If you open the Secrurity devices dialog and click the smartcard entry, this
> error shows up in the error console:
> 
> TypeError: Services.policies is undefined device_manager.js:166:11
> 	enableButtons chrome://pippki/content/device_manager.js:166:11
> 	onselect chrome://pippki/content/device_manager.xul:1:1
> 	onxblmousedown chrome://global/content/bindings/tree.xml:1156:14
> 	openOptionsDialog chrome://messenger/content/mailCore.js:446:7
> 	oncommand chrome://messenger/content/messenger.xul:1:1
> 
> And the problematic lines are:
> 
> if (!Services.policies.isAllowed("createMasterPassword") &&
>           selected_token.isInternalKeyToken &&
>           !selected_token.hasPassword) {
>         pw_toggle = "true";
>       }
> 
> Which looks exactly like the code that should enable the Log in button!
> 
> Seems like somebody removed Services.policies and the tests did not cover
> the email signing functionality (I can imagine automated testing of a PKCS11
> module can be quite difficult, if not impossible, so I wouldn't be surprised
> if the tests werent' there).

I guess you're using v60 now, don't you? Because in 52 branch the Log in button is there, and you can use it to force Thunderbird to authenticate. So, that's not really the problem I think, but could be related.

Sorry for my bad english, It's not my first language.

Comment 9

[Martin Pecka]

Yes, I tested now on 60 and 63 (Nightly).

Comment 10

[Wayne Mery (:wsmwk)]

Have you attempted to find the precise regression date?

Comment 11

[Martin Pecka]

It's been quite long... I reported it 15 months ago, and if I remember correctly, it was few weeks after it started to be buggy...

Comment 12

[Nicolas T]

(In reply to Wayne Mery (:wsmwk) from comment #10)
> Have you attempted to find the precise regression date?

I've been testing and, at least in my case, es working fine in v52.0. I'll keep testing until I find the exact stable version when it started to fail.

Comment 13

[Nicolas T]

Ok, I've been testing all day in two environments. One VM and one physical PC. Same OS (W10 x64 v1803 - 17134.285), same driver version, same Thunderbird versions, same user, same smart card, certificate, etc.

In my VM, I've tested v52.0 until v52.9.1. Today it never failed (yesterday it did), no matter what I did (fresh install or update, local admin user or domain one, with profile present or without, inserted smart card before or after opening Thunderbird, etc).
In my physical PC it was working, until it failed and since then it never worked again.

BUT... I've found what was the problem. Some user preference was created somehow that interfered with the authentication process.

The key was:

user_pref("mail.identity.id1.signing_cert_dbkey", "some_encrypted_value");

I removed that key from prefs.js and everything worked perfectly again, for now. Even in v52.9.1.

Comment 14

[Martin Pecka]

I cannot confirm you findings... Or, it depends on what "everything worked perfectly" means. By deleting that pref, you just remove the signing key. So you cannot sign emails anymore...

I tried resetting this pref and the related ones, quitting TB, then starting again and setting up the signing certificates again, but the bug is still there (now tested on version 60.0).

Comment 15

[Nicolas T]

(In reply to Martin Pecka from comment #14)
> I cannot confirm you findings... Or, it depends on what "everything worked
> perfectly" means. By deleting that pref, you just remove the signing key. So
> you cannot sign emails anymore...
> 
> I tried resetting this pref and the related ones, quitting TB, then starting
> again and setting up the signing certificates again, but the bug is still
> there (now tested on version 60.0).

"Everything worked perfectly" means that I COULD sign emails after that, because the authentication dialog popped up and requested the password. I could close Thunderbird, open it again and it continues to work as it should. Removed the smart card, inserted again, waited for a while to check if the problem came up again, but it never did.

If you close Thunderbird, when you open it again (and before you authenticate) you get the "no nickname" sometimes in some thunderbird versions (in others it doesn't change), but that never affected the correct behaviour in my VM.

That key I mentioned in my previous post was never created again, not even while Thunderbird was opened and after signing an email so I don't know what to tell you. You're saying it should not work, but it's not the case.

I'm not at work anymore, and I couldn't keep testing much after that finding. I intend to keep looking at this tomorrow.

Comment 16

[Martin Pecka]

Ah, ok, that sounds really weird. Did you do any tests with versions 60+?

I'm just in the process of downloading TB source code to be able to debug what's really going on there, because all of this just sounds weird.

I have several certs on the card, one of them is unusable at any time, even right after setting it in the config windows. The other ones work until restart. So it seems the bug is something more subtle...

Comment 17

[Martin Pecka]

So, I got C++ debugging working. Here's so far my little discovery, which might (or might not) be related:

In ncCertPicker::PickByUsage (https://hg.mozilla.org/comm-central/annotate/2a29ee0adb310b54a6a2df72034953fed8f2b043/mailnews/extensions/smime/src/nsCertPicker.cpp#l391), there is a line comparing the nickname of the certificate stored in preferences with the nickname "computed" from each certificate:

    if (nickWithSerial == nsDependentString(selectedNickname)) {

What's interesting is that if I open it in Visual Studio and set a breakpoint, then selectedNickname has value "[e-mail] [1C:AF:98:F4:44:11:55:F7:07:09:DA:6D:79:06:59:49]", whereas nickWithSerial has value "OKsmart 3.0 MiniOs:[e-mail] [1C:AF:98:F4:44:11:55:F7:07:09:DA:6D:79:06:59:49]". Notice the reader name prepended. And, when I try to send an email, after the trial fails, selectedNickname gets shortened to just "[e-mail]".

So the question is whether there wasn't some change to the way nicknames are generated, which would mean that half of TB would expect CN+serial, whereas the other part token+CN+serial... I'll keep investigating further, but if anybody has hints, please, guide me ;)

Comment 18

[Martin Pecka]

Further info from debugging the situation when correct certificate has just been set in account preferences, and I opened a compose window, wrote an email, mark it to be signed, and clicked Send. Now I tested it on 52.9.1 on Linux.

1. Smartcard PIN popup showed, I entered the PIN
2. Everything works correctly as far as GetSigningHashFunction ( https://hg.mozilla.org/comm-central/file/8f03509765da/mailnews/extensions/smime/src/nsMsgComposeSecure.cpp#l323 ). The member mSelfSigningCert (type nsNSSCertificate) contains valid data (the GDB listing at the end of this comment), however, after casting it to nsIX509Cert* and passing to GetSigningHashFunction, calling GetCert() from nsIX509Cert.idl fails, it returns a null pointer. I really don't know why, and here my debugging skills end. The GetCert() implementation should just return a copy of mCert which is there: "return (mCert) ? CERT_DupCertificate(mCert.get()) : nullptr;"

Here are the contents of mSelfSigningCert:



(gdb) p (*(nsNSSCertificate*)mSelfSigningCert.mRawPtr).mCert
$48 = {_M_t = std::tuple containing = {[1] = 0x7f3cd0c8e020, [2] = {<mozilla::UniqueCERTCertificateDeletePolicy> = {<No data fields>}, <No data fields>}}}
(gdb) p *((CERTCertificateStr*)0x7f3cd0c8e020)
$50 = {arena = 0x7f3cd4c500b0, subjectName = 0x7f3cd0c91468 "E=[e-mail],CN=[e-mail]", 
  issuerName = 0x7f3cd0c91490 "CN=StartCom Class 1 Client CA,OU=StartCom Certification Authority,O=StartCom Ltd.,C=IL", signatureWrap = {data = {
      type = siBuffer, 
      data = 0x7f3cd0c90824 "0\202\003֠\003\002\001\002\002\020\034\257\230\364D\021U\367\a\t\332my\006YI0\r\006\t*\206H\206\367\r\001\001\v\005", 
      len = 986}, signatureAlgorithm = {algorithm = {type = siBuffer, data = 0x7f3cd0c90c02 "*\206H\206\367\r\001\001\v\005", len = 9}, parameters = {
        type = siBuffer, data = 0x7f3cd0c90c0b "\005", len = 2}}, signature = {type = siBuffer, 
      data = 0x7f3cd0c90c12 "{*\347\224P\301\301\321\332\317\026\273=\177\a\202\357z\274\016\t\aO\205\266h\005l\300\245@\330\nkN\364", <incomplete sequence \303>, len = 2048}}, derCert = {type = siBuffer, 
    data = 0x7f3cd0c90820 "0\202\004\356\060\202\003֠\003\002\001\002\002\020\034\257\230\364D\021U\367\a\t\332my\006YI0\r\006\t*\206H\206\367\r\001\001\v\005", len = 1266}, derIssuer = {type = siBuffer, 
    data = 0x7f3cd0c9084e "0u1\v0\t\006\003U\004\006\023\002IL1\026\060\024\006\003U\004\n\023\rStartCom Ltd.1)0'\006\003U\004\v\023 StartCom Certification Authority1#0!\006\003U\004\003\023\032StartCom Class 1 Client CA0\036\027\r160922202657Z\027\r191222202657Z0:1\030\060\026\006\003U\004\003\f\[e-mail]1\036\060\034\006\t*\206H\206\367\r\001\t\001\026\017peci"..., len = 119}, derSubject = {type = siBuffer, 
    data = 0x7f3cd0c908e5 "0:1\030\060\026\006\003U\004\003\f\[e-mail]1\036\060\034\006\t*\206H\206\367\r\001\t\001\026\[e-mail]0\202\001\"0\r\006\t*\206H\206\367\r\001\001\001\005", len = 60}, derPublicKey = {type = siBuffer, 
    data = 0x7f3cd0c90921 "0\202\001\"0\r\006\t*\206H\206\367\r\001\001\001\005", len = 294}, certKey = {type = siBuffer, 
    data = 0x7f3cd0c91348 "\034\257\230\364D\021U\367\a\t\332my\006YI0u1\v0\t\006\003U\004\006\023\002IL1\026\060\024\006\003U\004\n\023\rStartCom Ltd.1)0'\006\003U\004\v\023 StartCom Certification Authority1#0!\006\003U\004\003\023\032StartCom Class 1 Client CA\[e-mail]", len = 135}, version = {
    type = siBuffer, data = 0x7f3cd0c9082c "\002\002\020\034\257\230\364D\021U\367\a\t\332my\006YI0\r\006\t*\206H\206\367\r\001\001\v\005", len = 1}, 
  serialNumber = {type = siBuffer, data = 0x7f3cd0c9082f "\034\257\230\364D\021U\367\a\t\332my\006YI0\r\006\t*\206H\206\367\r\001\001\v\005", len = 16}, 
  signature = {algorithm = {type = siBuffer, data = 0x7f3cd0c90843 "*\206H\206\367\r\001\001\v\005", len = 9}, parameters = {type = siBuffer, 
      data = 0x7f3cd0c9084c "\005", len = 2}}, issuer = {arena = 0x0, rdns = 0x7f3cd0c90d18}, validity = {arena = 0x0, notBefore = {type = siUTCTime, 
      data = 0x7f3cd0c908c9 "160922202657Z\027\r191222202657Z0:1\030\060\026\006\003U\004\003\f\[e-mail]1\036\060\034\006\t*\206H\206\367\r\001\t\001\026\[e-mail]0\202\001\"0\r\006\t*\206H\206\367\r\001\001\001\005", len = 13}, notAfter = {type = siUTCTime, 
      data = 0x7f3cd0c908d8 "191222202657Z0:1\030\060\026\006\003U\004\003\f\[e-mail]1\036\060\034\006\t*\206H\206\367\r\001\t\001\026\[e-mail]0\202\001\"0\r\006\t*\206H\206\367\r\001\001\001\005", len = 13}}, subject = {arena = 0x0, rdns = 0x7f3cd0c90e60}, subjectPublicKeyInfo = {arena = 0x0, 
    algorithm = {algorithm = {type = siBuffer, data = 0x7f3cd0c90929 "*\206H\206\367\r\001\001\001\005", len = 9}, parameters = {type = siBuffer, 
        data = 0x7f3cd0c90932 "\005", len = 2}}, subjectPublicKey = {type = siBuffer, data = 0x7f3cd0c90939 "0\202\001\n\002\202\001\001", len = 2160}}, 
  issuerID = {type = siBuffer, data = 0x0, len = 0}, subjectID = {type = siBuffer, data = 0x0, len = 0}, extensions = 0x7f3cd0c91020, 
  emailAddr = 0x7f3cd0c913d0 "[e-mail]", dbhandle = 0x7f3d03c4a030, subjectKeyID = {type = siBuffer, 
    data = 0x7f3cd0c913f8 "m\254\067a\004\370,\005\212v\376%\036\006N\274\350\035\357E\345\345\345", <incomplete sequence \345>, len = 20}, 
  keyIDGenerated = 0, keyUsage = 176, rawKeyUsage = 176, keyUsagePresent = 1, nsCertType = 160, keepSession = 0, timeOK = 0, domainOK = 0x0, isperm = 1, 
  istemp = 0, nickname = 0x0, dbnickname = 0x0, nssCertificate = 0x7f3cd0c8f0c0, trust = 0x7f3cd0c914e8, referenceCount = 1, subjectList = 0x0, 
  authKeyID = 0x7f3cd0c91410, isRoot = 0, options = {apointer = 0x0, bits = {hasUnsupportedCriticalExt = 0}}, series = 2, slot = 0x7f3d03f0f800, 
  pkcs11ID = 3103280928, ownSlot = 1}

Comment 19

[Nicolas T]

Your discoveries looks like it could be related with a bug ticket that was opened for more than ten years, releated to that matter.

Bug 278689 - Multiple Certificates with the same subject are not shown in the digital signature select cert combo (only one is shown)"
https://bugzilla.mozilla.org/show_bug.cgi?format=default&id=278689

Comment 20

[Martin Pecka]

Digging deeper: pipnss log:

[Main Thread]: D/pipnss nsCMSMessage::CreateSigned
[Main Thread]: D/pipnss nsCMSEncoder::Start
[Main Thread]: D/pipnss nsCMSEncoder::Finish
[Main Thread]: D/pipnss nsCMSEncoder::Finish - can't finish encoder

p7ecx->error ( https://hg.mozilla.org/releases/mozilla-esr52/annotate/dd227854175f/security/nss/lib/smime/cmsencode.c#l756 ) is set to SEC_ERROR_NO_KEY.

Comment 21

[Nicolas T]

I was testing all day, but I'm not a programmer really so I don't have your debuggin skills, sorry.
My case is not exactly the same as yours, as we use only ONE certificate for Smart Card, and Windows.

I din't test v60 so much because we have almost three thousand computers running Windows XP still, and those are not supported anymore.

I was able to reproduce the bug every time by doing:
- Loaded PKCS module (then the dbkey preference was created).
- Selected certificate in account security.
- Closed Thunderbird, and opened it again.
- Tried to send a signed email. It fails.

If I opened pref.js and manually deleted the mail.identity.id1.signing_cert_dbkey line, the functionality was restored. No matter if I closed Thunderbird, changed USB port, or whatever I tried. As long I started Thunderbird with the smart card plugged, of course.

If you remove the module, and then add it again, it fails because that dbkey preference is created again.

The strange thing is, my vm NEVER failed, even with the dbkey preference present.

Comment 22

[Martin Pecka]

As I continue to dig deeper, it starts to make sense to me.

The reason why there are two preferences (signing_cert_name and signing_cert_dbkey) is that before https://bugzilla.mozilla.org/show_bug.cgi?format=default&id=278689 TB only identified certificates by the (nick)name which was stored in signing_cert_name. However, poeple found valid cases where you have multiple keys with the same name and you have to differentiate between them e.g. based on the intended usage of the key, its serial number or the issuer of the certificate. These information are mangled together into a dbkey which is a unique identifier of the certificate. To retain backwards compatibility, the logic is exactly as you observe - if pref signing_cert_dbkey is found, search for the certificate is done by searching for its dbkey (that's apparently what's broken now). But if the dbkey pref doesn't exist, TB just tries to match the nickname of the certificate with the name stored in signing_cert_name. And that apparently works.

In the VM, how do you access the reader? Do you have some kind of "passthrough" of the real reader, or do you use a virtual reader there?

I also discovered a related bug which was the cause of a part of my problems: https://bugzilla.mozilla.org/show_bug.cgi?id=1296325 . It basically says that if you have signing keys set up and send an email to yourself (my favorite way of testing :) ), all further signing attempts fail (because TB stores your signing certificate to a list of "recipients' cert store", but without the private key, and then if you try to sign, it mistakenly first finds this copy of the certificate which is unusable for signing). In my case, it even picked up some of my old (now invalid) certificates.

Comment 23

[Nicolas T]

My smart card is a USB token (like a pendrive), and you only need to install the client software to use it. So, I passthrough the USB device to the VM (VirtualBox) and install the client software there.

I was trying to decode the dbkey value, and the part that was readable was indeed the issuer of the certificate.

In some of my tests I couldn't even send a signed mail once after the dbkey was created in the same session.

And the most strange thing happened only once and I couldn't reproduce it. Another preference was created somewhere (not in user\appdata\roaming) that constantly overwrited my pref.js. The key was:

- mail.identity.default.signing_cert_name

With that preference present, no matter if I didn't choose a certificate, it was set automatically everytime. It makes sense, judging by the preference name, but it wasn't intended and I can't tell how that happened.

Comment 24

[Nicolas T]

I woke up today thinking why dbkey is invalid in my case. Maybe it's because we are spanish people, and the certificate issuer (which is a great part of the base64 value of the key) has a few acute accented letters.

In my case the certificate issuer is:

CN = Autoridad Certificante de Firma Digital
SERIALNUMBER = CUIT 30680604572
O = Jefatura de Gabinete de Ministros
OU = Oficina Nacional de Tecnologías de Información
OU = Secretaría de Gestión Pública
OU = Subsecretaría de Tecnologías de Gestión
S = Ciudad Autónoma de Buenos Aires
C = AR

So, my issuerName should be something like:

issuerName = "CN=Autoridad Certificante de Firma Digital,OU=CUIT 30680604572,O=Jefatura de Gabinete de Ministros,OU=Oficina Nacional de Tecnologías de Información,OU=Secretaría de Gestión Pública,OU=Subsecretaría de Tecnologías de Gestión,S=Ciudad Autónoma de Buenos Aires,C=AR"


Or maybe it's too long?

Comment 25

[Nicolas T]

I guess I'll open another bug ticket, because my case is related but not identical to yours, as I've checked and my bug it's there since they added the Bug 278689 patch in v44. Or maybe I'll ask to reopen that one, if it's still possible.

Comment 26

[Kai Engert [:KaiE:]]

It's generally difficult for me to reproduce and analyze smartcard bugs, because I don't have the devices that you use. So it's difficult to say if the issue is caused by the smartcard device, the driver being used, or if it the problem is related to specific properties of the certificates you are using.

I suggest that you try to help to eliminate the possibility that the smartcard or its driver software are the cause of the problem.

Ideally, please try to reproduce the issue without using a smartcard. In other words, by using a certificate that is stored inside the profile directory (NSS software security device).

If the problem doesn't occur without a smartcard, then please check if the issue can be reproduced using a virtual smartcard. The softhsm virtual smartcard is available on Linux, and I'd prefer that: https://www.opendnssec.org/softhsm/

If you can reproduce using softhsm, and if you can provide instructions how you setup the certificate, then I could try to reproduce and debug.

Comment 27

[chico]

(In reply to Kai Engert (:kaie:) from comment #26)

It's generally difficult for me to reproduce and analyze smartcard bugs, because I don't have the devices that you use. So it's difficult to say if the issue is caused by the smartcard device, the driver being used, or if it the problem is related to specific properties of the certificates you are using.

I suggest that you try to help to eliminate the possibility that the smartcard or its driver software are the cause of the problem.

Ideally, please try to reproduce the issue without using a smartcard. In other words, by using a certificate that is stored inside the profile directory (NSS software security device).

If the problem doesn't occur without a smartcard, then please check if the issue can be reproduced using a virtual smartcard. The softhsm virtual smartcard is available on Linux, and I'd prefer that: https://www.opendnssec.org/softhsm/

If you can reproduce using softhsm, and if you can provide instructions how you setup the certificate, then I could try to reproduce and debug.

The "Login" box bug in Device Manager (Security Devices) is reproducible with ANY pkcs11 device. The error message is what Mr. Martin Pecka found, whenever a PKCS11 smartcard device entry is selected inside the Security Devices dialog box:
TypeError: Services.policies is undefined[Learn More] device_manager.js:166:11
enableButtons chrome://pippki/content/device_manager.js:166:11
onselect chrome://pippki/content/device_manager.xul:1:1
onxblmousedown chrome://global/content/bindings/tree.xml:1156:14
openOptionsDialog chrome://messenger/content/mailCore.js:446:7
oncommand chrome://messenger/content/messenger.xul:1:1

Previously in V52, I could click on the "Login" button, when I select the right smartcard device, but now it is always grayed out due to this bug.

I am using Thunderbird v60.8 stable.

Comment 28

[Martin Pecka]

Wow... This also happens with the key stored in Software security device... I'll attach the certificate I'm using. I won't however send the whole p12 file with the private key as I'm currently using the key I used for this test.

Comment 29

[Martin Pecka]

Attachment: Another problematic certificate - this one has problems even in Soft.Sec.Dev.

Comment 30

[Martin Pecka]

Okay, I also created a bogus self-signed cert for testing, the same results. I'll share it here.

Comment 31

[Martin Pecka]

Attachment: Self-signed cert for testing. Password is 0000.

Created following the guide at https://gist.github.com/richieforeman/3166387 .

Comment 32

[Kai Engert [:KaiE:]]

Martin, the certificate you attached in comment 29 has been issued by the StartCom CA. Mozilla has revoked all trust on the StartCom CA, that's why your certificate is no longer considered valid by Mozilla software.

The bogus self-signed cert from comment 30 doesn't follow recommendations for certs. For example, it doesn't even have a "Certificate Basic Constraints" extension which usually marks certs as either a CA or as not a CA, so I'm not surprised this certificate is rejected.

Comment 33

[Martin Pecka]

The problem is not in validity of the certificates. Both of them appear as options when selecting a signing cert in identity manager. Right after the selection, the "readonly" field showing the selected signing cert is showing some value, but after I try to send an email, it displays the "no nickname" value or something similarly wrong.

If you tell me a way how to generate a proper self-signed cert, I'll try. But I'm sure this was happenning earlier than Mozilla removed the trusted root of Startcom.

Comment 34

[Kai Engert [:KaiE:]]

Thanks for explaining again what your bug is.

Unfortunately I cannot reproduce the bug, and I have never seen it with the certificates I'm using.

I set up a new profile, and I've tried with both TB 60.8 and 68.0

I set my email to address to the addressed listed in your attached self signed certificate. I configured account security to use it for s/mime. I confirm. I go back to the dialog, and the select is still correct, NOT no nickname.

If you have reproduced the bug yourself using that certificate, could you please describe how you reproduce it using a fresh profile and that certificate?

Comment 35

[Martin Pecka]

Attachment: Screenshots of the steps on how to reproduce this bug on a clean profile.

I tried to replicate the bug on a brand new profile in TB 60.8 (Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0). Windows 10 1809 build. It's still there.

Comment 36

[Martin Pecka]

Kai, Could you try reproducing once more, now with an attempt to send a signed email after setting up the certs? As you can see from my screenshots, after the sending fails, the cert selection field loses track of the certificate's serial.

Comment 37

[Martin Pecka]

I've just replicated the bug on a fresh install of nightly to a completely new location with a fresh profile. So if there are some differences in the outcome between me and other people, they should be definitely given by the system properties and not by some leftovers in my TB folders (I hope).

Comment 38

[Martin Pecka]

One more test now - it happens even with earlybird 31, which is weird. I'm pretty sure that at the time I was using TB 31, cert signing was working. Might it be a problem of the signing certificate? But I remember that when this bug started for me, it immediately started happening for all identities (I have about 4 with different signing certs and email addresses). So it shouldn't be caused just by exchanging my old certificate for a new one. Much more it seems to be caused by some system-related thing that changed in the meantime.

Comment 39

[Martin Pecka]

Now I tried on Linux (on the same machine), and the result is still the same! The bug is there, even if the cert is stored in software security device...

Comment 40

[charlyfeck]

Hi everyone, I'm following this thread since a year ago. I'm using the same token (Athena ID Protect) and CA reported by Nicolas T. The problem is that every time I want to sign in Thunderbird I have to associate again the cert in the configuration of the account. That was the situation in Thunderbird 60.2.1. Downgrading to version Thunderbird 50.b2 I was able to sign without problems.

Now I'm triying to reproduce the scenario and the behavior is different. Testing with:
Ubuntu bionic 64 bits
Thunderbird 60.8.0 (64-bits)
Token Athena ID Protect

Sometimes when I sign it fails after asking for the token password showing the message:
"Sending of the message failed.
Unable to sign message. Please check that the certificates specified in Mail & Newsgroups Account Settings for this mail account are valid and trusted for mail."
But If I hit the button send again the message is dilivered correctly. Other times the message is send correctly in the first attemp. I tried conecting the device before and after starting Thunderbird, but that is not the reason of the diffent behaviors.

Comment 41

[Kai Engert [:KaiE:]]

Thanks for the screenshots. I can reproduce what happens with the test certificate. I don't know if this explains the failures you saw with other certificates.

The attached "self signed certificate" cannot be used with Thunderbird, because they aren't supported. Only certificates that are issued from a CA are supported.

The reason is, only certificates that can be verified as "valid" by Thunderbird are allowed. But Thunderbird doesn't allow to set explicit trust for a self-signed certificate.

If you create a test CA certificate, then use that CA to issue a test certificate, import both into Thunderbird, and mark the CA cert as valid for email, then you can use your own certificate.

At the time you try to send a signed email, we execute code that verifies the certificate. If it cannot be validated, we clear the preference. That explains why you see the signing_cert_dbkey being erased.

The same can happen with any certificate that is no longer valid. For example, if your certificate was previously valid, but then expired, it is no longer valid, and you'd get the same experience.

There's only scenario we could accept this as a bug report:
If you're trying to use a certificate that is still valid, but Thunderbird still complains about it and clears the pref.

Comment 42

[Kai Engert [:KaiE:]]

None of the attached certificates is valid. The first one is expired. The second one is issued by Startcom, which is no longer trusted. The third one is self-signed and cannot be marked as trusted in Thunderbird.

Comment 43

[Kai Engert [:KaiE:]]

You can check if Thunderbird considers your certificate valid. Open certificate manager, switch to the your certificates tab, find your cert, click view. The top area shows information if your cert seems valid/verified or not.

I'm closing this bug for now. Please feel free to attach more information. If you see a problem with a valid certificate, then we can reopen.

Comment 44

[Martin Pecka]

Thanks for the info, I'll test later with a better cert. However, shouldn't the same validity check be done at the time the user is selecting the certificate? I know that the checks are needed every time you send an email (because the cert may become invalid over time), but I'm strongly for having the same check applied to the certs that show up in the cert selection dialog.

Comment 45

[Nicolas T]

(In reply to Kai Engert (:kaie:) from comment #43)

You can check if Thunderbird considers your certificate valid. Open certificate manager, switch to the your certificates tab, find your cert, click view. The top area shows information if your cert seems valid/verified or not.

I'm closing this bug for now. Please feel free to attach more information. If you see a problem with a valid certificate, then we can reopen.

Kai Engert, what about pkcs devices scenario?

Now, on v68.0, happends exactly what "charlyfeck" described at comment 40.

You've changed something in all this months.

Before:
"mail.identity.id1.signing_cert_name" = "FIRMA DIGITAL:le-aea9c492-1629-447a-911a-035e2d33e370" (Security Device label + ":" + default container).
Account > Security > Selected certificate = the same value as signing_cert_name.

Now:
"mail.identity.id1.signing_cert_name" = "My full name" (CN)
Account > Security > Selected certificate = CN + "[" + Certificate's Serial Number + "]".

Before, I could fully restore functionality by manually deleting dbkey preference.
Now, it's not longer possible, because if that preference is removed, it will always fail at sending attempt. So I guess you've removed the fallback mechanism that was present before.

Every time you open up Thunderbird, the first digitally signed mail sending attempt will fail. If you retry it will be send, though. And it won't fail again until you close thunderbird.

Comment 46

[Kai Engert [:KaiE:]]

If there's an additional scenario for the described failure, which happens with valid certificates, then we need to identify how it can be reproduced. I don't know how to reproduce it yet.

Comment 47

[Kai Engert [:KaiE:]]

(In reply to Nicolas T from comment #45)

You've changed something in all this months.

The change I'm aware of is bug 1319185, which affects Thunderbird 53 and later.

Because Firefox had removed some core functionality, the TB maintainers were forced to adopt. The way it was changed in TB wasn't ideal IMO because the preference name was kept, but its contents were changed. Now we have to live with that change.

Before:
"mail.identity.id1.signing_cert_name" = "FIRMA DIGITAL:le-aea9c492-1629-447a-911a-035e2d33e370" (Security Device label + ":" + default container).
Account > Security > Selected certificate = the same value as signing_cert_name.

Now:
"mail.identity.id1.signing_cert_name" = "My full name" (CN)
Account > Security > Selected certificate = CN + "[" + Certificate's Serial Number + "]".

Right, instead of the nickname, now the display name is shown:
https://hg.mozilla.org/releases/comm-esr68/rev/e47002b53b5f9294a331d8b84e291ed6c362fb01#l1.83

Before, I could fully restore functionality by manually deleting dbkey preference.

You shouldn't tweak prefs.js manually. Could you use the user interface to re-select the correct certificate, which would then hopefully set the corect preference values to remember the correct cert?

Comment 48

[Nicolas T]

(In reply to Kai Engert (:kaie:) from comment #47)

You shouldn't tweak prefs.js manually. Could you use the user interface to re-select the correct certificate, which would then hopefully set the corect preference values to remember the correct cert?

I work for a State Court, and back then they needed to implement some backup mechanism with digitally signed emails when the primary system fails. I coudn't wait for a proper fix so, I created a simple vbs script to automate the process.
My certificate is valid, but it will expire in a few days. I don't know if they will give me a new one, because they only gave it to me for this specific purpose.

Now, I wanted to inform you that you've almost fixed the bug in v68.0. The only problem that remains is that error at first sending attempt, but no preference is cleared or lost. If you retry, it works as it should and will never fail again until you close Thunderbird.

Comment 49

[Kai Engert [:KaiE:]]

(In reply to Nicolas T from comment #48)

Now, I wanted to inform you that you've almost fixed the bug in v68.0. The only problem that remains is that error at first sending attempt, but no preference is cleared or lost. If you retry, it works as it should and will never fail again until you close Thunderbird.

If only we knew why!

Comment 50

[Martin Pecka]

(In reply to Kai Engert (:kaie:) from comment #42)

None of the attached certificates is valid. The first one is expired. The second one is issued by Startcom, which is no longer trusted. The third one is self-signed and cannot be marked as trusted in Thunderbird.

You're right! I probably haven't noticed Mozilla stopped trusting Comodo and Startcom. But now you see how impossible for the user is to "debug" this kind of issue. Maybe the culprit of this bug may be therefore marked as invalid, but I think it triggers a few more "UX" bugs. Should I report them?

I've got another address for which I have a trusted cert, and with that one, this bug doesn't happen. I also verified with a self-signed cert for which I added the CA to the trusted list. Later, I'll test with a cert on smartcard to see if this is finally resolved for all my use-cases.

Comment 51

[Martin Pecka]

Tested with a cert on SmartCard. It basically seems to be working, too. If I have the SC in, choose a valid cert, I can send a signed email as expected. If I remove the card and try to send a signed email, TB tells me it cannot find the cert and the serial number in signing options is lost, but CN stays. If I re-insert the card again, TB correctly finds the previously selected cert and uses it. I think this is how it should work ;)