990958 - Incompatibility with gpgsm

Status: RESOLVED INACTIVE

(NSS :: Libraries, defect)

Description

[Ed Finnerty]

Attachment: test_nss.sh

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0 (Beta/Release)
Build ID: 20140315085348

Steps to reproduce:

Generated a free S/MIME certificate from COMODO, imported it into Thunderbird and gpgsm, then ran the attached script (please edit it for your user and Thunderbird profile). The script basically encrypts random data with gpgsm and tries to decrypt it with cmsutil in a loop.

I've tried this with both NSS 3.16, and the source code from the Mercurial repository, and reproduced the issue on both Linux and FreeBSD.

My gpgsm:

$ gpgsm --version
gpgsm (GnuPG) 2.0.22
libgcrypt 1.5.3
libksba 1.3.0
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: ~/.gnupg
Supported algorithms:
Cipher: 3DES, AES, AES192, AES256, SERPENT128, SERPENT192, SERPENT256, SEED, CAMELLIA128, CAMELLIA192, CAMELLIA256
Pubkey: RSA, ECDSA
Hash: MD5, SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224, WHIRLPOOL


Actual results:

Eventually cmsutil fails:

NSS has been initialized.
Got default certdb
cmsutil: failed to decode message.
cmsutil: problem decoding: SEC_ERROR_BAD_DATABASE: security library: bad database.


Expected results:

The loop should have gone on forever (cmsutil should have never failed to decrypt gpgsm's output).

Comment 1

[Ed Finnerty]

I've contacted the GnuPG team, and they've said: "I checked your scenario. It looks like it is a bug in the NSS, not in gpgsm. openssl smime can correctly parse and decode the messages which cause cmsutil to return an error."

So according to them, the problem is in NSS.

Should I assume by the complete lack of replies that compatibility with gpgsm is not a goal at all for NSS?

Comment 2

[Eric Rescorla (:ekr)]

(In reply to Ed Finnerty from comment #1)
> I've contacted the GnuPG team, and they've said: "I checked your scenario.
> It looks like it is a bug in the NSS, not in gpgsm. openssl smime can
> correctly parse and decode the messages which cause cmsutil to return an
> error."
> 
> So according to them, the problem is in NSS.
> 
> Should I assume by the complete lack of replies that compatibility with
> gpgsm is not a goal at all for NSS?

I'm not an expert on this part of the code, but based on the error
you are reporting, this doesn't look like a problem with the S/MIME
code but rather a result of initializing and tearing down NSS
repeatedly. Questions:

1. How many times did you make it through the loop.
2. If you take the same message and send it as the first message
in the loop does it work?

Comment 3

[Ed Finnerty]

There's never a fixed ammount of steps in the loop before the error occurs. There have been times when it only took a second, and times when it took about 10 or 20 seconds - that's a lot of steps in the loop.

I've actually come across this when trying to read S/MIME messages encrypted with gpgsm in Thunderbird - once in many messages there's a broken, unreadable one. It's always the same one (maybe this is what you were asking in question 2), even after restarting Thunderbird or moving to a readable one and back, so it wouldn't seem to be exclusively an initialization problem.

Comment 4

[Eric Rescorla (:ekr)]

Can you please try the experiment I proposed in #2? I.e., dump the
message in question and verify that:

1. It is unreadable.
2. Another message is?

If that is true, then please submit the broken message and the
corresponding cert db.

Comment 5

[Ed Finnerty]

I've added a loop counter to the script and ran it 5 times. I've got 72, 58, 77, 111 and 301 steps before the error happened.

I don't have the Thunderbird messages anymore.

I'd be happy to try the experiment if it's possible with my script, but I'm not sure I understand what you're asking. The script leaves two files on disk on error: input.bin and out.bin. Out.bin is the file encrypted with gpgsm and it can be decrypted just fine with gpgsm:

$ gpgsm -d out.bin > input2.bin 
gpgsm: enabled debug flags: assuan
gpgsm: DBG: recp 0 - issuer: `CN=COMODO Client Authentication and Secure Email CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB'
gpgsm: DBG: recp 0 - serial: XXXX
gpgsm: DBG: connection to agent established
secmem usage: 0/16384 bytes in 0 blocks

$ cmp input2.bin input.bin 
$

Comment 6

[Eric Rescorla (:ekr)]

What I am trying to establish is that the problem is this particular message, not
the state of NSS.

Your script runs in a loop and produces a pile of messages, some of which can
be decoded and some of which cannot. What I'm looking for is for you to run it
and then after it has failed, verify that NSS can still decode a message which
previously worked.

Comment 7

[Ed Finnerty]

It's not the state of NSS:

$ cmsutil -D -d ~/.thunderbird/dir -i out437.bin -v -n
received commands
NSS has been initialized.
Got default certdb
cmsutil: failed to decode message.
cmsutil: problem decoding: SEC_ERROR_BAD_DATABASE: security library: bad database.

$ cmsutil -D -d ~/.thunderbird/dir -i out436.bin -v -n
received commands
NSS has been initialized.
Got default certdb

$ cmsutil -D -d ~/.thunderbird/dir -i out437.bin -v -n
received commands
NSS has been initialized.
Got default certdb
cmsutil: failed to decode message.
cmsutil: problem decoding: SEC_ERROR_BAD_DATABASE: security library: bad database.

$ cmsutil -D -d ~/.thunderbird/dir -i out436.bin -v -n
received commands
NSS has been initialized.
Got default certdb

Unfortunately I can't just submit my certs, as I'm using my real ones from Thunderbird, that's why I've written the script so anyone can test this without having to share private information.

Comment 8

[Wayne Mery (:wsmwk)]

FWIW, Jan 2019 per reporter "I'm not sure the problem still exists, I've moved on from that project and haven't checked in a while.... Daiki Ueno claims that it's potentially a problem in libksba: https://lists.gnupg.org/pipermail/gnupg-devel/2016-February.txt "

Comment 9

[Benjamin Beurdouche [:beurdouche]]

Since we can't need-info the author and have no other reports, I'll close as inactive for now, this can always be reopened if needed... : )