202384 - mozilla can't read encrypted mail from Lotus Notes

Status: RESOLVED FIXED

(NSS :: Libraries, defect, P1)

Description

[Nelson Bolyard (seldom reads bugmail)]

A certain Netscape user regularly receives encrypted SMIME email from 
his coworkers who are users of Lotus Notes.  He can read these encrypted
emails with no problems using Communicator 4.7x on Windows or Linux.
But using newer Netscape and mozilla browsers, he cannot read these 
encrypted emails.  Mozilla reports: 
"There are unknown problems with this encrypted message."  (sigh.)

When he attempts to read them using NSS's cmsutil test program, it reports:  
"cmsutil: problem decoding: security library: improperly formatted
DER-encoded message."

He compared encrypted messages generated by several SMIME email programs
and found a crucial difference between the messages generated by Lotus
Notes and those generated by other programs.  The RecipientInfos SET 
in the Enveloped Data type is indefinite-length encoded by Notes and 
definite length encoded by others.  He altered one message from Notes,
changing the RecipientInfos SET to definite length encoding and then 
mozilla was able to read the message.  He sent me both the original and
altered copies for me to test with.  I will attach them to this bug.

RecipientInfos is permitted to be indefinite length encoded.  
The decoding failure in NSS's fault.  It is a regression in libSMIME 
as compared to the old libPKCS7.  

I have been debugging his unaltered message using cmsutil in NSS 3.8 
and have found several problems.  
The RecipientInfos SET contains a single RecipientInfo.  The decoder
decodes that correctly, using NSSCMSRecipientInfoTemplate, then 
returns to the parent state.  It sees that it is parsing a SET, so
it goes to parse the next NSSCMSRecipientInfoTemplate, and then 
discovers the EndOfContents bytes.  But by that time, it thinks it's
parsing the CHOICEs inside the NSSCMSRecipientInfoTemplate, and it 
concludes that it has found no item that matches any of those CHOICEs.
I conclude that the EndOfContents bytes are being found at the wrong 
point in the decoding, but I'm not sure yet how to do it better.

In the old PKCS7 library, the RecipientInfoTemplate was not a CHOICE.
Rather, it was very similar to the NSSCMSKeyTransRecipientInfoTemplate.
I suspect it is the new combination of a SET of a CHOICE that causes 
the problem.

Comment 1

[Nelson Bolyard (seldom reads bugmail)]

I asked the reporter to detemine which Netscape/mozilla clients can, and 
which cannot, read encrypted emails from Lotus notes.  Here is his answer:

> Netscape 4.79 Linux               success
> Netscape 7.0 Linux                fails
> Netscape 7.0 Windows              fails
> Mozilla 1.0.1 Linux               fails
> Mozilla 1.3 Linux                 fails
> NSS 3.4.1 Linux                   fails
> Outlook Express 6.0               success
> openssl 0.9.6 Linux               success

Comment 2

[Nelson Bolyard (seldom reads bugmail)]

This bug appears to now be fixed in NSS by revisions 1.20-1.22 of file
nss/lib/util/secasn1d.c.  
Don't know how soon the mozilla browser will pick this up though.

I'm tentatively marking this fixed.  
However, testing of the ASN.1 BER decoder continues.

Comment 3

[Wan-Teh Chang]

Attachment: Patch for NSS_3_8_BRANCH

Comment 4

[Wan-Teh Chang]

Comment on attachment 121959 [details] [diff] [review]
Patch for NSS_3_8_BRANCH

Bob, could you review Nelson's fix for this bug?

This patch is the diffs between rev. 1.21 and rev. 1.23
adapted for the NSS_3_8_BRANCH.  (We can't apply the diffs
from the trunk to the NSS_3_8_BRANCH directly because
Nelson also made some white space changes on the trunk to
fix the coding style.)

Comment 5

[Robert Relyea]

Comment on attachment 121959 [details] [diff] [review]
Patch for NSS_3_8_BRANCH

As much as I can understand give that the ASN1 decoder is black magic;).

Comment 6

[Nelson Bolyard (seldom reads bugmail)]

Attachment: Updated patch to NSS 3.8 branch

This patch incorporates the changes from revisions 1.22-1.24 on the trunk.

Comment 7

[Nelson Bolyard (seldom reads bugmail)]

Wan-Teh,  If you agree, I'll check this patch into the 3.8 branch.

Comment 8

[Wan-Teh Chang]

Comment on attachment 122538 [details] [diff] [review]
Updated patch to NSS 3.8 branch

Nelson, you can check in this patch on the 3.8 branch.
The only change I would suggest is to use the old 2-space
indentation level in the "hunk" of your patch at
-2187,6 +2205,26.

Comment 9

[Nelson Bolyard (seldom reads bugmail)]

Comment on attachment 122538 [details] [diff] [review]
Updated patch to NSS 3.8 branch

Requesting review.  I hope to get this in mozilla 1.4b

Comment 10

[Nelson Bolyard (seldom reads bugmail)]

Comment on attachment 122538 [details] [diff] [review]
Updated patch to NSS 3.8 branch

Requesting approval for Moz 1.4b.  Note, SR not needed for NSS code.

Comment 11

[(not reading, please use [e-mail] instead)]

Comment on attachment 122538 [details] [diff] [review]
Updated patch to NSS 3.8 branch

a=sspitzer

so you are just going to back port this fix from the nss trunk to the nss 3.8
branch?

Comment 12

[Nelson Bolyard (seldom reads bugmail)]

Re comment 11, yes, that's exactly what patch attachment 122538 [details] [diff] [review] is, 
a back port from the trunk to the branch.

Comment 13

[Nelson Bolyard (seldom reads bugmail)]

Moved NSS_CLIENT_TAG for nss/lib/util/secasn1d.c from rev 1.19 to rev 1.19.26.1
for Mozilla 1.4

Comment 14

[Wan-Teh Chang]

Comment on attachment 122538 [details] [diff] [review]
Updated patch to NSS 3.8 branch

Nelson, could you check in this patch on the NSS_3_7_BRANCH?

Comment 15

[Nelson Bolyard (seldom reads bugmail)]

Done.  Patch applied to 3.7 branch.