107034 - OE requires special attribute in incoming signed messages to support dual key certificates

Status: VERIFIED FIXED

(MailNews Core :: Security: S/MIME, defect, P1)

Description

[Antonio Lam]

Build Used: 2001-10-24-OTIS commercial build.

If the sender use the dual-key certs (one for signing and one for encrypting) to 
sign a message, Outlook Express (v. 5.5) failed to store the encrypting cert 
included in the signed message to the local certdb.

Step to reproduce:

1. Use dual-key certs to sign a message to a recipient using the OTIS build.
2. In Outlook Express, open and read that signed message
3. In Outlook Express, go to Tools|Options|Security, Click on "Digital IDs", and 
click on "Other People" tab

Actual result:
The Sender's encrypting cert is not in the list. Try to sending back an 
encrypting msg to the sender will failed. (Complaining no digital ID for the 
sender).

Expected result:
The Sender's encrypting cert should be stored in the list, and the recipient 
should be able to send back an encrypted msg to the sender.


Remarks:

1. If the sender use only one certifcate (dual-Purpose) to sign the message, 
everyting worked fine in OE.

2. It worked fine if we use N6.2 build to read the signed message. (encrypting 
cert is stored in the local certdb)

3. If I use Outlook Express with the same dual-key certs to send a signed msg, 
Outlook Express can also store the encrypting cert, as well as the signing cert 
(it could be a bug in OE).

Comment 1

[Stephane Saux]

S/MIME bugs are automatically nsbeta1 candidates. (this is a bulk update - there
may be some adjustment of the list).

Comment 2

[Stephane Saux]

I don't see how we're going to solve the problem if the issue is with OE. It's
possible that we don't package the cert correctly.

cc thayes

needs qa:
Does OE handle signed emails generated from communicator which dual key certs
correctly?  May need PSM 1.X

Kai

Comment 3

[Kai Engert (:KaiE:)]

This bug was filed 5 months ago, please test, whether the bug as reported still
is reproducable.

If it is, please test what happens, when you use OE itself with dual key
certificates. Please create a mail account with OE, import a dual-key-cert from
a p12 file, and configure everything. Send signed mail to a different OE account
on a different profile/machine, and check whether that actually works.

Either we'll learn, that OE does not support dual-key-certs at all, which would
make this bug invalid.

Or, we learn that it works with OE. In that case, I'm interested in the raw
message that was produced by OE and sent to the other machine, it will be a
reference what we have to do to make it work.

Thanks.

Comment 4

[Antonio Lam]

Re-test it using 2002031303 trunk build, and the bug is still reproducible.

Here is my comments:

>If it is, please test what happens, when you use OE itself with dual key
>certificates. Please create a mail account with OE, import a dual-key-cert
>from a p12 file, and configure everything. Send signed mail to a different OE 
>account on a different profile/machine, and check whether that actually works.

Yes, it actually works with the same dual key cert when sending/receiving sign 
message using Outlook Express.

>Does OE handle signed emails generated from communicator which dual key certs
>correctly?  May need PSM 1.X

No, OE has the same problem when handling signed msg from communicator using 
dual-key certs.

Comment 5

[Aleksander Adamowski]

Antonio, can you please attach here a raw copy of the message generated by
Mozilla that OE can't handle properly?

Comment 6

[Marc Jadoul]

Reproduced this problem today with mozilla 1.1a/OE6 : OE6 do not see in a 
mozilla message any encryption certificate for the sender and any 'Recommended 
Encryption Algorithm'.
I will create a new attachement with the message sent to OE6 from mozilla and 
the needed P12 to decrypt/verify it and the dialog box from OE6.

Comment 7

[Marc Jadoul]

Attachment: Messages between Mozilla1.1a/OE6 + P12 (PW= 1234abC! )

There are 3 messages 4 certificates.
[e-mail] -> email on mozilla
[e-mail] -> email on OE6
P12 'MJ_?C_2' for mozilla1.1a (QC = signing only, NC = encryption) PW is
1234abC!
P12 'MJ_?C_3' for OE6 (QC = signing only, NC = encryption) PW is 1234abC!
1st message signed from mozilla (OE6 do not see sender's encryption cert)
2nd signed from OE6 (everything OK in mozilla)
3rd signed + encrypted (OE6 still do not see sender's encryption cert)

Comment 8

[Charles Rosendahl]

Unfortunately, it is my belief after thorough testing of Outlook, that Outlook
does not support dual key certificates.

I have imported dual key certs into outlook only to find that it is importing
the signing cert only and attempting to use it forencrypting purposes as well.

incoming messages using dual key certs will not import.

So, until Microsoft fixes their problem, I would suggest using dual use certs if
you need to interoperate among *all* mail clients.

If I am incorrect, or you have a test case wherein Outlook does use dual-key
certs successfully, I would be interested in seeing it.

Comment 9

[Kai Engert (:KaiE:)]

We could make a final test before giving up.

We could try to use IE to apply for a dual key certificate. If that does not
succeed, we can resolve this bug as invalid.

However, if it succeeds, and OE is indeed able to use that dual key certificate
for exchanging S/Mime messages, then we should export such a cert to a p12 file
and give it to the NSS team for investigation.

Comment 10

[Aleksander Adamowski]

Attachment: Cert from http://testca.netscape.com

The private key has strong protection witha a passphrase: 'test'.
Full certificate chain included in the .pfx file.

This file has been created by:
1) obtaining the test cert from http://testca.netscape.com (which supposedly is
a dual key one) using IE 6 latest version with all available fixes from Windows
Update
2) getting the cert from the web form using the Microsoft Strong Cryptography
Provider
3) exporting it from IE (Tools->Options->Content->Certificates) with private
key and full certification chain included.

Comment 11

[Aleksander Adamowski]

Using the certificate obtained at Netscape test CA, I can sign messages when I
send them from Outlook Express, Mozilla correctly verifies the signatures and is
able to send encrypted messages back to OE.

I've observed that when configuring the properties of an account in Outlook
Express (Tools->Accounts->Properties), I am able to set a signing cert and an
encryption cert separately for that account (on the Security tab).

So it would suggest that OE suports using separate certificates for signing and
encrypting.

However, as I understand it, dual key certificate is a different animal - a
single cert with corresponding two private and two public keys, am I correct? I
can only see one public key (Subject Public Key field) when viewing the
certificate in Mozilla, regardless of whether it's a certificate obtained using
IE or using Mozilla Browser. Where's the second key?

Anyway, I'm using the cert from attachment 91632 [details] in OE for both signing and
encrypting. Mozilla handles those signed messages gracefully and is able to send
bask encrypted mails.
When I obtain identical cert in Mozilla, and use it for signing messages, OE is
able to verify the signatures, but it doesn't import the encryption cert.
It's possible to open the signing cert from a message in OE, then to export it
to a file, then to manually assign it to a contact in OE address book. But when
sending encrypted mails to that contact, and receiving them via Mozilla, they
cannot be encrypted - it's probably because OE uses the signing cert for encryption.

Comment 12

[Aleksander Adamowski]

One question answer to which will be useful for me and other QA guys that test
S/MIME:
How to generate a dual key certificate using tools from OpenSSL (that crypto
library is bundled with most Linux distributions)?

Comment 13

[Kai Engert (:KaiE:)]

Reopening, as it is said that OE does support dual key certificates.

Comment 14

[Aleksander Adamowski]

Either OE supports them, or it ignores on of those two keys on a certificate.

I'd eagerly test that, but I need a link to comprehensive docs about dual key
certificates. And a method to create them myself using OpenSSL (if it is
possible with OpenSSL).

Comment 15

[Kai Engert (:KaiE:)]

Aleksander: You can get a test pair of dual key certificates at
  http://testca.netscape.com

Comment 16

[Aleksander Adamowski]

That's where I've got them when testing OE.
However, it would be useful in the testing process if I knew how to make my own
dual-key certs with OpenSSL tools, so that I can test for different x.509 cert
extensions etc.

Comment 17

[Kai Engert (:KaiE:)]

I couldn't find any references about the required characteristics of dual key
certificates. What I did is to just compare the attributes of the two individual
certs:

I think:
- you need two separate individual certificates
- most properties of the certificate should be identical, in my case, the
following are identical: subject, issuer, signature alg, validity (from-to),
keysize, extensions (except key usage)

Different are:
- serial numbers (in my case they are consecutive, no idea whether that matters)
- public and private key of your certificate
- one certificate has key usage "Key Encipherment"
- the other one has key usage "Digital Signature"

You can analyze the details yourself, get a cert from testca, export it to p12,
convert it to a pem (using openssl pkcs12 if you want), move your pem certs into
separate files, dump the attributes of your certs using openssl x509 -text.

Comment 18

[Aleksander Adamowski]

That being explained, I can say that Outlook Express only uses a single
certificate for both signing and encryption.

Either it doesn't support dual-key certs, or the Netscape test CA is configured
to  provide single-key certificates to Microsoft CryptoAPI clients.

Comment 19

[Charles Rosendahl]

I am not sure about 5.x, but I upgraded to IE/OE 6.0 and did the following:

Generated/Imported a dual key cert in my netscape client.  
Exported it to a p12 file.
Imported the p12 file into OE.
Configured OE to use the dual key cert.

It worked fine.  I did find some discprencies between the cert chains in OE
composed messages and Netscape composed messages that we are looking into.

Also, you need service release 1a (and registry mods) for office2k to enable
Outlook to use dual key certs.  Caveat:  Add the sender address to your address
book when reading a signed messageto get the encryption cert imported into the
Outlook address book.  From then on you can compose encrypted messages to that
recipient.

see
http://www.microsoft.com/technet/treeview/default.asp?url=/TechNet/prodtechnol/office/office2000/support/o2ksrwpr.asp

Comment 20

[Kai Engert (:KaiE:)]

Charles, you mention OE (Outlook Express) in your previous comment.

Did you get it to work with Outlook Express, or does it only work with Outlook
from Office 2000 and the mentioned service release?

Comment 21

[Charles Rosendahl]

With OE6 (IE6) I am able to use dual keys to configure my OE mail account and
sign messages.  I checked the cert configurations, and when I select the certs
from the imported p12 file, (which have that bizarre friendly name a million
chars long btw) I only see the appropriate cert in the selection list during
configuration (i.e. select to configure signing cert and only signing cert is in
selection list (sn 291), select to confiugre encryption cert and only encryption
cert is in selection list (sn 290))

I still have the problem receiving the encryption cert within Netscape messages
within OE perhaps due to the cert chain structures.

Comment 22

[Kai Engert (:KaiE:)]

Charles discovered that when Outlook exchanges S/Mime messages, where the user
uses a dual key certificate, it is sending two full certificate chains. It was
suggested we should do the same.

Comment 23

[Kai Engert (:KaiE:)]

I made a lot of tests and here are my results.

Summary: Sending full chains is not sufficient. I need help from the NSS team.
Analysis below.


Long story:

- the only version of Outlook I have available is Outlook Express 5.5
- Charles did tests with various Outlook versions, both from Office and Express
- I was unable to create a dual key cert message with OE 5.5
- However, Charles was able to send me a reference dual key cert message, sent
with Outlook, that works correctly in Mach V, OE 5.5 and the other versions of
Outlook
- we used dual key certificates from testca.netscape.com in order to allow
everybody involved to create messages using their own test cert and comparable
cert / cert chain characteristics
- although Outlook is using opaque signed messages by default, we found the
option in Outlook to make it send messages with detached signature - the
approach we are using in Mozilla

That means, I have a reference message from Outlook that works with all mail
clients we tested. We need to learn the characteristics of that message and make
Mozilla include the same attributes.


I used the openssl toolkit to convert the messages.
I used "base -d" and "asn1parse -i -dump" to create a raw dump of the
x-pkcs7-signature contents.
I created a small patch to Mozilla (see attachment titled "Patch to include all
certs"). With that patch, the messages that we create are mostly identical to
the Outlook message. We include the same number of certs, in the same order.
If I undersand correctly, we also include the same order/types of digests,
public keys


But there is one difference, and I guess that is what we need to add or change
to make it work.

In section S/Mime capabilities, Outlook is including an additional:
          SEQUENCE          
           OBJECT            :sha1WithRSA

I suspect the section that follows is the encryption cert preference attribute.
Mozilla is including a
       SEQUENCE
        OBJECT            :id-smime-aa-encrypKeyPref
        SET
         cont [ 0 ]
while Outlook is including a
       SEQUENCE
        OBJECT            :1.3.6.1.4.1.311.16.4
        SET
         SEQUENCE


Do you understand what the latter means, and whether we are able to include
that, too?

Comment 24

[Kai Engert (:KaiE:)]

Attachment: Patch to include all certs

Comment 25

[Kai Engert (:KaiE:)]

Attachment: x-pkcs7-signature diff

Diff between
  pkcs7-signature dump of a non-working message from Mozilla
and
  pkcs7-signature dump of a working message from Outlook

Comment 26

[Nelson Bolyard (seldom reads bugmail)]

OIDs that begin with 1.3.6.1.4.1.311. are reportedly Microsoft OIDs, just as
OIDs that begin with 2.16.840.1.113730 are Netscape OIDs.  
Maybe we'd have to use a Microsoft OID to interoperate with OE 6.

Comment 27

[Stephane Saux]

cc David.

Comment 28

[Robert Relyea]

Kai, can you tell if we just change the oid and not include the additional certs
if the email message works?

If so that is both good and bad news. Good news, we don't have to double our
message size to work with microsoft. Bad news, we have to break compatibility
with everyone else (using a microsoft OID can't possibly be spec'ed in the ietf
standard). Obviously we already accept both OID types.

bob

Comment 29

[Kai Engert (:KaiE:)]

I need help from you to make that test. How can I change the object type?

Comment 30

[Terry Hayes]

The quick way to test is to redefine the OID 
SEC_OID_SMIME_ENCRYPTION_KEY_PREFERENCE to refer to the Microsoft value. You 
can do this in secoid.c

A possible permanent fix is to define the Microsoft OID 
(SEC_OID_MSFT_ENCRYPTION_KEY_PREFERENCE?) in secoid.c, and then add both 
attributes to the SignedData object.  You would need to change lines 731-735 of 
cmssiginfo.c to do this.  I think the change is to create a second attribute 
with the MSFT OID, and add it to the list.  You can probably reuse the bits 
created by CreateSMIMEEncKeyPrefs call.

Comment 31

[Kai Engert (:KaiE:)]

Thanks for your helpful comments.

Simply replacing the OID unfortunately did not work, because the contents of the
object are also slightly different. The MS OID has a simpler nesting structure.
OE was unable to understand the message with our current nesting structure.

I was able to make it work by adding an additional attribute that matches
exactly what OE is doing. I will attach a patch.

At least in my tests it was NOT necessary to include more certs than we are
currently sending.

While this sounds like good news, note that I only had OE 5.5 available for
testing. I have sent a test message to Charles and asked him to do some
additional testing. We need to know whether there some versions of OE still have
problems, and whether we need to include all the certs for some of them.

Question: Would it be ok to always include the MS OID attribute with all
outgoing signed messages, or are there any reasons why we should not? Like not
being allowed to use that OID?

Also note that today I found a document on the web that confirms what we have
learned. Look for 1.3.6.1.4.1.311.16.4 in
http://ice-car.darmstadt.gmd.de/deliverables/icecar-R124.pdf
They are citing Netscape, but I believe they are probably refering to 4.x

Comment 32

[Kai Engert (:KaiE:)]

Comment on attachment 94421 [details] [diff] [review]
Patch to include all certs

This patch is still valid. It does not conflict with the other patch. It can be
used to include all certs, but right now I believe it is not necessary.

Comment 33

[Kai Engert (:KaiE:)]

Attachment: Patch to include the encryption cert preference using MS oid

Comment 34

[Kai Engert (:KaiE:)]

Bob, I suspect we are not yet supporting the MS OID type in incoming messages. I
suspect we behave correctly by simply finding an encryption cert in the message
and using it, even though we don't find an encryption cert preference in the
message.

This rises the next question: Should we add the capability to detect that
preference in incoming signed messages? (But that would be a separate
enhancement bug.)

Comment 35

[Charles Rosendahl]

Kai sent me a test message with the new OID.  I looked at it through NS 7.0,
Outlook Express 6.0, Outlook Office 2000, and Outlook Office XP successfully.

So it appears that the OID incompatibility is the issue preventing Outlook from
recognizing the presence of the encryption cert.

Comment 36

[Kai Engert (:KaiE:)]

So the attached patch indeed seems to fix the problem.

The message I had sent to Charles included the additional OID, but did not
include additional certs (meaning it is not necessary to include the root certs
or multiple full chains).


Should we include the MS OID attribute with all outgoing signed messages, or are
there any reasons why we should not?

Comment 37

[Kai Engert (:KaiE:)]

Comment on attachment 94421 [details] [diff] [review]
Patch to include all certs

(This patch is not required)

Comment 38

[Kai Engert (:KaiE:)]

I realize all my changes are completely within NSS.

So the question actually is: Should NSS itself always create that additional
attribute by default in outgoing signed messages (like the patch is suggesting)?


If you agree that NSS should be changed, can you please review the NSS patch?

Comment 39

[Kai Engert (:KaiE:)]

cc'ing jpierre, relyea, wtc, since this is now a change request to NSS.


Bob, I realized you were not cc'ed. Please see your question in comment 28, and
my answers from comment 34 and following.

Comment 40

[Robert Relyea]

Comment on attachment 95006 [details] [diff] [review]
Patch to include the encryption cert preference using MS oid

This patch looks very good. I only have two comments:
1) The definition of the oid string (line 212 secoid.c) , we should add a
define for The Microsoft space #define MICROSOFT_OID 0x2b, 0x6, 0x1, 0x82, 0x37
Then the actual oid becomes: { MICROSOFT_OID, 0x10, 0x4 }

The second thing is we probably should put the work "Microsoft" in the
definition string at line 1002 or so ("S/MIME Encryption Key Preference"  ->
"Microsoft S/MIME Encryption Key Preference" )

bob

Comment 41

[Robert Relyea]

comment 34 : No, our current code is more general. We can automatically find the
correct cert in the list. Microsoft's extension is non-standard, and we should
be able to read any standard message. We want to include the extension on output
because it's an easy way to keep compatibility without making our format
non-standard (standard implementations will ingore the unknown out, just like
ours does today).

Once the review changes are made, go ahead and check the change into the NSS tip
as well. Also, if there is some disagreement on including "Microsoft" in the oid
string (which could be displayed by various NSS programs when displaying the
cert), go ahead and check the patch with that line as stands (but fix the OID
definition).

Comment 42

[Terry Hayes]

Two comments:

1) I agree that "Microsoft" should appear in the name of the extension.  We 
need to clearly distinguish the two values.
2) Since CMS (PKCS-7) is used by applications other than S/MIME email, you 
might consider allowing the application to choose whether to include the MSFT 
value.  Some applications do not include the KEY_PREF at all. The patch is fine 
for them.  However, for applications that want to include a KEY_PREF, but do 
not need to support the private Microsoft version, there should be a way to do 
that.  I suggest that you create a separate function 
(NSS_CMSSignerInfo_AddMicroSoftEncKeyPrefs?) that does this.

Comment 43

[Kai Engert (:KaiE:)]

Bob,
   the next patch I attach will have the changes you suggested.

Terry,
   you suggest to allow the application to choose which encryption key prefs to
add. In the next patch, instead of extending the existing NSS function to add
the attribute, I'll leave that function unchanged, and add a new one (mostly
duplicated) which only adds the MS pref.

The new patch is larger, because I changed PSM and cmd/smimetools/cmsutil to
call both functions. I also had to export the new function in smime.def.

A disadvantage of this new approach is slower performance. If an application
adds both prefs, we'll call CERT_VerifyCert twice.

Can you please review again?

Comment 44

[Kai Engert (:KaiE:)]

Attachment: Patch v2

Comment 45

[Robert Relyea]

Comment on attachment 96314 [details] [diff] [review]
Patch v2

Sigh, one last thing. The export shouldn't have NSS 3.6 because this patch
isn't for 3.6, it's for 3.5. It should go in a should be a new 3.5.1 block. 

With that change I would mark this as has-review (everything else looks good).

bob

Comment 46

[Robert Relyea]

BTW I'm assuming you are trying for mozilla 1.1, if you are not, then we would
want this fix check into the NSS tip, with the new symbol in the NSS 3.6 block,
and then have mozilla pick it up when we roll the client tag forward.

bob

Comment 47

[Wan-Teh Chang]

We should try very hard to NOT add a new function
in a patch release (x.y.z).  The general convention
is that new functions may only be added in a minor
release (x.y).

I suggest that we check in the NSS portion of the
patch on the tip of NSS (3.6) and plan to move the
NSS_CLIENT_TAG over to NSS 3.6 (pre-release) as
soon as possible.

Comment 48

[Kai Engert (:KaiE:)]

Landing for NSS 3.6 / Mozilla 1.2 would be sufficient, I think.

Comment 49

[Robert Relyea]

Comment on attachment 96314 [details] [diff] [review]
Patch v2

patch approved for NSS 3.6 be sure to merge with NSS 3.6 nss.def file.

Comment 50

[Kai Engert (:KaiE:)]

Comment on attachment 96314 [details] [diff] [review]
Patch v2

The NSS portion of this patch has been checked in to the trunk of NSS.

Marking patch as obsolete to avoid confusion, since I need to leave this bug
open.

Comment 51

[Kai Engert (:KaiE:)]

Attachment: PSM subset of Patch v2

This portion of the patch depends on the new NSS 3.6 functionality.

We can not check it in until NSS_CLIENT_TAG has been moved to 3.6.

Comment 52

[Kai Engert (:KaiE:)]

Comment on attachment 96727 [details] [diff] [review]
PSM subset of Patch v2

carrying forward r=relyea

Comment 53

[Kai Engert (:KaiE:)]

I assume everybody wants us to include the MS key preference attribute in
outgoing signed mail. If you know a reason why we should not do it, please object.

If nobody objects, the change to activate the inclusion will land as soon as bug
160734 (switch Mozilla to use NSS 3.6) has been fixed.

Comment 54

[Scott MacGregor]

Comment on attachment 96727 [details] [diff] [review]
PSM subset of Patch v2

sr=mscott

Comment 55

[Kai Engert (:KaiE:)]

fixed on trunk

Comment 56

[Charles Rosendahl]

Verified - Outlook Express now properly handles incoming dual key certs from
Netscape.