211655 - NSS incorrectly DER-encodes SET-OF in cert DNs

Status: NEW

(NSS :: Libraries, defect, P3)

Description

[Nelson Bolyard (seldom reads bugmail)]

ASN.1 defines an abstract SET OF as an unordered set.  
The order of the members of the set in its abstract form is not meaningful.

But the Distinguished Encoding Rules in X.690 require the members of a DER 
ENCODED SET OF to be ordered.  This allows two DER encoded SET-OFs containing
the same members to be compared by simple binary comparison of the encoded 
SET-OF values, rather than by the more complicated method of parsing the
SET-OF encodings, sorting the members, and then comparing, or doing an O(n^2)
comparison of members.

X.690 section 11.6 says that the components of a DER encoded SET OF are in
ascending order when compared as octet strings.  SECITEM_CompareItem 
implements the octet string comparison algorithm of X.690 section 6.1-6.3.

So, the correct thing to do would seem to be to encode all the components 
of the SET OF, then sort them in ascending order using SECITEM_CompareItem
as the comparison function, before outputting them.

NSS's SEC_ASN1Encode* encoder doesn't appear to do that.  
This will cause failures to find matching names.

Example:

The names (in RFC 2253 form)  
O=AOL+C=US and 
C=US+O=AOL 

are the same abstract name.  They each contain one RDN, which is a SET OF 
two AVAs. In the abstract (and ASCII) form, the order of their AVAs are not
significant.  The two names are the same.  Either form, when DER encoded, 
should produce the same result, and consequently, a straight binary 
comparison of the two DER-encoded names should produce the same result.  

But NSS's encoder will create two different encodings, that will not compare
as identical.

Comment 1

[Julien Pierre]

Nelson,

Is this a DER vs BER issue ?
Ie. the ASN.1 DER encoding requires ordering for the SET OF, but ASN.1 BER
encoding does not ?
If so, it might be possible to handle things differently when we are using the
streaming encoder vs the non-streaming version.

Comment 2

[Nelson Bolyard (seldom reads bugmail)]

Yes, the ordering is a DER requirement, not a BER requirement.