ASN.1 defines an abstract SET OF as an unordered set. The order of the members of the set in its abstract form is not meaningful. But the Distinguished Encoding Rules in X.690 require the members of a DER ENCODED SET OF to be ordered. This allows two DER encoded SET-OFs containing the same members to be compared by simple binary comparison of the encoded SET-OF values, rather than by the more complicated method of parsing the SET-OF encodings, sorting the members, and then comparing, or doing an O(n^2) comparison of members. X.690 section 11.6 says that the components of a DER encoded SET OF are in ascending order when compared as octet strings. SECITEM_CompareItem implements the octet string comparison algorithm of X.690 section 6.1-6.3. So, the correct thing to do would seem to be to encode all the components of the SET OF, then sort them in ascending order using SECITEM_CompareItem as the comparison function, before outputting them. NSS's SEC_ASN1Encode* encoder doesn't appear to do that. This will cause failures to find matching names. Example: The names (in RFC 2253 form) O=AOL+C=US and C=US+O=AOL are the same abstract name. They each contain one RDN, which is a SET OF two AVAs. In the abstract (and ASCII) form, the order of their AVAs are not significant. The two names are the same. Either form, when DER encoded, should produce the same result, and consequently, a straight binary comparison of the two DER-encoded names should produce the same result. But NSS's encoder will create two different encodings, that will not compare as identical.
Nelson, Is this a DER vs BER issue ? Ie. the ASN.1 DER encoding requires ordering for the SET OF, but ASN.1 BER encoding does not ? If so, it might be possible to handle things differently when we are using the streaming encoder vs the non-streaming version.
Yes, the ordering is a DER requirement, not a BER requirement.