Closed Bug 2087611 Opened 23 days ago Closed 6 days ago

NSS S/MIME: Decode the RC2 capability parameter with QuickDER

Categories

(NSS :: Libraries, enhancement)

enhancement

Tracking

(nss 3.134)

RESOLVED FIXED
Tracking Status
nss --- 3.134

People

(Reporter: KaiE, Assigned: KaiE)

References

(Blocks 1 open bug)

Details

Attachments

(1 external file)

Below is a code path that Claude Code identified, and I think it would be good to change it to no longer use secasn1d.

Thunderbird's nsCMSMessage::CreateEncrypted calls NSS_SMIMEUtil_FindBulkAlgForRecipients

NSS: smime_choose_cipher reads each recipient's stored S/MIME profile (CERT_FindSMimeProfile).
It decodes the capability list with QuickDER and, per capability,
calls nss_SMIME_FindCipherForSMIMECap. For RC2 capabilities, that function decodes the parameters with
secasn1d and then calls SEC_ASN1DecodeInteger (734).

NSS_SMIMEUtil_FindBulkAlgForRecipients chooses the content cipher from the
recipients' stored S/MIME profiles. A profile is the SMIMECapabilities
attribute of an earlier signed message from that recipient, so its bytes come
from a remote sender. The capability list is already decoded with QuickDER
(smimeutil.c, smime_choose_cipher). Only the parameter of an RC2-CBC
capability, the key size as an INTEGER, was decoded with the streaming
decoder (SEC_ASN1DecodeItem). This patch decodes it with QuickDER too, so
choosing a cipher no longer runs the streaming decoder on remote data.

Why the result is the same for every caller:

The parameter reaches nss_SMIME_FindCipherForSMIMECap as cap->parameters,
which the outer QuickDER decode captured as an ANY (quickder.c, DecodeItem:
ANY is taken with GetItem(..., includeTag)). So it is exactly one TLV: a
single tag octet, a minimal definite length, and contents of exactly that
length, with nothing after it (quickder.c, definite_length_decoder). On such
input both decoders, with SEC_IntegerTemplate and a siBuffer destination,
behave the same:

  1. Tag: both accept only 0x02. The streaming decoder compares the
    constructed bit for INTEGER, since INTEGER is not one of the string types
    it allows in constructed form (secasn1d.c,
    sec_asn1d_init_state_based_on_template; sec_asn1d_confirm_identifier).
    QuickDER checks class, tag number, and primitive form for universal types
    (quickder.c, MatchComponentType).
  2. Length: already checked by the outer decode; no trailing data is possible.
  3. Contents: both return the contents octets unchanged. Leading zeros are
    only stripped for siUnsignedInteger (secasn1d.c, sec_asn1d_parse_leaf;
    quickder.c, DecodeItem), and keyItem is siBuffer.
  4. Empty contents: QuickDER rejects an empty INTEGER (quickder.c, DecodeItem).
    The streaming decoder returns an empty item with data NULL, which the
    following SEC_ASN1DecodeInteger rejects (secasn1d.c). Both end in
    SEC_OID_UNKNOWN.

After decoding, the code is unchanged: SEC_ASN1DecodeInteger and the key size
mapping. The only differences are internal: the error code set on a failed
decode (not reported; the function returns SEC_OID_UNKNOWN), and keyItem now
points into cap->parameters, which lives in the profile until the end of the
loop iteration, instead of a heap copy.

Testing: certdb_gtest SMimeCipherChoiceTest stores a profile [RC2-CBC with a
given parameter, AES-128-CBC] and checks the chosen cipher: key sizes 128 and
64 choose RC2; an unsupported key size, an empty INTEGER, a constructed
INTEGER, and an OCTET STRING are ignored and AES-128 is chosen. The tests
pass with and without the change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

There is an r+ patch which didn't land and no activity in this bug for 1 week.
:KaiE, could you have a look please?
If you still have some work to do, you can add an action "Plan Changes" in Phabricator.
For more information, please visit BugBot documentation.

Flags: needinfo?(kaie)
Attachment #9671847 - Attachment description: Bug 2087611 - NSS smime: decode the RC2 capability parameter with QuickDER. r=#nss-reviewers → Bug 2087611 - NSS smime: decode the RC2 capability parameter with QuickDER. r=rrelyea
Assignee: nobody → kaie
Flags: needinfo?(kaie)

Pushed by kaie@kuix.de:
https://hg.mozilla.org/projects/nss/rev/13e337887958
NSS smime: decode the RC2 capability parameter with QuickDER. r=nss-reviewers,rrelyea

Status: NEW → RESOLVED
Closed: 6 days ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: