RFC3560 specifies the use of RSAES-OAEP in S/MIME . It is available from http://www.ietf.org/rfc/rfc3560.txt . We should support it when we add support for RSAES-OAEP to NSS .
More recently, AES (RFC 3565) has been added as a SHOULD implement to S/MIME 3.1 (RFC 3851). http://www.apps.ietf.org/rfc/rfc3851.html#sec-2.7 http://www.apps.ietf.org/rfc/rfc3565.html 2.7 ContentEncryptionAlgorithmIdentifier Sending and receiving agents MUST support encryption and decryption with DES EDE3 CBC, hereinafter called "tripleDES" [CMSALG]. Receiving agents SHOULD support encryption and decryption using the RC2 [CMSALG] or a compatible algorithm at a key size of 40 bits, hereinafter called "RC2/40". Sending and receiving agents SHOULD support encryption and decryption with AES [CMSAES] at a key size of 128, 192, and 256 bits. ... [CMSAES] Schaad, J., "Use of the Advanced Encryption Standard (AES) Encryption Algorithm in Cryptographic Message Syntax (CMS)", RFC 3565, July 2003.
AES support in libSMIME should be a separate enhancement request. Nicholas, please file a new enhancement request, product NSS, component libraries. Give the same text you gave in comment 1 above (plus any more you deem necessary). Please CC me on that bug.