303340 - Add new bignum library for AMD64 with GCC

Status: RESOLVED WONTFIX

(NSS :: Libraries, enhancement)

Description

[Nelson Bolyard (seldom reads bugmail)]

TFM (Tom's Fast Math) is a bignum library that uses the MPI API 
(except that all the MP and mp symbols have been changed to FP and fp).
It makes use of GCC features such as in-line assembly and variable attributes.
It's VERY fast on AMD64.  

It is currently offered as open source under a public domain statement, 
not under a copyright license.  (I've discussed this with mozilla staff, 
and believe it's OK with them.)

We started with TFM 0.03 made some enhancements of our own to it.  
We're seriously thinking of using this code in NSS 3.11 for AMD64 CPUs.

The original TFM 0.03 code may be seen here:
http://bonsai.mozilla.org/cvsquery.cgi?treeid=default&module=NSS&branch=NSS_PERFORMANCE_HACKS_BRANCH&branchtype=match&dir=mozilla%2Fsecurity%2Fnss%2Flib%2Ffreebl%2Ftfm&file=&filetype=match&who=&whotype=match&sortby=Date&hours=2&date=explicit&mindate=2005-04-06&maxdate=2005-05-26&cvsroot=%2Fcvsroot

Our changes to it may be seen here:
http://bonsai.mozilla.org/cvsquery.cgi?treeid=default&module=NSS&branch=NSS_PERFORMANCE_HACKS_BRANCH&branchtype=match&dir=mozilla%2Fsecurity%2Fnss%2Flib%2Ffreebl%2Ftfm&file=&filetype=match&who=&whotype=match&sortby=Date&hours=2&date=explicit&mindate=2005-05-26&maxdate=2005-08-03&cvsroot=%2Fcvsroot

Presently it has a 4k bit limitation on bignums, which means that the largest 
RSA private key it can handle is 2k bits.  We can easily raise that limit, 
(It's a single #define) with some performance cost.  Another idea is to use 
TFM for keys that fit it, and use MPI for keys that don't fit TFM.  
We are exploring those options.

Some of TFM's speed advantage over MPI comes from the fact that TFM allocates
all bignums from the stack.  This means it uses considerably more stack 
than MPI.  Application threads will need larger stacks with this code than
with MPI.  However, since AMD64 support is new for NSS, we think it may be
reasonable to simply state that NSS on AMD64 requires larger thread stacks
than on some other CPUs.  Exact numbers are TBD.

Comment 1

[Nelson Bolyard (seldom reads bugmail)]

We've decided not to do this.  We're going to produce AMD64 specific
assembly optimizations for 2-3 files in MPI instead.  

Wan-Teh, Please don't spend any (more) time reviewing the code at the 
URLs listed above, because we're abandoning it on the branch.