213313 - shlibsign exception using VC++ 2003

Status: RESOLVED FIXED

(NSS :: Libraries, defect)

Description

[Julien Pierre]

I installed MS Visual Studio .NET 2003 from MSDN on my system and tried to build
NSS.

I got the following error :

gmake[3]: Leaving directory `C:/nss/39/mozilla/security/nss/cmd/shlibsign/mangl
'
WIN954.0_DBG.OBJ/shlibsign -v -i ../../../../dist/WIN954.0_DBG.OBJ/lib/softokn3
dll
Generating DSA Key Pair....[1] + Done(136) ?
  2356  Floating point exception        WIN954.0_DBG.OBJ/shlibsign
gmake[2]: *** [../../../../dist/WIN954.0_DBG.OBJ/lib/softokn3.chk] Error 136
gmake[2]: Leaving directory `C:/nss/39/mozilla/security/nss/cmd/shlibsign'
gmake[1]: *** [libs] Error 2
gmake[1]: Leaving directory `C:/nss/39/mozilla/security/nss/cmd'
gmake: *** [libs] Error 2

The same tool works with the previous version of VC7 (2001).

Comment 1

[Julien Pierre]

Here is the stack :

 	softokn3.dll!_s_mpv_div_2dx1d()  Line 339	Asm
 	softokn3.dll!mp_div(const mp_int * a=0x0012f934, const mp_int * b=0x0012f9d0,
mp_int * q=0x00000000, mp_int * r=0x0012f934)  Line 1004 + 0x11	C
 	softokn3.dll!mp_mod(const mp_int * a=0x0012f934, const mp_int * m=0x0012f9d0,
mp_int * c=0x0012f934)  Line 1188 + 0x13	C
>	softokn3.dll!mpp_pprime(mp_int * a=0x0012f9d0, int nt=18)  Line 331 + 0x11	C
 	softokn3.dll!PQG_ParamGenSeedLen(unsigned int j=8, unsigned int seedBytes=128,
PQGParamsStr * * pParams=0x0012fa38, PQGVerifyStr * * pVfy=0x0012fa3c)  Line
474 + 0xb	C
 	softokn3.dll!PQG_ParamGen(unsigned int j=8, PQGParamsStr * *
pParams=0x0012fa38, PQGVerifyStr * * pVfy=0x0012fa3c)  Line 366 + 0x15	C
 	softokn3.dll!nsc_parameter_gen(unsigned long key_type=1, PK11ObjectStr *
key=0x00547f70)  Line 2524 + 0x11	C
 	softokn3.dll!NSC_GenerateKey(unsigned long hSession=1, CK_MECHANISM *
pMechanism=0x0012fbbc, CK_ATTRIBUTE * pTemplate=0x0012fbd8, unsigned long
ulCount=1, unsigned long * phKey=0x0012fbd4)  Line 2891 + 0x10	C
 	nss3.dll!PK11_PQG_ParamGenSeedLen(unsigned int j=8, unsigned int seedBytes=0,
PQGParamsStr * * pParams=0x0012fd00, PQGVerifyStr * * pVfy=0x0012fd20)  Line
105 + 0x31	C
 	nss3.dll!PK11_PQG_ParamGen(unsigned int j=8, PQGParamsStr * *
pParams=0x0012fd00, PQGVerifyStr * * pVfy=0x0012fd20)  Line 187 + 0x13	C
 	shlibsign.exe!main(int argc=4, char * * argv=0x00532dc0)  Line 232 + 0x1a	C
 	shlibsign.exe!mainCRTStartup()  Line 398 + 0xe	C
 	KERNEL32.DLL!77ea847c() 	

The failure is in the assembly code in mpi_x86.asm , on a "div ebx".
Perhaps the calling conventions changed in VC++ 2003 ?

Comment 2

[Nelson Bolyard (seldom reads bugmail)]

I seem to recall that the floating point exception and the divide-by-zero 
exceptions are one and the same.  

Can you tell by looking at the caller whether it passed in a zero divisor 
or not?  I'll be happy to come by and take a look if you want.

Comment 3

[Julien Pierre]

Had Nelson look at it. The problem was that mp_digit was considered to be a
64-bit integer rather than 32-bit using VC++ 2003 .
Forcing it to 32-bits using a define worked and allowed the signing to pass, as
well as all.sh .

Comment 4

[Julien Pierre]

Attachment: Patch to make mp_digit 32 bits with MS VC++ .net 2003

Comment 5

[Wan-Teh Chang]

Comment on attachment 128190 [details] [diff] [review]
Patch to make mp_digit 32 bits with MS VC++ .net 2003

Julien, to review this patch I'd like to know
why mp_digit becomes 64-bit in VC++ 2003.  Looking
at mpi.h, I think this is because VC++ 2003 adds
the support of the long long type.  Could you
verify this?

mpi.h has three possible typedefs for mp_digit.
Could you find out which typedef is used if this
patch is not applied?  I guess it's the second
one:
    typedef unsigned long long mp_digit;
and it is because one of the following macros is
defined in VC++ 2003's header file <limits.h>:
    ULONG_LONG_MAX
    ULLONG_MAX
    ULONGLONG_MAX

Thanks.

Comment 6

[Julien Pierre]

Yes, indeed, without the patch, mp_digit is defined as

typedef unsigned long long mp_digit;

This is caused by ULLONG_MAX being defined in limits.h :

C:\MSVSNET2003\Vc7\include>grep -i ullong_max *.*
limits.h:#define ULLONG_MAX    0xffffffffffffffff  /*maximum unsigned __int64
value */

Comment 7

[Wan-Teh Chang]

Comment on attachment 128190 [details] [diff] [review]
Patch to make mp_digit 32 bits with MS VC++ .net 2003

Thank you, Julien.  r=wtc.

Comment 8

[Nelson Bolyard (seldom reads bugmail)]

Comment on attachment 128190 [details] [diff] [review]
Patch to make mp_digit 32 bits with MS VC++ .net 2003

In addition to defining  -DMP_USE_UINT_DIGIT  we also need to define
-DMP_NO_MP_WORD
With both of those symbols defined, the same code is generated on both old and
new MSVC compilers.

Comment 9

[Julien Pierre]

Attachment: update

Comment 10

[Nelson Bolyard (seldom reads bugmail)]

Comment on attachment 128201 [details] [diff] [review]
update

sr=nelsonb

Comment 11

[Julien Pierre]

Checking in Makefile;
/cvsroot/mozilla/security/nss/lib/freebl/Makefile,v  <--  Makefile
new revision: 1.49; previous revision: 1.48
done

Comment 12

[Julien Pierre]

fixed

Comment 13

[Stephen Walker]

Can a NSS person get this fix on the NSS_CLIENT_TAG?

Comment 14

[Julien Pierre]

Stephen,

This bug was fixed on the tip of NSS - the yet unreleased NSS version 3.9. I
have updated the target milestone to reflect this.
 
The NSS_CLIENT_TAG is based on NSS version 3.8 . To get this patch into version
3.8 :
- please open a separate bug with a target milestone of NSS 3.8
- attach the patch
- cc me
- send email to [e-mail] for approval about the new bug, since the
Mozilla trunk is currently closed

Closing this bug.