shlibsign exception using VC++ 2003

RESOLVED FIXED in 3.9

Status

NSS
Libraries
RESOLVED FIXED
15 years ago
15 years ago

People

(Reporter: Julien Pierre, Assigned: Julien Pierre)

Tracking

unspecified
x86
Windows 2000

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment, 1 obsolete attachment)

664 bytes, patch
Nelson Bolyard (seldom reads bugmail)
: superreview+
Details | Diff | Splinter Review
(Assignee)

Description

15 years ago
I installed MS Visual Studio .NET 2003 from MSDN on my system and tried to build
NSS.

I got the following error :

gmake[3]: Leaving directory `C:/nss/39/mozilla/security/nss/cmd/shlibsign/mangl
'
WIN954.0_DBG.OBJ/shlibsign -v -i ../../../../dist/WIN954.0_DBG.OBJ/lib/softokn3
dll
Generating DSA Key Pair....[1] + Done(136) ?
  2356  Floating point exception        WIN954.0_DBG.OBJ/shlibsign
gmake[2]: *** [../../../../dist/WIN954.0_DBG.OBJ/lib/softokn3.chk] Error 136
gmake[2]: Leaving directory `C:/nss/39/mozilla/security/nss/cmd/shlibsign'
gmake[1]: *** [libs] Error 2
gmake[1]: Leaving directory `C:/nss/39/mozilla/security/nss/cmd'
gmake: *** [libs] Error 2

The same tool works with the previous version of VC7 (2001).
(Assignee)

Comment 1

15 years ago
Here is the stack :

 	softokn3.dll!_s_mpv_div_2dx1d()  Line 339	Asm
 	softokn3.dll!mp_div(const mp_int * a=0x0012f934, const mp_int * b=0x0012f9d0,
mp_int * q=0x00000000, mp_int * r=0x0012f934)  Line 1004 + 0x11	C
 	softokn3.dll!mp_mod(const mp_int * a=0x0012f934, const mp_int * m=0x0012f9d0,
mp_int * c=0x0012f934)  Line 1188 + 0x13	C
>	softokn3.dll!mpp_pprime(mp_int * a=0x0012f9d0, int nt=18)  Line 331 + 0x11	C
 	softokn3.dll!PQG_ParamGenSeedLen(unsigned int j=8, unsigned int seedBytes=128,
PQGParamsStr * * pParams=0x0012fa38, PQGVerifyStr * * pVfy=0x0012fa3c)  Line
474 + 0xb	C
 	softokn3.dll!PQG_ParamGen(unsigned int j=8, PQGParamsStr * *
pParams=0x0012fa38, PQGVerifyStr * * pVfy=0x0012fa3c)  Line 366 + 0x15	C
 	softokn3.dll!nsc_parameter_gen(unsigned long key_type=1, PK11ObjectStr *
key=0x00547f70)  Line 2524 + 0x11	C
 	softokn3.dll!NSC_GenerateKey(unsigned long hSession=1, CK_MECHANISM *
pMechanism=0x0012fbbc, CK_ATTRIBUTE * pTemplate=0x0012fbd8, unsigned long
ulCount=1, unsigned long * phKey=0x0012fbd4)  Line 2891 + 0x10	C
 	nss3.dll!PK11_PQG_ParamGenSeedLen(unsigned int j=8, unsigned int seedBytes=0,
PQGParamsStr * * pParams=0x0012fd00, PQGVerifyStr * * pVfy=0x0012fd20)  Line
105 + 0x31	C
 	nss3.dll!PK11_PQG_ParamGen(unsigned int j=8, PQGParamsStr * *
pParams=0x0012fd00, PQGVerifyStr * * pVfy=0x0012fd20)  Line 187 + 0x13	C
 	shlibsign.exe!main(int argc=4, char * * argv=0x00532dc0)  Line 232 + 0x1a	C
 	shlibsign.exe!mainCRTStartup()  Line 398 + 0xe	C
 	KERNEL32.DLL!77ea847c() 	

The failure is in the assembly code in mpi_x86.asm , on a "div ebx".
Perhaps the calling conventions changed in VC++ 2003 ?
(Assignee)

Updated

15 years ago
Summary: shlibsign won't run using VC++ 2003 → shlibsign exception using VC++ 2003
I seem to recall that the floating point exception and the divide-by-zero 
exceptions are one and the same.  

Can you tell by looking at the caller whether it passed in a zero divisor 
or not?  I'll be happy to come by and take a look if you want.
(Assignee)

Comment 3

15 years ago
Had Nelson look at it. The problem was that mp_digit was considered to be a
64-bit integer rather than 32-bit using VC++ 2003 .
Forcing it to 32-bits using a define worked and allowed the signing to pass, as
well as all.sh .
(Assignee)

Comment 4

15 years ago
Created attachment 128190 [details] [diff] [review]
Patch to make mp_digit 32 bits with MS VC++ .net 2003
(Assignee)

Updated

15 years ago
Attachment #128190 - Flags: superreview?(nelsonb)
Attachment #128190 - Flags: review?(wtc)

Comment 5

15 years ago
Comment on attachment 128190 [details] [diff] [review]
Patch to make mp_digit 32 bits with MS VC++ .net 2003

Julien, to review this patch I'd like to know
why mp_digit becomes 64-bit in VC++ 2003.  Looking
at mpi.h, I think this is because VC++ 2003 adds
the support of the long long type.  Could you
verify this?

mpi.h has three possible typedefs for mp_digit.
Could you find out which typedef is used if this
patch is not applied?  I guess it's the second
one:
    typedef unsigned long long mp_digit;
and it is because one of the following macros is
defined in VC++ 2003's header file <limits.h>:
    ULONG_LONG_MAX
    ULLONG_MAX
    ULONGLONG_MAX

Thanks.
(Assignee)

Comment 6

15 years ago
Yes, indeed, without the patch, mp_digit is defined as

typedef unsigned long long mp_digit;

This is caused by ULLONG_MAX being defined in limits.h :

C:\MSVSNET2003\Vc7\include>grep -i ullong_max *.*
limits.h:#define ULLONG_MAX    0xffffffffffffffff  /*maximum unsigned __int64
value */

Comment 7

15 years ago
Comment on attachment 128190 [details] [diff] [review]
Patch to make mp_digit 32 bits with MS VC++ .net 2003

Thank you, Julien.  r=wtc.
Attachment #128190 - Flags: review?(wtc) → review+
Comment on attachment 128190 [details] [diff] [review]
Patch to make mp_digit 32 bits with MS VC++ .net 2003

In addition to defining  -DMP_USE_UINT_DIGIT  we also need to define
-DMP_NO_MP_WORD
With both of those symbols defined, the same code is generated on both old and
new MSVC compilers.
Attachment #128190 - Flags: superreview?(nelsonb) → superreview-
(Assignee)

Comment 9

15 years ago
Created attachment 128201 [details] [diff] [review]
update
Attachment #128190 - Attachment is obsolete: true
Comment on attachment 128201 [details] [diff] [review]
update

sr=nelsonb
Attachment #128201 - Flags: superreview+
(Assignee)

Comment 11

15 years ago
Checking in Makefile;
/cvsroot/mozilla/security/nss/lib/freebl/Makefile,v  <--  Makefile
new revision: 1.49; previous revision: 1.48
done
Assignee: wtc → jpierre
(Assignee)

Comment 12

15 years ago
fixed
Status: NEW → RESOLVED
Last Resolved: 15 years ago
Resolution: --- → FIXED

Comment 13

15 years ago
Can a NSS person get this fix on the NSS_CLIENT_TAG?
Blocks: 215224
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 14

15 years ago
Stephen,

This bug was fixed on the tip of NSS - the yet unreleased NSS version 3.9. I
have updated the target milestone to reflect this.
 
The NSS_CLIENT_TAG is based on NSS version 3.8 . To get this patch into version
3.8 :
- please open a separate bug with a target milestone of NSS 3.8
- attach the patch
- cc me
- send email to drivers@mozilla.org for approval about the new bug, since the
Mozilla trunk is currently closed

Closing this bug.
Status: REOPENED → RESOLVED
Last Resolved: 15 years ago15 years ago
Resolution: --- → FIXED
Target Milestone: --- → 3.9

Updated

15 years ago
No longer blocks: 215224
You need to log in before you can comment on or make changes to this bug.