170379 - HPUX (ia64) Browser crashes on Verisign site key generation

Status: RESOLVED FIXED

(SeaMonkey :: General, defect)

Description

[Martin Lawyer]

This only happens on the IPF (ia64) version of HPUX.

Here are the steps to reproduce the problem discovered while trying to obtain a
digital ID from verisign:

   1. Go to http://www.verisign.com/client/enrollment/index.html and click on
the Enroll Now button at the bottom of the page
   2. Fill out the information on the ensuing form (select 60 day trial period
radio) and click the accept button at the bottom of the page
          * You may have to enter or create a master password at this time

Expected Results:

   1. A dialog should pop up indicating that a private key is being generated
   2. Browser loads a page telling user to go check email

Actual Results:

   1. Browser Crashes
   2. Console displays the following: 

sendsig: useracc failed. 0x200000007ea1dc00 0x00000000005020

Pid 10705 was killed due to failure in writing the signal context - possible
stack overflow.
/opt/netscape7/run-mozilla.sh[38]: 10705 Illegal instruction

Comment 1

[Rick Takeda]

Reassigning QA contact to Scott Chapman.

Comment 2

[Martin Lawyer]

Reassigning to Dan. 

Comment 3

[Jim Dunn]

I saw this...  here is the traceback that I got when i crashed
(always a good thing to paste into the bug if you can get to it)

#0  0x2000000079f33830:0 in mpp_make_prime+0x70 () from ./libsoftokn3.sl
#1  0x2000000079f2eca0:0 in generate_prime+0x150 () from ./libsoftokn3.sl
#2  0x2000000079f2f310:0 in RSA_NewKey+0x4e0 () from ./libsoftokn3.sl
#3  0x2000000079eff340:0 in NSC_GenerateKeyPair+0xe80 () from ./libsoftokn3.sl
#4  0x2000000079e158e0:0 in PK11_GenerateKeyPair+0xd60 () from ./libnss3.sl
#5  0x200000007a0c4640:0 in nsKeygenThread::Run+0x130 () from
/builds/jdunn/oem/mozilla/obj/dist/bin/components/libpipnss.sl
#6  0x200000007a0c3260:0 in nsKeygenThreadRunner+0x30 () from
/builds/jdunn/oem/mozilla/obj/dist/bin/components/libpipnss.sl
#7  0x200000007e8ee780:0 in _pt_root+0x2c0 () from ./libnspr4.sl

Also cc'ing wtc

Comment 4

[Wan-Teh Chang]

Nelson, could you look at this crash on HP-UX/ia64?

Comment 5

[Nelson Bolyard (seldom reads bugmail)]

Looks like another stack overflow when we try to allocate a 32KB array on the 
stack as an automatic array.  We've seen this previously on Macs and OS/2.  
See bug http://bugzilla.mozilla.org/show_bug.cgi?id=115012

I'd say the suggested fixes are, in order:

a) IF there is a way (e.g. some compiler option) to make automatic allocation of 
32KB not overflow the stack then change the compiler flags to make that work.
It works on everything else but old MACs and OS/2, including HPUX-PARisc.

b) In mpp_make_prime the following #if line appears in two places:
#if defined(macintosh) || defined (XP_OS2)
Change it to read
#if defined(macintosh) || defined (XP_OS2) || (defined(HPUX) && defined(ia64))
or whatever is appropriate to detect HPUX on ia64.  That will surely work.

Comment 6

[Wan-Teh Chang]

It may not be a compiler flag but rather the stack size
argument that you pass to PR_CreateThread.  By default
(if you pass 0 as the stack size argument) NSPR's thread
stack size is 64KB, and 32KB is half of that.

Comment 7

[Nelson Bolyard (seldom reads bugmail)]

We probably don't want to double the size of the stack for every thread
just so the occasional RSA key gen will work, so the #if changes may be
the better option.  

While investigating this bug, I was reminded that we (NSS group) never 
did the basic work to "port" mpi to hpux/ia64.  The code is sufficiently
portable that it just worked, mostly.  The makefiles in nss/lib/freebl/mpi 
were never even changed to build the test programs on that platform.  
I think this was largely due to not having the hardware on which to test.  
If the test programs had been run, they probably would have caught this.

If we now have a hpux/ia64 box that is sufficiently idle to do performance 
testing, perhaps we should do that work now.

Comment 8

[Wan-Teh Chang]

Nelson, you can use the HP-UX/ia64 machines listed in
http://haathi.mcom.com/machines.html, such as 'asti'.

I ran all.sh successfully (at least on a debug NSS
build) on HP-UX/ia64 before and all tests passed.  It
is possible that the threads created by the NSS tests
have shallow call stacks so that we do not exceed the
max thread stack size.  I did not port the mpi test
programs in nss/lib/freebl/mpi though.

I think we should malloc that 32K array on all platforms.

Comment 9

[Martin Lawyer]

I've cleared "verona" for Nelson's exclusive use for two days, and sent him the
information he needs for his testing on verona.

Comment 10

[Nelson Bolyard (seldom reads bugmail)]

I've done some testing on a system named verona, and have posted some 
findings in bug 131171.  

Please test the following patch on the version of NSS on the branch you
are using for ia64 work.  I believe it will eliminate your crash.
Please let us know the result here.  I will check this in on the NSS 
trunk if it works for you.  


Index: mpprime.c
===================================================================
RCS file: /cvsroot/mozilla/security/nss/lib/freebl/mpi/mpprime.c,v
retrieving revision 1.19
diff -r1.19 mpprime.c
431c431
< #if defined(macintosh) || defined (XP_OS2)
---
> #if defined(macintosh) || defined (XP_OS2) || defined(HPUX)
573c573
< #if defined(macintosh) || defined(XP_OS2)
---
> #if defined(macintosh) || defined (XP_OS2) || defined(HPUX)

Comment 11

[Martin Lawyer]

Note that "__hpux" and "__ia64" are defined by the compiler on HPUX 11.20/11.22,
so using those is probably a safer way to specify the ia64 version of HPUX
instead of the more generic "HPUX"  (unless this patch should apply to both
PA-risc and ia64). 

Comment 12

[Jim Dunn]

Attachment: new suggested patch so that we only do this on ia64

Comment 13

[Jim Dunn]

my previous posted patch fixes the crash that I was seeing with
the verisign key generator

Comment 14

[Wan-Teh Chang]

Attachment: suggested patch v2

I broke the very long #if into two lines and use
our HPUX macro instead of the compiler-defined
__hpux macro.

I've checked this patch into NSS 3.6.

Comment 15

[Jim Dunn]

Dan is gone... WTC checked in the fix...
marking "fixed"