703812 - shlibsign gets SIGSEGV on 64bit HP-UX

Reporter

Description

•

13 years ago

The build calls shlibsign to sign libsoftokn3.sl, and crashes due to a seg fault.  Seen on the tip as well as the 3.13.1 release.  But, doesn't appear to happen on 32bit.

Command ran:
sh mozilla/security/nss/cmd/shlibsign/sign.sh "mozilla/dist/HP-UXB.11.11_64_OPT.OBJ" "mozilla/security/nss/cmd/shlibsign/HP-UXB.11.11_64_OPT.OBJ" HP-UX "mozilla/dist/HP-UXB.11.11_64_OPT.OBJ/lib" "mozilla/dist/HP-UXB.11.11_64_OPT.OBJ/lib/libsoftokn3.sl"

Output:
mozilla/security/nss/cmd/shlibsign/HP-UXB.11.11_64_OPT.OBJ/shlibsign -v -i mozilla/dist/HP-UXB.11.11_64_OPT.OBJ/lib/libsoftokn3.sl
moduleSpec configdir='' certPrefix='' keyPrefix='' secmod='' flags=noCertDB, noModDB
Generate a DSA key pair ...
mozilla/security/nss/cmd/shlibsign/sign.sh[38]: 19641 Memory fault(coredump)


Stack from the core:
#0  0xc00000000019de18 in free+0x148 () from /usr/lib/pa20_64/libc.2
#1  0xc00000000038c088 in s_mpv_mul_d+0x118 ()
   from mozilla/dist/HP-UXB.11.11_64_OPT.OBJ/lib/libfreebl3.sl
#2  0xc000000000387fe4 in s_mp_mul_d+0xa4 ()
   from mozilla/dist/HP-UXB.11.11_64_OPT.OBJ/lib/libfreebl3.sl
#3  0xc0000000003894ec in s_mp_div+0x25c ()
   from mozilla/dist/HP-UXB.11.11_64_OPT.OBJ/lib/libfreebl3.sl
#4  0xc000000000383e9c in mp_div+0x1c4 ()
   from mozilla/dist/HP-UXB.11.11_64_OPT.OBJ/lib/libfreebl3.sl
#5  0xc0000000003842e8 in mp_mod+0x68 ()
   from mozilla/dist/HP-UXB.11.11_64_OPT.OBJ/lib/libfreebl3.sl
#6  0xc000000000373c78 in fips186Change_ReduceModQForDSA+0xd0 ()
   from mozilla/dist/HP-UXB.11.11_64_OPT.OBJ/lib/libfreebl3.sl
#7  0xc000000000373e7c in dsa_GenerateGlobalRandomBytes+0xcc ()
   from mozilla/dist/HP-UXB.11.11_64_OPT.OBJ/lib/libfreebl3.sl
#8  0xc0000000003743b4 in DSA_NewRandom+0x74 ()
   from mozilla/dist/HP-UXB.11.11_64_OPT.OBJ/lib/libfreebl3.sl
#9  0xc0000000003744b8 in DSA_NewKey+0x30 ()
   from mozilla/dist/HP-UXB.11.11_64_OPT.OBJ/lib/libfreebl3.sl
#10 0xc0000000002853dc in DSA_NewKey+0x54 ()
   from mozilla/dist/HP-UXB.11.11_64_OPT.OBJ/lib/libsoftokn3.sl
#11 0xc00000000026d820 in NSC_GenerateKeyPair+0xab8 ()
   from mozilla/dist/HP-UXB.11.11_64_OPT.OBJ/lib/libsoftokn3.sl
#12 0x4000000000005e1c in main+0x8cc ()

tnzzbugs

Reporter

Updated

•

13 years ago

Severity: normal → blocker

Priority: -- → P2

rajendra

Comment 1

•

12 years ago

Even i have noticed similur issues for NSS 3.13.4 code base on PA-RISC system.

the error message:

./mozilla/security/nss/cmd/shlibsign/./sign.sh[38]: 10664 Memory fault(coredump)
gmake[2]: *** [../../../../dist/HP-UXB.11.23_64_OPT.OBJ/lib/libsoftokn3.chk] Error 139
gmake[2]: Leaving directory `./nss/mozilla/security/nss/cmd/shlibsign'
gmake[1]: *** [libs] Error 2
gmake[1]: Leaving directory `./nss/mozilla/security/nss/cmd'
gmake: *** [libs] Error 2
=== do_make() FAILED for nss on HP-UXB.11.23_64_OPT.OBJ 

Here is the stack 
(gdb) bt
#0  0xc0000000001a1434 in free+0x29c () from /usr/lib/pa20_64/libc.2
#1  0xc000000002f86268 in s_mpv_mul_d+0x118 ()
   from ./nss/mozilla/security/nss/cmd/shlibsign/../../../../dist/HP-UXB.11.23_64_OPT.OBJ/lib/libfreebl3.sl
#2  0xc000000002f821c4 in s_mp_mul_d+0xa4 ()
   from ./nss/mozilla/security/nss/cmd/shlibsign/../../../../dist/HP-UXB.11.23_64_OPT.OBJ/lib/libfreebl3.sl
#3  0xc000000002f836cc in s_mp_div+0x25c ()
   from ./nss/mozilla/security/nss/cmd/shlibsign/../../../../dist/HP-UXB.11.23_64_OPT.OBJ/lib/libfreebl3.sl
#4  0xc000000002f7e07c in mp_div+0x1c4 ()
   from ./nss/mozilla/security/nss/cmd/shlibsign/../../../../dist/HP-UXB.11.23_64_OPT.OBJ/lib/libfreebl3.sl
#5  0xc000000002f7e4c8 in mp_mod+0x68 ()
   from ./nss/mozilla/security/nss/cmd/shlibsign/../../../../dist/HP-UXB.11.23_64_OPT.OBJ/lib/libfreebl3.sl
#6  0xc000000002f6de58 in fips186Change_ReduceModQForDSA+0xd0 ()
   from ./nss/mozilla/security/nss/cmd/shlibsign/../../../../dist/HP-UXB.11.23_64_OPT.OBJ/lib/libfreebl3.sl
#7  0xc000000002f6e05c in dsa_GenerateGlobalRandomBytes+0xcc ()
   from ./nss/mozilla/security/nss/cmd/shlibsign/../../../../dist/HP-UXB.11.23_64_OPT.OBJ/lib/libfreebl3.sl
#8  0xc000000002f6e594 in DSA_NewRandom+0x74 ()
   from ./nss/mozilla/security/nss/cmd/shlibsign/../../../../dist/HP-UXB.11.23_64_OPT.OBJ/lib/libfreebl3.sl
#9  0xc000000002f6e698 in DSA_NewKey+0x30 ()
   from ./nss/mozilla/security/nss/cmd/shlibsign/../../../../dist/HP-UXB.11.23_64_OPT.OBJ/lib/libfreebl3.sl
#10 0xc000000002e8023c in DSA_NewKey+0x54 ()
   from ./nss/mozilla/security/nss/cmd/shlibsign/../../../../dist/HP-UXB.11.23_64_OPT.OBJ/lib/libsoftokn3.sl
#11 0xc000000002e68804 in NSC_GenerateKeyPair+0x2ec ()
   from ./nss/mozilla/security/nss/cmd/shlibsign/../../../../dist/HP-UXB.11.23_64_OPT.OBJ/lib/libsoftokn3.sl
#12 0x4000000000005e1c in main+0x8cc ()


Please let me know if any one fixed this issue....

rajendra

Comment 2

•

12 years ago

Hi ,

Any inputs on this issues?

rajendra

Comment 3

•

12 years ago

(In reply to rajendra from comment #2)
> Hi ,

Any inputs on this issues?

Ashwani Kadian

Comment 4

•

11 years ago

I am also facing the same problem with NSS 3.14.3 while building for 64 bit. But it works well while building 32 bit code.
Any resolution to this issue or any workaround? Please, update.

Wan-Teh Chang

Comment 5

•

11 years ago

This crash should be easy to track down, but I don't have
an HP-UX PA-RISC computer.

The s_mpv_mul_d function in the crash call stack is in
mozilla/security/nss/lib/freebl/mpi/mpi_hp.c. It has not
changed since year 2001. The code is:

#define MAX_STACK_DIGITS 258
#define MULTACC512_LEN   (512 / MP_DIGIT_BIT)
#define HP_MPY_ADD_FN    (a_len == MULTACC512_LEN ? multacc512 : maxpy_little)

/* c = a * b */
void
s_mpv_mul_d(const mp_digit *a, mp_size a_len, mp_digit b, mp_digit *c)
{
    mp_digit x[MAX_STACK_DIGITS];
    mp_digit *px = x;
    size_t   xSize = 0;

    if (a == c) {
        if (a_len > MAX_STACK_DIGITS) {
            xSize = sizeof(mp_digit) * (a_len + 2);
            px = malloc(xSize);
            if (!px)
                return;
        }
        memcpy(px, a, a_len * sizeof(*a));
        a = px;
    }
    s_mp_setz(c, a_len + 1);
    HP_MPY_ADD_FN(a_len, &b, a, c);
    if (px != x && px) {
        memset(px, 0, xSize);
        free(px);
    }
}

The code related to the free() call looks correct to me.
So I don't know what's wrong. Is it possible for you to
do a debug build and get as much info as you can from the
core dump?

Assignee: nobody → wtc

Status: UNCONFIRMED → ASSIGNED

Ever confirmed: true

Ashwani Kadian

Comment 6

•

11 years ago

If I try building NSS 3.14.3 in debug mode then it goes well and shlibsign does not crash while signing libsoftokn3.sl. Is there anything else that we can try?

Wan-Teh Chang

Comment 7

•

11 years ago

Thank you for the info. In that case we'll need to debug the
optimized build.

You can print the relevant variables before the problematic
free() call:

    if (px != x && px) {
        memset(px, 0, xSize);
+       fprintf(stderr, "px=%p, x=%p, xSize=%lu\n", px, x, (unsigned long)xSize);
        free(px);
    }

Assuming this is a compiler bug, you can rewrite the code to
see if the compiler can handle it better, for example,

-   if (px != x && px) {
+   if (xSize && px) {
        memset(px, 0, xSize);
        free(px);
    }

Note: here we are using the property that xSize != 0 if and only if
px != x.

Ashwani Kadian

Comment 8

•

11 years ago

Making the following change also resulted in same crash :

-   if (px != x && px) {
+   if (xSize && px) {
        memset(px, 0, xSize);
        free(px);
    }

Julien Pierre

Comment 9

•

11 years ago

FYI, building without mpi_hp or the PA-RISC assembly optimizations gets us past this crash problem in shlibsign during the build.

This is done by commenting the lines in the USE_64 section below in nss/lib/freebl/Makefile :

233 ifdef USE_64
234 # this builds for DA2.0W (HP PA 2.0 Wide), the LP64 ABI, using 64-bit digits 
235     MPI_SRCS += mpi_hp.c 
236     ASFILES  += hpma512.s hppa20.s 
237     DEFINES  += -DMP_ASSEMBLY_MULTIPLY -DMP_ASSEMBLY_SQUARE
238 else

Even though mpi_hp.c and the two .s files have not changed in 10 years, it appears that the new calling DSA code is exercising them differently from before.

Commenting out the HP optimizations does not help bug 918697, which is another HP-UX DSA problem.

The Written Word

Comment 10

•

6 years ago

This issue occurs with 3.33 as well.

Franziskus Kiefer [:franziskus]

Updated

•

6 years ago

Priority: P2 → P5

BugBot [:suhaib / :marco/ :calixte]

Comment 11

•

2 years ago

The bug assignee didn't login in Bugzilla in the last 7 months, so the assignee is being reset.

Assignee: wtc → nobody

Status: ASSIGNED → NEW

BugBot [:suhaib / :marco/ :calixte]

Comment 12

•

2 years ago

In the process of migrating remaining bugs to the new severity system, the severity for this bug cannot be automatically determined. Please retriage this bug using the new severity system.

Severity: blocker → --

John Schanck [:jschanck]

Updated

•

2 years ago

Severity: -- → S4

Bugzilla

Quick Search

shlibsign gets SIGSEGV on 64bit HP-UX

Categories

(NSS :: Libraries, defect, P5)

Tracking

(Not tracked)

People

(Reporter: tnzzbugs, Unassigned)

References

Details

Crash Data

Security

(public)

User Story

Description

Updated

Comment 1

Comment 2

Comment 3

Comment 4

Comment 5

Comment 6

Comment 7

Comment 8

Comment 9

Comment 10

Updated

Comment 11

Comment 12

Updated