323977 - Crash when build runs shlibsign on FreeBSD 5.4

Status: VERIFIED FIXED

(NSS :: Libraries, defect, P2)

Description

[Glenn Randers-Pehrson]

This is a spinoff from bug #317620.

While building seamonkey from trunk on a FreeBSD-5.4 platform, the build dumps core while attempting to run shlibsign.

$ uname -a
FreeBSD [host].org 5.4-RELEASE FreeBSD 5.4-RELEASE #2: Mon Jun 13 08:42:36
   CDT 2005 [user]@[host].org:/usr/obj/usr/src/sys/DELLSMP54  i386

$ gcc -v
Using built-in specs.
Configured with: FreeBSD/i386 system compiler
Thread model: posix
gcc version 3.4.2 [FreeBSD] 200407

$ ld -v
GNU ld version 2.15 [FreeBSD] 2004-05-23

$ ld -V
GNU ld version 2.15 [FreeBSD] 2004-05-23
  Supported emulations:
   elf_i386_fbsd

Stack trace from mozilla/security/nss/cmd/shlibsign/shlibsign.core:

#0  0x280a833f in RNG_RNGInit ()
   from /scratch/glennrp/Mozilla/Mozilla-1.9a1-060117/mozilla/dist/lib/libssl3.s
o
#1  0x280a836e in RNG_RNGInit ()
   from /scratch/glennrp/Mozilla/Mozilla-1.9a1-060117/mozilla/dist/lib/libssl3.s
o
#2  0x280a836e in RNG_RNGInit ()
   from /scratch/glennrp/Mozilla/Mozilla-1.9a1-060117/mozilla/dist/lib/libssl3.s
o
#3  0x280a836e in RNG_RNGInit ()
   from /scratch/glennrp/Mozilla/Mozilla-1.9a1-060117/mozilla/dist/lib/libssl3.s
o

ad infinitum.

Comment 1

[Nelson Bolyard (seldom reads bugmail)]

This means that libfreebl*.so was/were not built with -Bsymbolic.
Since I don't have a FreeBSD system on which to build, let me tell you
the general idea, and ask you to implement it and report the results.

Look at http://lxr.mozilla.org/security/source/security/nss/lib/freebl/Makefile#191

There you will see how this is done on Solaris.  In particular, for GCC,
when using gnu ld, we add some command line args.  e.g.
   MKSHLIB += -Wl,-Bsymbolic,-z,now,-z,text

I suggest adding a section in that same Makefile, at about line 205 (right
after the SunOS section) that looks something like this:

ifeq ($(OS_TARGET),FreeBSD)
   MKSHLIB += -Wl,-Bsymbolic,-z,now,-z,text
endif

and then gmake clean in lib/freebl, and then remake nss.  
This should fix it.  
It MAY cause new unresolved symbols errors in linking libfreebl*.so.
If that happens, show us all the linker errors in linking that DSO.

Comment 2

[Glenn Randers-Pehrson]

Nelson: with your change the build doesn't crash.  Instead, shlibsign generates
some key pairs and then the build proceeds.  I didn't see any complaints about
unsatisfied external references.

Comment 3

[Nelson Bolyard (seldom reads bugmail)]

Attachment: Proposed patch

Is this the change you made?

Comment 4

[Glenn Randers-Pehrson]

I added the lines at Makefile line 359, outside the SunOS block.

Comment 5

[Glenn Randers-Pehrson]

The "-z,text" is ignored by FreeBSD, according to "man ld" and can be omitted.

Comment 6

[Wan-Teh Chang]

Attachment: Patch to reproduce this crash on Linux

I can reproduce this crash on Linux (2.6 kernel) with
this patch.  This patch makes us build shared libraries
on Linux the same way we build shared libraries on
FreeBSD: without using a GNU ld version script to control
exported symbols.  With a debug build of the NSS_3_11_BRANCH,
I get this stack trace:

(gdb) where 10
#0  0x0037b90a in RNG_RNGInit () at loader.c:943
#1  0x0037b93f in RNG_RNGInit () at loader.c:946
#2  0x0037b93f in RNG_RNGInit () at loader.c:946
#3  0x0037b93f in RNG_RNGInit () at loader.c:946
#4  0x0037b93f in RNG_RNGInit () at loader.c:946
#5  0x0037b93f in RNG_RNGInit () at loader.c:946
#6  0x0037b93f in RNG_RNGInit () at loader.c:946
#7  0x0037b93f in RNG_RNGInit () at loader.c:946
#8  0x0037b93f in RNG_RNGInit () at loader.c:946
#9  0x0037b93f in RNG_RNGInit () at loader.c:946
(More stack frames follow...)
(gdb) where -20
#775069 0x0037b93f in RNG_RNGInit () at loader.c:946
#775070 0x0037b93f in RNG_RNGInit () at loader.c:946
#775071 0x0037b93f in RNG_RNGInit () at loader.c:946
#775072 0x0037b93f in RNG_RNGInit () at loader.c:946
#775073 0x0037b93f in RNG_RNGInit () at loader.c:946
#775074 0x0037b93f in RNG_RNGInit () at loader.c:946
#775075 0x0037b93f in RNG_RNGInit () at loader.c:946
#775076 0x0037b93f in RNG_RNGInit () at loader.c:946
#775077 0x0037b93f in RNG_RNGInit () at loader.c:946
#775078 0x0037b93f in RNG_RNGInit () at loader.c:946
#775079 0x0037b93f in RNG_RNGInit () at loader.c:946
#775080 0x00a257e3 in nsc_CommonInitialize (pReserved=0xbffd3b40, isFIPS=0)
    at pkcs11.c:2972
#775081 0x00a259d9 in NSC_Initialize (pReserved=0xbffd3b40) at pkcs11.c:3052
#775082 0x00873ff9 in secmod_ModuleInit (mod=0x8660538,
    alreadyLoaded=0xbffd3ba4) at pk11load.c:150
#775083 0x00874486 in SECMOD_LoadPKCS11Module (mod=0x8660538)
    at pk11load.c:322
#775084 0x0087e9b2 in SECMOD_LoadModule (
    modulespec=0x86602b0 "library= name=\"NSS Internal PKCS #11 Module\" paramet
ers=\"configdir='' certPrefix='' keyPrefix='' secmod='' flags=readOnly,noCertDB,
noModDB,forceOpen,optimizeSpace \" NSS=\"Flags=internal,critical trustO"..., par
ent=0x8660000, recurse=1) at pk11pars.c:323
#775085 0x0087ea26 in SECMOD_LoadModule (
    modulespec=0x865f578 "name=\"NSS Internal Module\" parameters=\"configdir=''
 certPrefix='' keyPrefix='' secmod='' flags=readOnly,noCertDB,noModDB,forceOpen,
optimizeSpace \" NSS=\"flags=internal,moduleDB,moduleDBOnly,critical\"",
    parent=0x0, recurse=1) at pk11pars.c:338
#775086 0x00850706 in nss_Init (configdir=0x8c474e "",
    certPrefix=0x8c474e "", keyPrefix=0x8c474e "", secmodName=0x8c474e "",
    readOnly=1, noCertDB=1, noModDB=1, forceOpen=1, noRootInit=1,
    optimizeSpace=1, noSingleThreadedModules=0,
    allowAlreadyInitializedModules=0, dontFinalizeModules=0) at nssinit.c:467
#775087 0x00850925 in NSS_NoDB_Init (configdir=0x805428a "") at nssinit.c:584
#775088 0x0804b112 in main (argc=4, argv=0xbffd5174) at shlibsign.c:224
(gdb)

The mozilla/security/nss/lib/freebl/loader.c file is
rev. 1.26.2.1.  The relevent code is reproduced here:

   940
   941 SECStatus
   942 RNG_RNGInit(void)
   943 {
   944   if (!vector && PR_SUCCESS != freebl_RunLoaderOnce())
   945       return SECFailure;
   946   return (vector->p_RNG_RNGInit)();
   947 }

With this stack trace, Glenn's stack trace, and the
knowledge that libssl3.so, libsoftokn3.so, and libfreebl.so
all export the symbol RNG_RNGInit and shlibsign is
linked with the NSS libraries in this order:
    -lssl3 -lsmime3 -lnss3
we can conclude that it is the copy of RNG_RNGInit in
libssl3.so (the first in the linking order) that gets
used in all libraries, and this caused the infinite
recursion:
    RNG_RNGInit in libssl3.so
 -> vector->p_RNG_RNGInit (a function pointer variable
    in libfreebl3.so)
 -> RNG_RNGInit (a reference in libfreebl3.so.
                 intended target: RNG_RNGInit in libfreebl3.so.
                 actual target: RNG_RNGInit in libssl3.so)

So we need to make the RNG_RNGInit reference in libfreebl3.so
bind to the definition of RNG_RNGInit in libfreebl3.so.  I'll
attach two fixes that Glenn tested next.

Comment 7

[Wan-Teh Chang]

Attachment: Patch 1: link libfreebl3.so with -Bsymbolic on FreeBSD

If we make this change, we should also use the -Bsymbolic
linker flag on other platforms that use the GNU ld, such
as Linux.

Comment 8

[Wan-Teh Chang]

Attachment: Patch 2: use mapfile files (*.def) on FreeBSD

This patch kills two birds with one stone.  We should
definitely checks in this patch whether we check in
patch 1 or not.

Comment 9

[Nelson Bolyard (seldom reads bugmail)]

Comment on attachment 208983 [details] [diff] [review]
Patch 1: link libfreebl3.so with -Bsymbolic on FreeBSD

r=nelson

Comment 10

[Wan-Teh Chang]

Comment on attachment 208984 [details] [diff] [review]
Patch 2: use mapfile files (*.def) on FreeBSD

Glenn, Mikhail,

Does ld have the --version-script option on all
the FreeBSD versions we need to support?

Comment 11

[Nelson Bolyard (seldom reads bugmail)]

> This patch kills two birds with one stone.  

I found that the documentation for "linker scripts" on numerous platforms
does indicate that "local" symbols will be handled as they would for -Bsymbolic,
that is, a DSO's local symbols will always satisfy that DSO's own references
to those symbols.  

But in the past, we found that that was not true on at least one platform.
On Solaris, we found it necessary to do both a linker script (.def file)
and also use -Bsymbolic to avoid the problem described in this bug.

Personally, I think -Bsymbolic is desirable in a shared lib, because a DSO
generally wants to use its own symbols, whether they are global or local.

I'd suggest we do both, script and -Bsymbolic, on all relevant platforms,
but I have also read that on some platforms, -Bsymbolic requires explicitly
linking with all dependencies, direct and indirect, including (e.g.) -lc.
So, I'd say we should use scripts as widely as possible, and use -Bsymbolic
with freebl on any platform on which it doesn't cause linker problems.

Comment 12

[Glenn Randers-Pehrson]

  From "man ld" on FreeBSD5.4, ld 2.15:

         --version-script=version-scriptfile
           Specify the name of a version script to the linker.  This is  typi-
           cally  used  when  creating  shared libraries ...

Comment 13

[Glenn Randers-Pehrson]

Chang:  I tested both patches separately in a seamonkey build and in a
standalone NSS build on the FreeBSD 5.4 platform and they both worked
(i.e., the builds ran to completion instead of dumping core, and the
resulting seamonkey was useable).  I don't think I am entitled to do
official reviews though.

Comment 14

[Mikhail T.]

My two cents: builds of any software, that uses nspr or nss ought to rely on the devel/nspr and security/nss ports instead of building the software's own versions.

The port of seamonkey should follow the path of firefox and simply LIB_DEPEND on nss.

This not only makes the installs leaner, cleaner, and more secure, but also allows us to leverage the time spent researching and patching those two "cornerstone" packages.

Mozilla really ought to stop bundling them with each application...

Comment 15

[Wan-Teh Chang]

Attachment: Patch 1: link libfreebl3.so with -Bsymbolic, v2

Made the patch applicable to all platforms known
to use the GNU ld.  Added a comment to explain what
why the freebl libraries need to be linked with
-Bsymbolic.

Comment 16

[Glenn Randers-Pehrson]

Mikhail, re comment #14: Looking at configure.in, the default behavior appears
to be already the way you are suggesting.  It only uses the embedded nss and
nspr libraries when it cannot find them on the system, or if the user explicitly
specifies "--without-system-nss" or "--without-system-nspr" to force use of the
embedded libraries.  So maybe all that's needed is some education on
how to build and use these system libraries.

Comment 17

[Mikhail T.]

(In reply to comment #16)
> So maybe all that's needed is some education on
> how to build and use these system libraries.

The easiest of all:

% cd /usr/ports/security/nss
% make
# make test is advised here, but is optional
% make install clean

This will also install devel/nspr, as it is listed as a dependency.

Comment 18

[Glenn Randers-Pehrson]

Easy if you have root permission.  Without it, naturally it doesn't
work.  I assume it's simple enough to put the nss library somewhere in
user space, though.

$ make
===>  Vulnerability check disabled, database not found
=> nss-3.9.2.tar.gz doesn't seem to exist in /usr/ports/distfiles/.
=> /usr/ports/distfiles is not writable by you; cannot fetch.
*** Error code 1

Stop in /usr/ports/security/nss.

Comment 19

[Glenn Randers-Pehrson]

The seamonkey build creates a new libnspr4.so in my $prefix/lib/seamonkey-1.5a,
but according to "ldd seamonkey-bin" it actually uses /usr/local/lib/libnspr4.so.

It also creates libnss3.so and libsoftokn3.so but these don't show up in
the "ldd" listing at all.

It seems that my build script is doing a lot more work than it needs to.

Comment 20

[Mikhail T.]

(In reply to comment #18)
> Easy if you have root permission.  Without it, naturally it doesn't
> work.  I assume it's simple enough to put the nss library somewhere in
> user space, though.

Yes. You can specify an alternative location for the downloaded source codes and an alternative PREFIX:

      make DISTDIR=/home/glennrp/distfiles PREFIX=/home/glennrp build test install

(In reply to comment #19)
> The seamonkey build creates a new libnspr4.so in my
> $prefix/lib/seamonkey-1.5a,
> but according to "ldd seamonkey-bin" it actually uses
> /usr/local/lib/libnspr4.so.

Yes, this is, what I mean, when I say, it is cleaner to use each module's own port.

Comment 21

[Wan-Teh Chang]

Glenn, Mikhail,

We recently added --with-system-nspr and --with-system-nss
support to the Mozilla trunk.  --with-system-nspr already
existed, but it wasn't fully implemented (bug 288647).
--with-system-nss is new (bug 255408).  These two configure
options use the nspr-config and nss-config scripts to get
the locations of the NSPR and NSS headers and libraries.
I'd appreciate it if you could test the --with-system-nspr
and --with-system-nss configure options on FreeBSD.  Since
this issue is off topic for this bug report, if you discover
any problems or want to suggest changes to --with-system-nspr
and --with-system-nss for FreeBSD, please open a new bug report
against product:Core component:Build Config and cc me.

Comment 22

[Nelson Bolyard (seldom reads bugmail)]

Comment on attachment 209007 [details] [diff] [review]
Patch 1: link libfreebl3.so with -Bsymbolic, v2

I don't actually know that this will work on all the named platforms, but I like the comment. :)

Comment 23

[Wan-Teh Chang]

Comment on attachment 208984 [details] [diff] [review]
Patch 2: use mapfile files (*.def) on FreeBSD

Nelson, could you also approve this patch for the
NSS_3_11_BRANCH?  I copied the mapfile (ld version
script) related code from Linux2.6.mk.  This patch
has been tested by Glenn Randers-Pehrson (glennrp).

Comment 24

[Nelson Bolyard (seldom reads bugmail)]

Comment on attachment 208984 [details] [diff] [review]
Patch 2: use mapfile files (*.def) on FreeBSD

Yes.  r=nelson
Glenn, Thanks again for testing these patches!

Comment 25

[Wan-Teh Chang]

I checked in the patch on the NSS trunk (NSS 3.12) and
NSS_3_11_BRANCH (NSS 3.11.1), and moved the NSS_CLIENT_TAG
(Mozilla 1.9 alpha) on the two files.

Checking in coreconf/FreeBSD.mk;
/cvsroot/mozilla/security/coreconf/FreeBSD.mk,v  <--  FreeBSD.mk
new revision: 1.9; previous revision: 1.8
done
Checking in nss/lib/freebl/Makefile;
/cvsroot/mozilla/security/nss/lib/freebl/Makefile,v  <--  Makefile
new revision: 1.71; previous revision: 1.70
done

Checking in coreconf/FreeBSD.mk;
/cvsroot/mozilla/security/coreconf/FreeBSD.mk,v  <--  FreeBSD.mk
new revision: 1.8.2.1; previous revision: 1.8
done
Checking in nss/lib/freebl/Makefile;
/cvsroot/mozilla/security/nss/lib/freebl/Makefile,v  <--  Makefile
new revision: 1.70.2.1; previous revision: 1.70
done

Comment 26

[Glenn Randers-Pehrson]

Re comment #14, #16, #20, and #21: see bug #324283

Comment 27

[Glenn Randers-Pehrson]

Verified that my build on FreeBSD no longer crashes.  Thanks!