468701 - No debug symbols for NSS libraries on Mac & Linux

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[(not currently active) Ted Mielczarek]

Apparently at some point we stopped getting symbols for NSS libraries on Mac (and maybe Linux?). I'm pretty sure this used to work, but I could be wrong. We export CFLAGS/CXXFLAGS="-g", which is supposed to make this work, but maybe it broke. (Or never worked?)

Comment 1

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

In practice,

Firefox 3.0.x has "public" symbols from NSS dylibs, e.g. http://symbols.mozilla.org/symbols_ffx/firefox-3.0.12pre-Darwin-2009052104-symbols.txt --> http://symbols.mozilla.org/symbols_ffx/libnss3.dylib/4CCF5CE9BCCD502D8439E3B6594B91DE0/libnss3.dylib.sym or http://symbols.mozilla.org/symbols_ffx/libnss3.dylib/C7FDC39094F8B8997EE85B78997DB82E0/libnss3.dylib.sym

Firefox 3.5.x has no symbols at all (last time I looked the module ID was all 0s, but today there's a real id, assuming I found the right tinderbox and log  for the nightly build to look up the buildid), e.g. http://symbols.mozilla.org/symbols_ffx/firefox-3.5pre-Darwin-20090521030957-symbols.txt --> http://symbols.mozilla.org/symbols_ffx/libnss3.dylib/3E488CB1BA8E5D7943CA831B7C6CC2630/libnss3.dylib.sym or http://symbols.mozilla.org/symbols_ffx/libnss3.dylib/78879DD1CE2AA57BCB26E55F5A02C19C0/libnss3.dylib.sym

I'm sort-of flailing my way around the NSS build system, but I can read a build logfile (this is from the 21st's Camino 2.0b3pre/Gecko 1.9.0.x nightly, http://tinderbox.mozilla.org/showlog.cgi?log=Camino/1242889260.1242891819.31962.gz&fulltext=1)

The compile command for cmsarray.c (the first file compiled "in" libsmime3.dylib, one of the affected NSS dylibs) looks like this:

Thu May 21 00:15:33 PDT 2009
cd smime; /usr/bin/make -j1 libs
gcc-4.0 -arch ppc -o /builds/tinderbox/Cm2-M1.9/Darwin_8.10.2_Depend/build/ppc/nss/smime/cmsarray.o -c -Oz -fPIC -Dppc -Wmost -fpascal-strings -fno-common -pipe -DDARWIN -DHAVE_STRERROR -DHAVE_BSD_FLOCK -isysroot /Developer/SDKs/MacOSX10.4u.sdk -DXP_UNIX -UDEBUG -DNDEBUG -DNSS_ENABLE_ECC -DUSE_UTIL_DIRECTLY [lots of -l paths omitted] cmsarray.c

Observe the total lack of a -g anything (in Camino's case, we export CFLAGS/CXXFLAGS=-gdwarf-2 on our release tinderbox, http://mxr.mozilla.org/seamonkey/source/tools/tinderbox-configs/camino/macosx/mozconfig#7).

Flailing around the NSS build system using mxr ("seamonkey" root, so what's being pulled by Gecko 1.9.0.x and my Camino build logfile), I see 
1) in Darwin.mk, OS_CFLAGS defined as http://mxr.mozilla.org/seamonkey/source/security/coreconf/Darwin.mk#104 (accounting for everything from "-fPIC" through the 10.4u SDK in my example above)
2) CFLAGS ultimately defined, *I think*, in command.mk http://mxr.mozilla.org/seamonkey/source/security/coreconf/command.mk#48 as OS_CFLAGS with the optimize flag and some other defines and includes postpended, and 
3) the NSS compile command ultimately defined here, I think: http://mxr.mozilla.org/seamonkey/source/security/coreconf/rules.mk#284 (again with nothing else interesting for us).

I didn't see anything else included by http://mxr.mozilla.org/seamonkey/source/security/coreconf/config.mk that seemed to look for any existing CFLAGS exports (or otherwise really define them for our purposes), i.e. no "CFLAGS += foo" or "bar = $(CFLAGS)" or "baz += $(CFLAGS)".

Unless I'm missing something--and again, I certainly could be; I'm really flailing around in the NSS build system--that seems to be our problem.

Comment 2

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

(Er, in case anyone decides to go looking at Camino symbol files to compare and gets confused....  In spite of being "-gdwarf-2", Camino on 1.9.0.x still has public NSS symbols just like Firefox 3.0.x on 1.9.0.x [which is "-g -gfull" instead]  because Camino has a forked symbol generator script that has fallback support when there are no DWARF symbols.  Firefox 3.5.x on 1.9.1.x, which is also "-gdwarf-2", has no NSS symbols precisely because of this difference in the symbol generator.  Even more confused?  Sorry :( )

Comment 3

[(not currently active) Ted Mielczarek]

Ok, so the reality is that we've *never* had real debug symbols for NSS on Mac/Linux, but we used to have public symbols before the switch to DWARF. Does that sound correct?

For Windows, we do have real NSS symbols on 1.9.0.x:
http://symbols.mozilla.org/firefox/nss3.pdb/09AA70BD289F42579CB3592DE73ED0161/nss3.sym
but not on 1.9.1:
http://symbols.mozilla.org/firefox/firefox-3.5b4-WINNT-20090423204732-symbols.txt (note the complete lack of NSS in there)

so either something changed in the Makefile where we build NSS:
http://mxr.mozilla.org/mozilla-central/source/security/manager/Makefile.in
or the difference is due to setting MOZ_DEBUG_SYMBOLS=1 in the mozconfig:
http://hg.mozilla.org/build/buildbot-configs/file/d7cd3e71df36/mozilla2/win32/mozilla-1.9.1/release/mozconfig#l10
versus setting it in the environment in tinderbox:
http://mxr.mozilla.org/mozilla/source/tools/tinderbox/build-seamonkey-util.pl#460

For the former, it will only be set during the configure phase (although we do propogate it into autoconf.mk: http://mxr.mozilla.org/mozilla-central/source/config/autoconf.mk.in#595 , that won't make it to NSS). For the latter, it will propogate all the way down.

Comment 4

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

(In reply to comment #3)
> Ok, so the reality is that we've *never* had real debug symbols for NSS on
> Mac/Linux, but we used to have public symbols before the switch to DWARF. Does
> that sound correct?

That's definitely correct for Mac OS X.

On Linux, it looks like there are no symbols at all on 1.9.0 (with "-gstabs+") and 1.9.1; I assume the problem persists in m-c (I can't actually find the 1.9.1 mozconfig, but m-c is still "-gstabs+").

Today's Linux 1.9.0 nightly shows a complete lack of .so for NSS: 
http://symbols.mozilla.org/symbols_ffx/firefox-3.0.12pre-Linux-2009052204-symbols.txt  and the build log 
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox3.0/1242990180.1242992461.24023.gz&fulltext=1
has 10 pairs of 
Stab section not found.
Failed to write symbol file.
(so unless NSS consists of 10 libs on Linux, Linux is probably missing more symbols)
The 3.5pre nightly log shows the same:
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox3.5/1242986520.1242989081.17954.gz&fulltext=1

Bug 458553 has now fixed the Windows case, correct?

Comment 5

[(not currently active) Ted Mielczarek]

(In reply to comment #4)
> On Linux, it looks like there are no symbols at all on 1.9.0 (with "-gstabs+")
> and 1.9.1; I assume the problem persists in m-c (I can't actually find the
> 1.9.1 mozconfig, but m-c is still "-gstabs+").

Linux is still using stabs, yes.

> Bug 458553 has now fixed the Windows case, correct?

It looks so, yes. The Mac/Linux case will need a different fix, though.

Comment 7

[(not currently active) Ted Mielczarek]

When bug 498509 lands in NSS and then on trunk, we'll need to set MOZ_DEBUG_SYMBOLS=1 in the Linux/OS X nightly/release builder mozconfigs.

Comment 8

[(not currently active) Ted Mielczarek]

Actually we could just do it proactively, since it's harmless without that bug.

Comment 9

[Nelson Bolyard (seldom reads bugmail)]

Yes, please do it for Linux and OS/X too

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Go nuts with MOZ_DEBUG_SYMBOLS

This sets MOZ_DEBUG_SYMBOLS on the world, with the exception of codecoverage and xulrunner. We don't need symbols there I don't think. The symlinking might appear to make this an incomplete set, but grep follows the symlinks and says there are no gaps.

This patch applies on top of attachment 392198 [details] [diff] [review] in bug 506401.

Comment 12

[[e-mail]]

Wouldn't we be better off if security/manager/Makefile.in set MOZ_DEBUG_SYMBOLS automatically based on the --enable-debugger-info-modules configure flag?

Comment 13

[[e-mail] (:bhearsum)]

Comment on attachment 392421 [details] [diff] [review]
Go nuts with MOZ_DEBUG_SYMBOLS

These ones are fine, we probably want to do this on the tryserver, too though. And what about XULRunner and mobile?

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #12)
> Wouldn't we be better off if security/manager/Makefile.in set MOZ_DEBUG_SYMBOLS
> automatically based on the --enable-debugger-info-modules configure flag?

I don't think that would get us symbols on release builds, but it agree it would be nice to set this globally somewhere.

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 392421 [details] [diff] [review]
Go nuts with MOZ_DEBUG_SYMBOLS

As checked in: http://hg.mozilla.org/build/buildbot-configs/rev/6ecca6648817

Carried over r+ to make similar changes to */*/mozilla-1.9.2/release and */win32/mozilla-1.9.2/*, which were created since the initial patch.

The situation with XULRunner is that we don't process crashes on Socorro, and the symbol server is no good because we're only getting MOZCRT19.pdb currently. I can file a followup bug in Core:Build Config if people want that to get resolved, and we can add the lines to the mozconfigs then.

For mobile-browser and desktop mobile builds we don't call 'make buildsymbols' at all, and I'd prefer to not have dead lines in the mozconfig. S'ok ?

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Looping around to verify the mozconfig change is working it appears that it's .... not.

Today's Mac mozilla-central nightly produced
http://symbols.mozilla.org/firefox/firefox-3.7a1pre-Darwin-20090831030956-symbols.txt and refers to 
http://symbols.mozilla.org/firefox/libnss3.dylib/53AB990CAD209A01FD8478876B84CA0E0/libnss3.dylib.sym
which only has PUBLIC declarations.

m-c has already merged NSS to pick up bug 498509
http://hg.mozilla.org/mozilla-central/file/beecc75d1001/security/coreconf/Darwin.mk#l116
but there's no "-gdwarf-2 -gfull" in the log, eg
gcc-4.0 -arch ppc -o /builds/moz2_slave/mozilla-central-macosx-nightly/build/obj-firefox/ppc/nss/dbm/db.o -c -O2 -fPIC -Dppc -Wmost -fpascal-strings -fno-common -pipe -DDARWIN -DHAVE_STRERROR -DHAVE_BSD_FLOCK -isysroot /Developer/SDKs/MacOSX10.4u.sdk -DXP_UNIX -UDEBUG -DNDEBUG -DNSS_ENABLE_ECC -DUSE_UTIL_DIRECTLY -DSTDC_HEADERS -DHAVE_STRERROR -DHAVE_SNPRINTF -DMEMMOVE -D__DBINTERFACE_PRIVATE  -I/builds/moz2_slave/mozilla-central-macosx-nightly/build/obj-firefox/ppc/dist/include/nspr -I/builds/moz2_slave/mozilla-central-macosx-nightly/build/obj-firefox/ppc/dist/include -I/builds/moz2_slave/mozilla-central-macosx-nightly/build/obj-firefox/ppc/dist/public/dbm -I/builds/moz2_slave/mozilla-central-macosx-nightly/build/obj-firefox/ppc/dist/private/dbm -I/builds/moz2_slave/mozilla-central-macosx-nightly/build/obj-firefox/ppc/dist/include  -I../../../dbm/include  ../../../dbm/src/db.c

The equivalent Linux build
http://symbols.mozilla.org/firefox/firefox-3.7a1pre-Linux-20090831031028-symbols.txt refers to none of the nss libraries. The call there is 
/tools/gcc/bin/gcc -o /builds/moz2_slave/mozilla-central-linux-nightly/build/obj-firefox/nss/dbm/db.o -c -O2 -fPIC -DLINUX1_2 -Di386 -D_XOPEN_SOURCE -DLINUX2_1  -ansi -Wall -Werror-implicit-function-declaration -Wno-switch -pipe -DLINUX -Dlinux -D_POSIX_SOURCE -D_BSD_SOURCE -DHAVE_STRERROR -DXP_UNIX -UDEBUG -DNDEBUG -D_REENTRANT -DNSS_ENABLE_ECC -DUSE_UTIL_DIRECTLY -DSTDC_HEADERS -DHAVE_STRERROR -DHAVE_SNPRINTF -DHAVE_SYS_CDEFS_H -DMEMMOVE -D__DBINTERFACE_PRIVATE  -I/builds/moz2_slave/mozilla-central-linux-nightly/build/obj-firefox/dist/include/nspr -I/builds/moz2_slave/mozilla-central-linux-nightly/build/obj-firefox/dist/include -I/builds/moz2_slave/mozilla-central-linux-nightly/build/obj-firefox/dist/public/dbm -I/builds/moz2_slave/mozilla-central-linux-nightly/build/obj-firefox/dist/private/dbm -I/builds/moz2_slave/mozilla-central-linux-nightly/build/obj-firefox/dist/include  -I../../../dbm/include  ../../../dbm/src/db.c

Comment 17

[(not currently active) Ted Mielczarek]

Oh, I think I know why this is:
http://mxr.mozilla.org/mozilla-central/source/security/manager/Makefile.in#211

That "export MOZ_DEBUG_SYMBOLS" is inside a WINNT block. It just needs to be moved outside of that.

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Export MOZ_DEBUG_SYMBOLS on all platforms

Comment 19

[(not currently active) Ted Mielczarek]

Comment on attachment 398017 [details] [diff] [review]
Export MOZ_DEBUG_SYMBOLS on all platforms

+# Expose MOZ_DEBUG_SYMBOLS to get NSS symbols in crash stacks from
+# released builds

I'd leave out "crash stacks" here, just say that setting that makes NSS produce debug symbols.

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 398017 [details] [diff] [review]
Export MOZ_DEBUG_SYMBOLS on all platforms

Tweaked the comment on checkin:
http://hg.mozilla.org/mozilla-central/rev/75af94f85a98

Lets make sure this works on m-c and then figure out the other branches.

Comment 21

[(not currently active) Ted Mielczarek]

Comment on attachment 398017 [details] [diff] [review]
Export MOZ_DEBUG_SYMBOLS on all platforms

presumptive a=me if today's builds look good.

Did that NSS fix land on 1.9.1? If so we should land this there too.

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

Looking good.

Mac m-c nightly:
http://symbols.mozilla.org/firefox/firefox-3.7a1pre-Darwin-20090902030655-symbols.txt
http://symbols.mozilla.org/firefox/libnss3.dylib/9CD50B5E98BB37A459A0DC000D2EDB0A0/libnss3.dylib.sym

Linux m-c nightly:
http://symbols.mozilla.org/firefox/firefox-3.7a1pre-Linux-20090902030656-symbols.txt
http://symbols.mozilla.org/firefox/libnss3.so/29E7C29C6F749658F5645807A67CD0760/libnss3.so.sym

(In reply to comment #21)
> Did that NSS fix land on 1.9.1? If so we should land this there too.

1.9.1 still has NSS 3.12.3, one point release too few to pick up bug 498509. There are plans to upgrade it though, see bug 504080.

Comment 23

[(not currently active) Ted Mielczarek]

Comment on attachment 398017 [details] [diff] [review]
Export MOZ_DEBUG_SYMBOLS on all platforms

This should be harmless to land on 1.9.1 in advance of the NSS upgrade, ensuring that we get good NSS symbols when that happens. I'd just grant you a+, but apparently I don't have that power, so I'll play by the rules and request it.

Comment 24

[[e-mail] (:bhearsum)]

This caused build bustage on WinCE 5.0 mozilla-central:
cd util; make -j1 libs
make[6]: Entering directory `/e/builds/moz2_slave/mozilla-central-wince/build/security/nss/lib/util'
nsinstall -m 664 /e/builds/moz2_slave/mozilla-central-wince/build/obj-firefox/nss/nssutil/nssutil.lib e:/builds/moz2_slave/mozilla-central-wince/build/obj-firefox/dist/lib
nsinstall -m 775 /e/builds/moz2_slave/mozilla-central-wince/build/obj-firefox/nss/nssutil/nssutil3.dll e:/builds/moz2_slave/mozilla-central-wince/build/obj-firefox/dist/lib
nsinstall -m 644 /e/builds/moz2_slave/mozilla-central-wince/build/obj-firefox/nss/nssutil/nssutil3.pdb e:/builds/moz2_slave/mozilla-central-wince/build/obj-firefox/dist/lib
make[6]: Leaving directory `/e/builds/moz2_slave/mozilla-central-wince/build/security/nss/lib/util'
make[5]: Leaving directory `/e/builds/moz2_slave/mozilla-central-wince/build/security/nss/lib'
nsinstall: e:\builds\moz2_slave\mozilla-central-wince\build\obj-firefox\nss\nssutil\nssutil3.pdb: No such file or directory
make[6]: *** [libs] Error 3
make[5]: *** [libs] Error 2

http://production-master.build.mozilla.org:8010/builders/WINCE%205.0%20mozilla-central%20build/builds/251/steps/compile/logs/stdio

Comment 25

[[e-mail] (:bhearsum)]

Also, I just noticed that the mozconfig patch in this bug is different than the one that was actually landed (in that the one landed adds MOZ_DEBUG_SYMBOLS to the wince configs) - is that intentional?

Comment 26

[(not currently active) Ted Mielczarek]

I think you went a little too nuts with MOZ_DEBUG_SYMBOLS in the previous patch. Apparently that doesn't work right on the WinCE builds (which probably deserves its own bug), so we should drop it from their mozconfigs. Then it should be safe to reland this.

Comment 27

[Nelson Bolyard (seldom reads bugmail)]

Did they do a full clobber build on WinCE? If not, then probably no new 
libs were built, and hence no new pdbs were built.

Comment 28

[[e-mail] (:bhearsum)]

(In reply to comment #27)
> Did they do a full clobber build on WinCE? If not, then probably no new 
> libs were built, and hence no new pdbs were built.

The nightly build failed, too - which was a complete rebuild.

Comment 29

[(not currently active) Ted Mielczarek]

It's possible there's some weird wince-specific bug there, the wince builds are still using their intermediate tools that pretend to be gcc but call visual C++ internally. We should just stop setting MOZ_DEBUG_SYMBOLS in the wince mozconfigs, and spin that error off into a separate bug.

Comment 30

[Daniel Veditz [:dveditz]]

Comment on attachment 398017 [details] [diff] [review]
Export MOZ_DEBUG_SYMBOLS on all platforms

Approved for 1.9.1.4, a=dveditz for release-drivers

Please do file the wince bug mentioned in comment 29

Comment 31

[Nick Thomas [:nthomas] (UTC+12)]

Bug 514295 for the WinCE issue. Not setting that as a dep here.

Comment 32

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Comment out MOZ_DEBUG_SYMBOLS for WinCE

Comment 33

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 398247 [details] [diff] [review]
Comment out MOZ_DEBUG_SYMBOLS for WinCE

http://hg.mozilla.org/build/buildbot-configs/rev/a15836392b6e

Comment 34

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 398017 [details] [diff] [review]
Export MOZ_DEBUG_SYMBOLS on all platforms

Relanded on m-c - http://hg.mozilla.org/mozilla-central/rev/16c141deddba

Comment 35

[Nick Thomas [:nthomas] (UTC+12)]

WinCE is broken in the same way as before, I think because config/autoconf.mk still has MOZ_DEBUG_SYMBOLS=1. I've set a clobber for all slaves.

Comment 36

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 398017 [details] [diff] [review]
Export MOZ_DEBUG_SYMBOLS on all platforms

Clobber did fix m-c WinCE.

http://hg.mozilla.org/releases/mozilla-1.9.2/rev/fffff571cfb0
http://hg.mozilla.org/releases/mozilla-1.9.1/rev/f9e9ccea893d

Comment 37

[Nick Thomas [:nthomas] (UTC+12)]

Oops, forgot the try server mozconfigs.

Comment 38

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Try server mozconfigs

Comment 39

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 400671 [details] [diff] [review]
Try server mozconfigs

http://hg.mozilla.org/build/buildbot-configs/rev/a32e6149819a
and try master updated.

Comment 40

[Nick Thomas [:nthomas] (UTC+12)]

Coop is sorting out XULRunner in bug 514700. Code coverage needs bug 512341 instead.

Comment 41

[Yonggang Luo]

I've seen the following option for output debug symbol
ac_add_options --enable-debug-symbols

The question is: Why export MOZ_DEBUG_SYMBOLS=1 didn't wipe out?

Comment 42

[Nick Thomas [:nthomas] (UTC+12)]

Sorry, I don't understand the question. The build system probably changed quite a bit in the last 5 years, we're not exporting MOZ_DEBUG_SYMBOLS in mozconfigs any more.

Comment 43

[Yonggang Luo]

Yeap, that's why I am confusing, because there is still have MOZ_DEBUG_SYMBOLS 
in some configure files in mozilla source tree, I hope it would be wipe out completely.

Comment 44

[(not currently active) Ted Mielczarek]

This conversation is not relevant to this old, fixed bug.

Comment 45

[Yonggang Luo]

I need to file a new bug for removing all the things related to MOZ_DEBUG_SYMBOLS?