No debug symbols for NSS libraries on Mac & Linux

RESOLVED FIXED

Status

Release Engineering
General
P2
normal
RESOLVED FIXED
9 years ago
3 years ago

People

(Reporter: ted, Assigned: nthomas)

Tracking

(Depends on: 1 bug)

other
x86
Mac OS X
Dependency tree / graph

Firefox Tracking Flags

(status1.9.2 beta1-fixed, status1.9.1 .4-fixed)

Details

(Whiteboard: [sg:want P4])

Attachments

(4 attachments)

(Reporter)

Description

9 years ago
Apparently at some point we stopped getting symbols for NSS libraries on Mac (and maybe Linux?). I'm pretty sure this used to work, but I could be wrong. We export CFLAGS/CXXFLAGS="-g", which is supposed to make this work, but maybe it broke. (Or never worked?)
In practice,

Firefox 3.0.x has "public" symbols from NSS dylibs, e.g. http://symbols.mozilla.org/symbols_ffx/firefox-3.0.12pre-Darwin-2009052104-symbols.txt --> http://symbols.mozilla.org/symbols_ffx/libnss3.dylib/4CCF5CE9BCCD502D8439E3B6594B91DE0/libnss3.dylib.sym or http://symbols.mozilla.org/symbols_ffx/libnss3.dylib/C7FDC39094F8B8997EE85B78997DB82E0/libnss3.dylib.sym

Firefox 3.5.x has no symbols at all (last time I looked the module ID was all 0s, but today there's a real id, assuming I found the right tinderbox and log  for the nightly build to look up the buildid), e.g. http://symbols.mozilla.org/symbols_ffx/firefox-3.5pre-Darwin-20090521030957-symbols.txt --> http://symbols.mozilla.org/symbols_ffx/libnss3.dylib/3E488CB1BA8E5D7943CA831B7C6CC2630/libnss3.dylib.sym or http://symbols.mozilla.org/symbols_ffx/libnss3.dylib/78879DD1CE2AA57BCB26E55F5A02C19C0/libnss3.dylib.sym

I'm sort-of flailing my way around the NSS build system, but I can read a build logfile (this is from the 21st's Camino 2.0b3pre/Gecko 1.9.0.x nightly, http://tinderbox.mozilla.org/showlog.cgi?log=Camino/1242889260.1242891819.31962.gz&fulltext=1)

The compile command for cmsarray.c (the first file compiled "in" libsmime3.dylib, one of the affected NSS dylibs) looks like this:

Thu May 21 00:15:33 PDT 2009
cd smime; /usr/bin/make -j1 libs
gcc-4.0 -arch ppc -o /builds/tinderbox/Cm2-M1.9/Darwin_8.10.2_Depend/build/ppc/nss/smime/cmsarray.o -c -Oz -fPIC -Dppc -Wmost -fpascal-strings -fno-common -pipe -DDARWIN -DHAVE_STRERROR -DHAVE_BSD_FLOCK -isysroot /Developer/SDKs/MacOSX10.4u.sdk -DXP_UNIX -UDEBUG -DNDEBUG -DNSS_ENABLE_ECC -DUSE_UTIL_DIRECTLY [lots of -l paths omitted] cmsarray.c

Observe the total lack of a -g anything (in Camino's case, we export CFLAGS/CXXFLAGS=-gdwarf-2 on our release tinderbox, http://mxr.mozilla.org/seamonkey/source/tools/tinderbox-configs/camino/macosx/mozconfig#7).

Flailing around the NSS build system using mxr ("seamonkey" root, so what's being pulled by Gecko 1.9.0.x and my Camino build logfile), I see 
1) in Darwin.mk, OS_CFLAGS defined as http://mxr.mozilla.org/seamonkey/source/security/coreconf/Darwin.mk#104 (accounting for everything from "-fPIC" through the 10.4u SDK in my example above)
2) CFLAGS ultimately defined, *I think*, in command.mk http://mxr.mozilla.org/seamonkey/source/security/coreconf/command.mk#48 as OS_CFLAGS with the optimize flag and some other defines and includes postpended, and 
3) the NSS compile command ultimately defined here, I think: http://mxr.mozilla.org/seamonkey/source/security/coreconf/rules.mk#284 (again with nothing else interesting for us).

I didn't see anything else included by http://mxr.mozilla.org/seamonkey/source/security/coreconf/config.mk that seemed to look for any existing CFLAGS exports (or otherwise really define them for our purposes), i.e. no "CFLAGS += foo" or "bar = $(CFLAGS)" or "baz += $(CFLAGS)".

Unless I'm missing something--and again, I certainly could be; I'm really flailing around in the NSS build system--that seems to be our problem.
(Er, in case anyone decides to go looking at Camino symbol files to compare and gets confused....  In spite of being "-gdwarf-2", Camino on 1.9.0.x still has public NSS symbols just like Firefox 3.0.x on 1.9.0.x [which is "-g -gfull" instead]  because Camino has a forked symbol generator script that has fallback support when there are no DWARF symbols.  Firefox 3.5.x on 1.9.1.x, which is also "-gdwarf-2", has no NSS symbols precisely because of this difference in the symbol generator.  Even more confused?  Sorry :( )
(Reporter)

Comment 3

9 years ago
Ok, so the reality is that we've *never* had real debug symbols for NSS on Mac/Linux, but we used to have public symbols before the switch to DWARF. Does that sound correct?

For Windows, we do have real NSS symbols on 1.9.0.x:
http://symbols.mozilla.org/firefox/nss3.pdb/09AA70BD289F42579CB3592DE73ED0161/nss3.sym
but not on 1.9.1:
http://symbols.mozilla.org/firefox/firefox-3.5b4-WINNT-20090423204732-symbols.txt (note the complete lack of NSS in there)

so either something changed in the Makefile where we build NSS:
http://mxr.mozilla.org/mozilla-central/source/security/manager/Makefile.in
or the difference is due to setting MOZ_DEBUG_SYMBOLS=1 in the mozconfig:
http://hg.mozilla.org/build/buildbot-configs/file/d7cd3e71df36/mozilla2/win32/mozilla-1.9.1/release/mozconfig#l10
versus setting it in the environment in tinderbox:
http://mxr.mozilla.org/mozilla/source/tools/tinderbox/build-seamonkey-util.pl#460

For the former, it will only be set during the configure phase (although we do propogate it into autoconf.mk: http://mxr.mozilla.org/mozilla-central/source/config/autoconf.mk.in#595 , that won't make it to NSS). For the latter, it will propogate all the way down.
(In reply to comment #3)
> Ok, so the reality is that we've *never* had real debug symbols for NSS on
> Mac/Linux, but we used to have public symbols before the switch to DWARF. Does
> that sound correct?

That's definitely correct for Mac OS X.

On Linux, it looks like there are no symbols at all on 1.9.0 (with "-gstabs+") and 1.9.1; I assume the problem persists in m-c (I can't actually find the 1.9.1 mozconfig, but m-c is still "-gstabs+").

Today's Linux 1.9.0 nightly shows a complete lack of .so for NSS: 
http://symbols.mozilla.org/symbols_ffx/firefox-3.0.12pre-Linux-2009052204-symbols.txt  and the build log 
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox3.0/1242990180.1242992461.24023.gz&fulltext=1
has 10 pairs of 
Stab section not found.
Failed to write symbol file.
(so unless NSS consists of 10 libs on Linux, Linux is probably missing more symbols)
The 3.5pre nightly log shows the same:
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox3.5/1242986520.1242989081.17954.gz&fulltext=1

Bug 458553 has now fixed the Windows case, correct?
(Reporter)

Comment 5

9 years ago
(In reply to comment #4)
> On Linux, it looks like there are no symbols at all on 1.9.0 (with "-gstabs+")
> and 1.9.1; I assume the problem persists in m-c (I can't actually find the
> 1.9.1 mozconfig, but m-c is still "-gstabs+").

Linux is still using stabs, yes.

> Bug 458553 has now fixed the Windows case, correct?

It looks so, yes. The Mac/Linux case will need a different fix, though.
(Reporter)

Updated

9 years ago
Duplicate of this bug: 496041

Updated

9 years ago
Depends on: 506796
(Reporter)

Comment 7

9 years ago
When bug 498509 lands in NSS and then on trunk, we'll need to set MOZ_DEBUG_SYMBOLS=1 in the Linux/OS X nightly/release builder mozconfigs.
Component: Build Config → Release Engineering
Product: Core → mozilla.org
QA Contact: build-config → release
Version: Trunk → other
(Reporter)

Comment 8

9 years ago
Actually we could just do it proactively, since it's harmless without that bug.
Yes, please do it for Linux and OS/X too
(Assignee)

Updated

9 years ago
Assignee: nobody → nthomas
Priority: -- → P2
Summary: no debug symbols for NSS libraries → No debug symbols for NSS libraries on Mac & Linux
(Assignee)

Updated

9 years ago
Duplicate of this bug: 507833
(Assignee)

Comment 11

9 years ago
Created attachment 392421 [details] [diff] [review]
Go nuts with MOZ_DEBUG_SYMBOLS

This sets MOZ_DEBUG_SYMBOLS on the world, with the exception of codecoverage and xulrunner. We don't need symbols there I don't think. The symlinking might appear to make this an incomplete set, but grep follows the symlinks and says there are no gaps.

This patch applies on top of attachment 392198 [details] [diff] [review] in bug 506401.
Attachment #392421 - Flags: review?(bhearsum)
Blocks: 508212
Wouldn't we be better off if security/manager/Makefile.in set MOZ_DEBUG_SYMBOLS automatically based on the --enable-debugger-info-modules configure flag?
Attachment #392421 - Flags: review?(bhearsum) → review+
Comment on attachment 392421 [details] [diff] [review]
Go nuts with MOZ_DEBUG_SYMBOLS

These ones are fine, we probably want to do this on the tryserver, too though. And what about XULRunner and mobile?
(Assignee)

Comment 14

9 years ago
(In reply to comment #12)
> Wouldn't we be better off if security/manager/Makefile.in set MOZ_DEBUG_SYMBOLS
> automatically based on the --enable-debugger-info-modules configure flag?

I don't think that would get us symbols on release builds, but it agree it would be nice to set this globally somewhere.
(Assignee)

Comment 15

9 years ago
Comment on attachment 392421 [details] [diff] [review]
Go nuts with MOZ_DEBUG_SYMBOLS

As checked in: http://hg.mozilla.org/build/buildbot-configs/rev/6ecca6648817

Carried over r+ to make similar changes to */*/mozilla-1.9.2/release and */win32/mozilla-1.9.2/*, which were created since the initial patch.

The situation with XULRunner is that we don't process crashes on Socorro, and the symbol server is no good because we're only getting MOZCRT19.pdb currently. I can file a followup bug in Core:Build Config if people want that to get resolved, and we can add the lines to the mozconfigs then.

For mobile-browser and desktop mobile builds we don't call 'make buildsymbols' at all, and I'd prefer to not have dead lines in the mozconfig. S'ok ?
(Assignee)

Comment 16

9 years ago
Looping around to verify the mozconfig change is working it appears that it's .... not.

Today's Mac mozilla-central nightly produced
http://symbols.mozilla.org/firefox/firefox-3.7a1pre-Darwin-20090831030956-symbols.txt and refers to 
http://symbols.mozilla.org/firefox/libnss3.dylib/53AB990CAD209A01FD8478876B84CA0E0/libnss3.dylib.sym
which only has PUBLIC declarations.

m-c has already merged NSS to pick up bug 498509
http://hg.mozilla.org/mozilla-central/file/beecc75d1001/security/coreconf/Darwin.mk#l116
but there's no "-gdwarf-2 -gfull" in the log, eg
gcc-4.0 -arch ppc -o /builds/moz2_slave/mozilla-central-macosx-nightly/build/obj-firefox/ppc/nss/dbm/db.o -c -O2 -fPIC -Dppc -Wmost -fpascal-strings -fno-common -pipe -DDARWIN -DHAVE_STRERROR -DHAVE_BSD_FLOCK -isysroot /Developer/SDKs/MacOSX10.4u.sdk -DXP_UNIX -UDEBUG -DNDEBUG -DNSS_ENABLE_ECC -DUSE_UTIL_DIRECTLY -DSTDC_HEADERS -DHAVE_STRERROR -DHAVE_SNPRINTF -DMEMMOVE -D__DBINTERFACE_PRIVATE  -I/builds/moz2_slave/mozilla-central-macosx-nightly/build/obj-firefox/ppc/dist/include/nspr -I/builds/moz2_slave/mozilla-central-macosx-nightly/build/obj-firefox/ppc/dist/include -I/builds/moz2_slave/mozilla-central-macosx-nightly/build/obj-firefox/ppc/dist/public/dbm -I/builds/moz2_slave/mozilla-central-macosx-nightly/build/obj-firefox/ppc/dist/private/dbm -I/builds/moz2_slave/mozilla-central-macosx-nightly/build/obj-firefox/ppc/dist/include  -I../../../dbm/include  ../../../dbm/src/db.c

The equivalent Linux build
http://symbols.mozilla.org/firefox/firefox-3.7a1pre-Linux-20090831031028-symbols.txt refers to none of the nss libraries. The call there is 
/tools/gcc/bin/gcc -o /builds/moz2_slave/mozilla-central-linux-nightly/build/obj-firefox/nss/dbm/db.o -c -O2 -fPIC -DLINUX1_2 -Di386 -D_XOPEN_SOURCE -DLINUX2_1  -ansi -Wall -Werror-implicit-function-declaration -Wno-switch -pipe -DLINUX -Dlinux -D_POSIX_SOURCE -D_BSD_SOURCE -DHAVE_STRERROR -DXP_UNIX -UDEBUG -DNDEBUG -D_REENTRANT -DNSS_ENABLE_ECC -DUSE_UTIL_DIRECTLY -DSTDC_HEADERS -DHAVE_STRERROR -DHAVE_SNPRINTF -DHAVE_SYS_CDEFS_H -DMEMMOVE -D__DBINTERFACE_PRIVATE  -I/builds/moz2_slave/mozilla-central-linux-nightly/build/obj-firefox/dist/include/nspr -I/builds/moz2_slave/mozilla-central-linux-nightly/build/obj-firefox/dist/include -I/builds/moz2_slave/mozilla-central-linux-nightly/build/obj-firefox/dist/public/dbm -I/builds/moz2_slave/mozilla-central-linux-nightly/build/obj-firefox/dist/private/dbm -I/builds/moz2_slave/mozilla-central-linux-nightly/build/obj-firefox/dist/include  -I../../../dbm/include  ../../../dbm/src/db.c
(Reporter)

Comment 17

9 years ago
Oh, I think I know why this is:
http://mxr.mozilla.org/mozilla-central/source/security/manager/Makefile.in#211

That "export MOZ_DEBUG_SYMBOLS" is inside a WINNT block. It just needs to be moved outside of that.
(Assignee)

Comment 18

9 years ago
Created attachment 398017 [details] [diff] [review]
Export MOZ_DEBUG_SYMBOLS on all platforms
Attachment #398017 - Flags: review?(ted.mielczarek)
(Reporter)

Comment 19

9 years ago
Comment on attachment 398017 [details] [diff] [review]
Export MOZ_DEBUG_SYMBOLS on all platforms

+# Expose MOZ_DEBUG_SYMBOLS to get NSS symbols in crash stacks from
+# released builds

I'd leave out "crash stacks" here, just say that setting that makes NSS produce debug symbols.
Attachment #398017 - Flags: review?(ted.mielczarek) → review+
(Assignee)

Comment 20

9 years ago
Comment on attachment 398017 [details] [diff] [review]
Export MOZ_DEBUG_SYMBOLS on all platforms

Tweaked the comment on checkin:
http://hg.mozilla.org/mozilla-central/rev/75af94f85a98

Lets make sure this works on m-c and then figure out the other branches.
(Reporter)

Updated

9 years ago
Attachment #398017 - Flags: approval1.9.2+
(Reporter)

Comment 21

9 years ago
Comment on attachment 398017 [details] [diff] [review]
Export MOZ_DEBUG_SYMBOLS on all platforms

presumptive a=me if today's builds look good.

Did that NSS fix land on 1.9.1? If so we should land this there too.
(Reporter)

Comment 23

9 years ago
Comment on attachment 398017 [details] [diff] [review]
Export MOZ_DEBUG_SYMBOLS on all platforms

This should be harmless to land on 1.9.1 in advance of the NSS upgrade, ensuring that we get good NSS symbols when that happens. I'd just grant you a+, but apparently I don't have that power, so I'll play by the rules and request it.
Attachment #398017 - Flags: approval1.9.1.4?
This caused build bustage on WinCE 5.0 mozilla-central:
cd util; make -j1 libs
make[6]: Entering directory `/e/builds/moz2_slave/mozilla-central-wince/build/security/nss/lib/util'
nsinstall -m 664 /e/builds/moz2_slave/mozilla-central-wince/build/obj-firefox/nss/nssutil/nssutil.lib e:/builds/moz2_slave/mozilla-central-wince/build/obj-firefox/dist/lib
nsinstall -m 775 /e/builds/moz2_slave/mozilla-central-wince/build/obj-firefox/nss/nssutil/nssutil3.dll e:/builds/moz2_slave/mozilla-central-wince/build/obj-firefox/dist/lib
nsinstall -m 644 /e/builds/moz2_slave/mozilla-central-wince/build/obj-firefox/nss/nssutil/nssutil3.pdb e:/builds/moz2_slave/mozilla-central-wince/build/obj-firefox/dist/lib
make[6]: Leaving directory `/e/builds/moz2_slave/mozilla-central-wince/build/security/nss/lib/util'
make[5]: Leaving directory `/e/builds/moz2_slave/mozilla-central-wince/build/security/nss/lib'
nsinstall: e:\builds\moz2_slave\mozilla-central-wince\build\obj-firefox\nss\nssutil\nssutil3.pdb: No such file or directory
make[6]: *** [libs] Error 3
make[5]: *** [libs] Error 2

http://production-master.build.mozilla.org:8010/builders/WINCE%205.0%20mozilla-central%20build/builds/251/steps/compile/logs/stdio
Also, I just noticed that the mozconfig patch in this bug is different than the one that was actually landed (in that the one landed adds MOZ_DEBUG_SYMBOLS to the wince configs) - is that intentional?
(Reporter)

Comment 26

9 years ago
I think you went a little too nuts with MOZ_DEBUG_SYMBOLS in the previous patch. Apparently that doesn't work right on the WinCE builds (which probably deserves its own bug), so we should drop it from their mozconfigs. Then it should be safe to reland this.
Did they do a full clobber build on WinCE? If not, then probably no new 
libs were built, and hence no new pdbs were built.
(In reply to comment #27)
> Did they do a full clobber build on WinCE? If not, then probably no new 
> libs were built, and hence no new pdbs were built.

The nightly build failed, too - which was a complete rebuild.
(Reporter)

Comment 29

9 years ago
It's possible there's some weird wince-specific bug there, the wince builds are still using their intermediate tools that pretend to be gcc but call visual C++ internally. We should just stop setting MOZ_DEBUG_SYMBOLS in the wince mozconfigs, and spin that error off into a separate bug.

Updated

9 years ago
Whiteboard: [sg:want P4]
Attachment #398017 - Flags: approval1.9.1.4? → approval1.9.1.4+
Comment on attachment 398017 [details] [diff] [review]
Export MOZ_DEBUG_SYMBOLS on all platforms

Approved for 1.9.1.4, a=dveditz for release-drivers

Please do file the wince bug mentioned in comment 29
(Assignee)

Comment 31

9 years ago
Bug 514295 for the WinCE issue. Not setting that as a dep here.
(Assignee)

Comment 32

9 years ago
Created attachment 398247 [details] [diff] [review]
Comment out MOZ_DEBUG_SYMBOLS for WinCE
Attachment #398247 - Flags: review?(aki)

Updated

9 years ago
Attachment #398247 - Flags: review?(aki) → review+
(Assignee)

Comment 34

9 years ago
Comment on attachment 398017 [details] [diff] [review]
Export MOZ_DEBUG_SYMBOLS on all platforms

Relanded on m-c - http://hg.mozilla.org/mozilla-central/rev/16c141deddba
(Assignee)

Comment 35

9 years ago
WinCE is broken in the same way as before, I think because config/autoconf.mk still has MOZ_DEBUG_SYMBOLS=1. I've set a clobber for all slaves.
(Assignee)

Updated

9 years ago
status1.9.1: --- → .4-fixed
status1.9.2: --- → beta1-fixed
(Reporter)

Updated

9 years ago
Status: NEW → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → FIXED
(Assignee)

Comment 37

9 years ago
Oops, forgot the try server mozconfigs.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 38

9 years ago
Created attachment 400671 [details] [diff] [review]
Try server mozconfigs
Attachment #400671 - Flags: review?(bhearsum)
Attachment #400671 - Flags: review?(bhearsum) → review+
(Assignee)

Updated

9 years ago
Attachment #400671 - Flags: checked-in+
(Assignee)

Comment 40

9 years ago
Coop is sorting out XULRunner in bug 514700. Code coverage needs bug 512341 instead.
Status: REOPENED → RESOLVED
Last Resolved: 9 years ago9 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering

Comment 41

3 years ago
I've seen the following option for output debug symbol
ac_add_options --enable-debug-symbols

The question is: Why export MOZ_DEBUG_SYMBOLS=1 didn't wipe out?
Flags: needinfo?(nthomas)
(Assignee)

Comment 42

3 years ago
Sorry, I don't understand the question. The build system probably changed quite a bit in the last 5 years, we're not exporting MOZ_DEBUG_SYMBOLS in mozconfigs any more.
Flags: needinfo?(nthomas)

Comment 43

3 years ago
Yeap, that's why I am confusing, because there is still have MOZ_DEBUG_SYMBOLS 
in some configure files in mozilla source tree, I hope it would be wipe out completely.
(Reporter)

Comment 44

3 years ago
This conversation is not relevant to this old, fixed bug.

Comment 45

3 years ago
I need to file a new bug for removing all the things related to MOZ_DEBUG_SYMBOLS?
You need to log in before you can comment on or make changes to this bug.