Closed
Bug 468701
Opened 16 years ago
Closed 15 years ago
No debug symbols for NSS libraries on Mac & Linux
Categories
(Release Engineering :: General, defect, P2)
Tracking
(status1.9.2 beta1-fixed, status1.9.1 .4-fixed)
RESOLVED
FIXED
Tracking | Status | |
---|---|---|
status1.9.2 | --- | beta1-fixed |
status1.9.1 | --- | .4-fixed |
People
(Reporter: ted, Assigned: nthomas)
References
(Depends on 1 open bug)
Details
(Whiteboard: [sg:want P4])
Attachments
(4 files)
27.14 KB,
patch
|
bhearsum
:
review+
|
Details | Diff | Splinter Review |
1.16 KB,
patch
|
ted
:
review+
ted
:
approval1.9.2+
dveditz
:
approval1.9.1.4+
|
Details | Diff | Splinter Review |
2.18 KB,
patch
|
mozilla
:
review+
|
Details | Diff | Splinter Review |
2.23 KB,
patch
|
bhearsum
:
review+
nthomas
:
checked-in+
|
Details | Diff | Splinter Review |
Apparently at some point we stopped getting symbols for NSS libraries on Mac (and maybe Linux?). I'm pretty sure this used to work, but I could be wrong. We export CFLAGS/CXXFLAGS="-g", which is supposed to make this work, but maybe it broke. (Or never worked?)
In practice, Firefox 3.0.x has "public" symbols from NSS dylibs, e.g. http://symbols.mozilla.org/symbols_ffx/firefox-3.0.12pre-Darwin-2009052104-symbols.txt --> http://symbols.mozilla.org/symbols_ffx/libnss3.dylib/4CCF5CE9BCCD502D8439E3B6594B91DE0/libnss3.dylib.sym or http://symbols.mozilla.org/symbols_ffx/libnss3.dylib/C7FDC39094F8B8997EE85B78997DB82E0/libnss3.dylib.sym Firefox 3.5.x has no symbols at all (last time I looked the module ID was all 0s, but today there's a real id, assuming I found the right tinderbox and log for the nightly build to look up the buildid), e.g. http://symbols.mozilla.org/symbols_ffx/firefox-3.5pre-Darwin-20090521030957-symbols.txt --> http://symbols.mozilla.org/symbols_ffx/libnss3.dylib/3E488CB1BA8E5D7943CA831B7C6CC2630/libnss3.dylib.sym or http://symbols.mozilla.org/symbols_ffx/libnss3.dylib/78879DD1CE2AA57BCB26E55F5A02C19C0/libnss3.dylib.sym I'm sort-of flailing my way around the NSS build system, but I can read a build logfile (this is from the 21st's Camino 2.0b3pre/Gecko 1.9.0.x nightly, http://tinderbox.mozilla.org/showlog.cgi?log=Camino/1242889260.1242891819.31962.gz&fulltext=1) The compile command for cmsarray.c (the first file compiled "in" libsmime3.dylib, one of the affected NSS dylibs) looks like this: Thu May 21 00:15:33 PDT 2009 cd smime; /usr/bin/make -j1 libs gcc-4.0 -arch ppc -o /builds/tinderbox/Cm2-M1.9/Darwin_8.10.2_Depend/build/ppc/nss/smime/cmsarray.o -c -Oz -fPIC -Dppc -Wmost -fpascal-strings -fno-common -pipe -DDARWIN -DHAVE_STRERROR -DHAVE_BSD_FLOCK -isysroot /Developer/SDKs/MacOSX10.4u.sdk -DXP_UNIX -UDEBUG -DNDEBUG -DNSS_ENABLE_ECC -DUSE_UTIL_DIRECTLY [lots of -l paths omitted] cmsarray.c Observe the total lack of a -g anything (in Camino's case, we export CFLAGS/CXXFLAGS=-gdwarf-2 on our release tinderbox, http://mxr.mozilla.org/seamonkey/source/tools/tinderbox-configs/camino/macosx/mozconfig#7). Flailing around the NSS build system using mxr ("seamonkey" root, so what's being pulled by Gecko 1.9.0.x and my Camino build logfile), I see 1) in Darwin.mk, OS_CFLAGS defined as http://mxr.mozilla.org/seamonkey/source/security/coreconf/Darwin.mk#104 (accounting for everything from "-fPIC" through the 10.4u SDK in my example above) 2) CFLAGS ultimately defined, *I think*, in command.mk http://mxr.mozilla.org/seamonkey/source/security/coreconf/command.mk#48 as OS_CFLAGS with the optimize flag and some other defines and includes postpended, and 3) the NSS compile command ultimately defined here, I think: http://mxr.mozilla.org/seamonkey/source/security/coreconf/rules.mk#284 (again with nothing else interesting for us). I didn't see anything else included by http://mxr.mozilla.org/seamonkey/source/security/coreconf/config.mk that seemed to look for any existing CFLAGS exports (or otherwise really define them for our purposes), i.e. no "CFLAGS += foo" or "bar = $(CFLAGS)" or "baz += $(CFLAGS)". Unless I'm missing something--and again, I certainly could be; I'm really flailing around in the NSS build system--that seems to be our problem.
(Er, in case anyone decides to go looking at Camino symbol files to compare and gets confused.... In spite of being "-gdwarf-2", Camino on 1.9.0.x still has public NSS symbols just like Firefox 3.0.x on 1.9.0.x [which is "-g -gfull" instead] because Camino has a forked symbol generator script that has fallback support when there are no DWARF symbols. Firefox 3.5.x on 1.9.1.x, which is also "-gdwarf-2", has no NSS symbols precisely because of this difference in the symbol generator. Even more confused? Sorry :( )
Reporter | ||
Comment 3•15 years ago
|
||
Ok, so the reality is that we've *never* had real debug symbols for NSS on Mac/Linux, but we used to have public symbols before the switch to DWARF. Does that sound correct? For Windows, we do have real NSS symbols on 1.9.0.x: http://symbols.mozilla.org/firefox/nss3.pdb/09AA70BD289F42579CB3592DE73ED0161/nss3.sym but not on 1.9.1: http://symbols.mozilla.org/firefox/firefox-3.5b4-WINNT-20090423204732-symbols.txt (note the complete lack of NSS in there) so either something changed in the Makefile where we build NSS: http://mxr.mozilla.org/mozilla-central/source/security/manager/Makefile.in or the difference is due to setting MOZ_DEBUG_SYMBOLS=1 in the mozconfig: http://hg.mozilla.org/build/buildbot-configs/file/d7cd3e71df36/mozilla2/win32/mozilla-1.9.1/release/mozconfig#l10 versus setting it in the environment in tinderbox: http://mxr.mozilla.org/mozilla/source/tools/tinderbox/build-seamonkey-util.pl#460 For the former, it will only be set during the configure phase (although we do propogate it into autoconf.mk: http://mxr.mozilla.org/mozilla-central/source/config/autoconf.mk.in#595 , that won't make it to NSS). For the latter, it will propogate all the way down.
(In reply to comment #3) > Ok, so the reality is that we've *never* had real debug symbols for NSS on > Mac/Linux, but we used to have public symbols before the switch to DWARF. Does > that sound correct? That's definitely correct for Mac OS X. On Linux, it looks like there are no symbols at all on 1.9.0 (with "-gstabs+") and 1.9.1; I assume the problem persists in m-c (I can't actually find the 1.9.1 mozconfig, but m-c is still "-gstabs+"). Today's Linux 1.9.0 nightly shows a complete lack of .so for NSS: http://symbols.mozilla.org/symbols_ffx/firefox-3.0.12pre-Linux-2009052204-symbols.txt and the build log http://tinderbox.mozilla.org/showlog.cgi?log=Firefox3.0/1242990180.1242992461.24023.gz&fulltext=1 has 10 pairs of Stab section not found. Failed to write symbol file. (so unless NSS consists of 10 libs on Linux, Linux is probably missing more symbols) The 3.5pre nightly log shows the same: http://tinderbox.mozilla.org/showlog.cgi?log=Firefox3.5/1242986520.1242989081.17954.gz&fulltext=1 Bug 458553 has now fixed the Windows case, correct?
Reporter | ||
Comment 5•15 years ago
|
||
(In reply to comment #4) > On Linux, it looks like there are no symbols at all on 1.9.0 (with "-gstabs+") > and 1.9.1; I assume the problem persists in m-c (I can't actually find the > 1.9.1 mozconfig, but m-c is still "-gstabs+"). Linux is still using stabs, yes. > Bug 458553 has now fixed the Windows case, correct? It looks so, yes. The Mac/Linux case will need a different fix, though.
Reporter | ||
Comment 7•15 years ago
|
||
When bug 498509 lands in NSS and then on trunk, we'll need to set MOZ_DEBUG_SYMBOLS=1 in the Linux/OS X nightly/release builder mozconfigs.
Component: Build Config → Release Engineering
Product: Core → mozilla.org
QA Contact: build-config → release
Version: Trunk → other
Reporter | ||
Comment 8•15 years ago
|
||
Actually we could just do it proactively, since it's harmless without that bug.
Comment 9•15 years ago
|
||
Yes, please do it for Linux and OS/X too
Assignee | ||
Updated•15 years ago
|
Assignee: nobody → nthomas
Priority: -- → P2
Summary: no debug symbols for NSS libraries → No debug symbols for NSS libraries on Mac & Linux
Assignee | ||
Comment 11•15 years ago
|
||
This sets MOZ_DEBUG_SYMBOLS on the world, with the exception of codecoverage and xulrunner. We don't need symbols there I don't think. The symlinking might appear to make this an incomplete set, but grep follows the symlinks and says there are no gaps. This patch applies on top of attachment 392198 [details] [diff] [review] in bug 506401.
Attachment #392421 -
Flags: review?(bhearsum)
Comment 12•15 years ago
|
||
Wouldn't we be better off if security/manager/Makefile.in set MOZ_DEBUG_SYMBOLS automatically based on the --enable-debugger-info-modules configure flag?
Updated•15 years ago
|
Attachment #392421 -
Flags: review?(bhearsum) → review+
Comment 13•15 years ago
|
||
Comment on attachment 392421 [details] [diff] [review] Go nuts with MOZ_DEBUG_SYMBOLS These ones are fine, we probably want to do this on the tryserver, too though. And what about XULRunner and mobile?
Assignee | ||
Comment 14•15 years ago
|
||
(In reply to comment #12) > Wouldn't we be better off if security/manager/Makefile.in set MOZ_DEBUG_SYMBOLS > automatically based on the --enable-debugger-info-modules configure flag? I don't think that would get us symbols on release builds, but it agree it would be nice to set this globally somewhere.
Assignee | ||
Comment 15•15 years ago
|
||
Comment on attachment 392421 [details] [diff] [review] Go nuts with MOZ_DEBUG_SYMBOLS As checked in: http://hg.mozilla.org/build/buildbot-configs/rev/6ecca6648817 Carried over r+ to make similar changes to */*/mozilla-1.9.2/release and */win32/mozilla-1.9.2/*, which were created since the initial patch. The situation with XULRunner is that we don't process crashes on Socorro, and the symbol server is no good because we're only getting MOZCRT19.pdb currently. I can file a followup bug in Core:Build Config if people want that to get resolved, and we can add the lines to the mozconfigs then. For mobile-browser and desktop mobile builds we don't call 'make buildsymbols' at all, and I'd prefer to not have dead lines in the mozconfig. S'ok ?
Assignee | ||
Comment 16•15 years ago
|
||
Looping around to verify the mozconfig change is working it appears that it's .... not. Today's Mac mozilla-central nightly produced http://symbols.mozilla.org/firefox/firefox-3.7a1pre-Darwin-20090831030956-symbols.txt and refers to http://symbols.mozilla.org/firefox/libnss3.dylib/53AB990CAD209A01FD8478876B84CA0E0/libnss3.dylib.sym which only has PUBLIC declarations. m-c has already merged NSS to pick up bug 498509 http://hg.mozilla.org/mozilla-central/file/beecc75d1001/security/coreconf/Darwin.mk#l116 but there's no "-gdwarf-2 -gfull" in the log, eg gcc-4.0 -arch ppc -o /builds/moz2_slave/mozilla-central-macosx-nightly/build/obj-firefox/ppc/nss/dbm/db.o -c -O2 -fPIC -Dppc -Wmost -fpascal-strings -fno-common -pipe -DDARWIN -DHAVE_STRERROR -DHAVE_BSD_FLOCK -isysroot /Developer/SDKs/MacOSX10.4u.sdk -DXP_UNIX -UDEBUG -DNDEBUG -DNSS_ENABLE_ECC -DUSE_UTIL_DIRECTLY -DSTDC_HEADERS -DHAVE_STRERROR -DHAVE_SNPRINTF -DMEMMOVE -D__DBINTERFACE_PRIVATE -I/builds/moz2_slave/mozilla-central-macosx-nightly/build/obj-firefox/ppc/dist/include/nspr -I/builds/moz2_slave/mozilla-central-macosx-nightly/build/obj-firefox/ppc/dist/include -I/builds/moz2_slave/mozilla-central-macosx-nightly/build/obj-firefox/ppc/dist/public/dbm -I/builds/moz2_slave/mozilla-central-macosx-nightly/build/obj-firefox/ppc/dist/private/dbm -I/builds/moz2_slave/mozilla-central-macosx-nightly/build/obj-firefox/ppc/dist/include -I../../../dbm/include ../../../dbm/src/db.c The equivalent Linux build http://symbols.mozilla.org/firefox/firefox-3.7a1pre-Linux-20090831031028-symbols.txt refers to none of the nss libraries. The call there is /tools/gcc/bin/gcc -o /builds/moz2_slave/mozilla-central-linux-nightly/build/obj-firefox/nss/dbm/db.o -c -O2 -fPIC -DLINUX1_2 -Di386 -D_XOPEN_SOURCE -DLINUX2_1 -ansi -Wall -Werror-implicit-function-declaration -Wno-switch -pipe -DLINUX -Dlinux -D_POSIX_SOURCE -D_BSD_SOURCE -DHAVE_STRERROR -DXP_UNIX -UDEBUG -DNDEBUG -D_REENTRANT -DNSS_ENABLE_ECC -DUSE_UTIL_DIRECTLY -DSTDC_HEADERS -DHAVE_STRERROR -DHAVE_SNPRINTF -DHAVE_SYS_CDEFS_H -DMEMMOVE -D__DBINTERFACE_PRIVATE -I/builds/moz2_slave/mozilla-central-linux-nightly/build/obj-firefox/dist/include/nspr -I/builds/moz2_slave/mozilla-central-linux-nightly/build/obj-firefox/dist/include -I/builds/moz2_slave/mozilla-central-linux-nightly/build/obj-firefox/dist/public/dbm -I/builds/moz2_slave/mozilla-central-linux-nightly/build/obj-firefox/dist/private/dbm -I/builds/moz2_slave/mozilla-central-linux-nightly/build/obj-firefox/dist/include -I../../../dbm/include ../../../dbm/src/db.c
Reporter | ||
Comment 17•15 years ago
|
||
Oh, I think I know why this is: http://mxr.mozilla.org/mozilla-central/source/security/manager/Makefile.in#211 That "export MOZ_DEBUG_SYMBOLS" is inside a WINNT block. It just needs to be moved outside of that.
Assignee | ||
Comment 18•15 years ago
|
||
Attachment #398017 -
Flags: review?(ted.mielczarek)
Reporter | ||
Comment 19•15 years ago
|
||
Comment on attachment 398017 [details] [diff] [review] Export MOZ_DEBUG_SYMBOLS on all platforms +# Expose MOZ_DEBUG_SYMBOLS to get NSS symbols in crash stacks from +# released builds I'd leave out "crash stacks" here, just say that setting that makes NSS produce debug symbols.
Attachment #398017 -
Flags: review?(ted.mielczarek) → review+
Assignee | ||
Comment 20•15 years ago
|
||
Comment on attachment 398017 [details] [diff] [review] Export MOZ_DEBUG_SYMBOLS on all platforms Tweaked the comment on checkin: http://hg.mozilla.org/mozilla-central/rev/75af94f85a98 Lets make sure this works on m-c and then figure out the other branches.
Reporter | ||
Updated•15 years ago
|
Attachment #398017 -
Flags: approval1.9.2+
Reporter | ||
Comment 21•15 years ago
|
||
Comment on attachment 398017 [details] [diff] [review] Export MOZ_DEBUG_SYMBOLS on all platforms presumptive a=me if today's builds look good. Did that NSS fix land on 1.9.1? If so we should land this there too.
Assignee | ||
Comment 22•15 years ago
|
||
Looking good. Mac m-c nightly: http://symbols.mozilla.org/firefox/firefox-3.7a1pre-Darwin-20090902030655-symbols.txt http://symbols.mozilla.org/firefox/libnss3.dylib/9CD50B5E98BB37A459A0DC000D2EDB0A0/libnss3.dylib.sym Linux m-c nightly: http://symbols.mozilla.org/firefox/firefox-3.7a1pre-Linux-20090902030656-symbols.txt http://symbols.mozilla.org/firefox/libnss3.so/29E7C29C6F749658F5645807A67CD0760/libnss3.so.sym (In reply to comment #21) > Did that NSS fix land on 1.9.1? If so we should land this there too. 1.9.1 still has NSS 3.12.3, one point release too few to pick up bug 498509. There are plans to upgrade it though, see bug 504080.
Reporter | ||
Comment 23•15 years ago
|
||
Comment on attachment 398017 [details] [diff] [review] Export MOZ_DEBUG_SYMBOLS on all platforms This should be harmless to land on 1.9.1 in advance of the NSS upgrade, ensuring that we get good NSS symbols when that happens. I'd just grant you a+, but apparently I don't have that power, so I'll play by the rules and request it.
Attachment #398017 -
Flags: approval1.9.1.4?
Comment 24•15 years ago
|
||
This caused build bustage on WinCE 5.0 mozilla-central: cd util; make -j1 libs make[6]: Entering directory `/e/builds/moz2_slave/mozilla-central-wince/build/security/nss/lib/util' nsinstall -m 664 /e/builds/moz2_slave/mozilla-central-wince/build/obj-firefox/nss/nssutil/nssutil.lib e:/builds/moz2_slave/mozilla-central-wince/build/obj-firefox/dist/lib nsinstall -m 775 /e/builds/moz2_slave/mozilla-central-wince/build/obj-firefox/nss/nssutil/nssutil3.dll e:/builds/moz2_slave/mozilla-central-wince/build/obj-firefox/dist/lib nsinstall -m 644 /e/builds/moz2_slave/mozilla-central-wince/build/obj-firefox/nss/nssutil/nssutil3.pdb e:/builds/moz2_slave/mozilla-central-wince/build/obj-firefox/dist/lib make[6]: Leaving directory `/e/builds/moz2_slave/mozilla-central-wince/build/security/nss/lib/util' make[5]: Leaving directory `/e/builds/moz2_slave/mozilla-central-wince/build/security/nss/lib' nsinstall: e:\builds\moz2_slave\mozilla-central-wince\build\obj-firefox\nss\nssutil\nssutil3.pdb: No such file or directory make[6]: *** [libs] Error 3 make[5]: *** [libs] Error 2 http://production-master.build.mozilla.org:8010/builders/WINCE%205.0%20mozilla-central%20build/builds/251/steps/compile/logs/stdio
Comment 25•15 years ago
|
||
Also, I just noticed that the mozconfig patch in this bug is different than the one that was actually landed (in that the one landed adds MOZ_DEBUG_SYMBOLS to the wince configs) - is that intentional?
Reporter | ||
Comment 26•15 years ago
|
||
I think you went a little too nuts with MOZ_DEBUG_SYMBOLS in the previous patch. Apparently that doesn't work right on the WinCE builds (which probably deserves its own bug), so we should drop it from their mozconfigs. Then it should be safe to reland this.
Comment 27•15 years ago
|
||
Did they do a full clobber build on WinCE? If not, then probably no new libs were built, and hence no new pdbs were built.
Comment 28•15 years ago
|
||
(In reply to comment #27) > Did they do a full clobber build on WinCE? If not, then probably no new > libs were built, and hence no new pdbs were built. The nightly build failed, too - which was a complete rebuild.
Reporter | ||
Comment 29•15 years ago
|
||
It's possible there's some weird wince-specific bug there, the wince builds are still using their intermediate tools that pretend to be gcc but call visual C++ internally. We should just stop setting MOZ_DEBUG_SYMBOLS in the wince mozconfigs, and spin that error off into a separate bug.
Updated•15 years ago
|
Whiteboard: [sg:want P4]
Updated•15 years ago
|
Attachment #398017 -
Flags: approval1.9.1.4? → approval1.9.1.4+
Comment 30•15 years ago
|
||
Comment on attachment 398017 [details] [diff] [review] Export MOZ_DEBUG_SYMBOLS on all platforms Approved for 1.9.1.4, a=dveditz for release-drivers Please do file the wince bug mentioned in comment 29
Assignee | ||
Comment 31•15 years ago
|
||
Bug 514295 for the WinCE issue. Not setting that as a dep here.
Assignee | ||
Comment 32•15 years ago
|
||
Attachment #398247 -
Flags: review?(aki)
Updated•15 years ago
|
Attachment #398247 -
Flags: review?(aki) → review+
Assignee | ||
Comment 33•15 years ago
|
||
Comment on attachment 398247 [details] [diff] [review] Comment out MOZ_DEBUG_SYMBOLS for WinCE http://hg.mozilla.org/build/buildbot-configs/rev/a15836392b6e
Assignee | ||
Comment 34•15 years ago
|
||
Comment on attachment 398017 [details] [diff] [review] Export MOZ_DEBUG_SYMBOLS on all platforms Relanded on m-c - http://hg.mozilla.org/mozilla-central/rev/16c141deddba
Assignee | ||
Comment 35•15 years ago
|
||
WinCE is broken in the same way as before, I think because config/autoconf.mk still has MOZ_DEBUG_SYMBOLS=1. I've set a clobber for all slaves.
Assignee | ||
Comment 36•15 years ago
|
||
Comment on attachment 398017 [details] [diff] [review] Export MOZ_DEBUG_SYMBOLS on all platforms Clobber did fix m-c WinCE. http://hg.mozilla.org/releases/mozilla-1.9.2/rev/fffff571cfb0 http://hg.mozilla.org/releases/mozilla-1.9.1/rev/f9e9ccea893d
Assignee | ||
Updated•15 years ago
|
status1.9.1:
--- → .4-fixed
status1.9.2:
--- → beta1-fixed
Reporter | ||
Updated•15 years ago
|
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
Assignee | ||
Comment 37•15 years ago
|
||
Oops, forgot the try server mozconfigs.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Assignee | ||
Comment 38•15 years ago
|
||
Attachment #400671 -
Flags: review?(bhearsum)
Updated•15 years ago
|
Attachment #400671 -
Flags: review?(bhearsum) → review+
Assignee | ||
Updated•15 years ago
|
Attachment #400671 -
Flags: checked-in+
Assignee | ||
Comment 39•15 years ago
|
||
Comment on attachment 400671 [details] [diff] [review] Try server mozconfigs http://hg.mozilla.org/build/buildbot-configs/rev/a32e6149819a and try master updated.
Assignee | ||
Comment 40•15 years ago
|
||
Coop is sorting out XULRunner in bug 514700. Code coverage needs bug 512341 instead.
Status: REOPENED → RESOLVED
Closed: 15 years ago → 15 years ago
Resolution: --- → FIXED
Updated•11 years ago
|
Product: mozilla.org → Release Engineering
Comment 41•10 years ago
|
||
I've seen the following option for output debug symbol ac_add_options --enable-debug-symbols The question is: Why export MOZ_DEBUG_SYMBOLS=1 didn't wipe out?
Flags: needinfo?(nthomas)
Assignee | ||
Comment 42•10 years ago
|
||
Sorry, I don't understand the question. The build system probably changed quite a bit in the last 5 years, we're not exporting MOZ_DEBUG_SYMBOLS in mozconfigs any more.
Flags: needinfo?(nthomas)
Comment 43•10 years ago
|
||
Yeap, that's why I am confusing, because there is still have MOZ_DEBUG_SYMBOLS in some configure files in mozilla source tree, I hope it would be wipe out completely.
Reporter | ||
Comment 44•10 years ago
|
||
This conversation is not relevant to this old, fixed bug.
Comment 45•10 years ago
|
||
I need to file a new bug for removing all the things related to MOZ_DEBUG_SYMBOLS?
You need to log in
before you can comment on or make changes to this bug.
Description
•