27.14 KB, patch
|Details | Diff | Splinter Review|
1.16 KB, patch
|Details | Diff | Splinter Review|
2.18 KB, patch
|Details | Diff | Splinter Review|
2.23 KB, patch
|Details | Diff | Splinter Review|
Apparently at some point we stopped getting symbols for NSS libraries on Mac (and maybe Linux?). I'm pretty sure this used to work, but I could be wrong. We export CFLAGS/CXXFLAGS="-g", which is supposed to make this work, but maybe it broke. (Or never worked?)
In practice, Firefox 3.0.x has "public" symbols from NSS dylibs, e.g. http://symbols.mozilla.org/symbols_ffx/firefox-3.0.12pre-Darwin-2009052104-symbols.txt --> http://symbols.mozilla.org/symbols_ffx/libnss3.dylib/4CCF5CE9BCCD502D8439E3B6594B91DE0/libnss3.dylib.sym or http://symbols.mozilla.org/symbols_ffx/libnss3.dylib/C7FDC39094F8B8997EE85B78997DB82E0/libnss3.dylib.sym Firefox 3.5.x has no symbols at all (last time I looked the module ID was all 0s, but today there's a real id, assuming I found the right tinderbox and log for the nightly build to look up the buildid), e.g. http://symbols.mozilla.org/symbols_ffx/firefox-3.5pre-Darwin-20090521030957-symbols.txt --> http://symbols.mozilla.org/symbols_ffx/libnss3.dylib/3E488CB1BA8E5D7943CA831B7C6CC2630/libnss3.dylib.sym or http://symbols.mozilla.org/symbols_ffx/libnss3.dylib/78879DD1CE2AA57BCB26E55F5A02C19C0/libnss3.dylib.sym I'm sort-of flailing my way around the NSS build system, but I can read a build logfile (this is from the 21st's Camino 2.0b3pre/Gecko 1.9.0.x nightly, http://tinderbox.mozilla.org/showlog.cgi?log=Camino/1242889260.1242891819.31962.gz&fulltext=1) The compile command for cmsarray.c (the first file compiled "in" libsmime3.dylib, one of the affected NSS dylibs) looks like this: Thu May 21 00:15:33 PDT 2009 cd smime; /usr/bin/make -j1 libs gcc-4.0 -arch ppc -o /builds/tinderbox/Cm2-M1.9/Darwin_8.10.2_Depend/build/ppc/nss/smime/cmsarray.o -c -Oz -fPIC -Dppc -Wmost -fpascal-strings -fno-common -pipe -DDARWIN -DHAVE_STRERROR -DHAVE_BSD_FLOCK -isysroot /Developer/SDKs/MacOSX10.4u.sdk -DXP_UNIX -UDEBUG -DNDEBUG -DNSS_ENABLE_ECC -DUSE_UTIL_DIRECTLY [lots of -l paths omitted] cmsarray.c Observe the total lack of a -g anything (in Camino's case, we export CFLAGS/CXXFLAGS=-gdwarf-2 on our release tinderbox, http://mxr.mozilla.org/seamonkey/source/tools/tinderbox-configs/camino/macosx/mozconfig#7). Flailing around the NSS build system using mxr ("seamonkey" root, so what's being pulled by Gecko 1.9.0.x and my Camino build logfile), I see 1) in Darwin.mk, OS_CFLAGS defined as http://mxr.mozilla.org/seamonkey/source/security/coreconf/Darwin.mk#104 (accounting for everything from "-fPIC" through the 10.4u SDK in my example above) 2) CFLAGS ultimately defined, *I think*, in command.mk http://mxr.mozilla.org/seamonkey/source/security/coreconf/command.mk#48 as OS_CFLAGS with the optimize flag and some other defines and includes postpended, and 3) the NSS compile command ultimately defined here, I think: http://mxr.mozilla.org/seamonkey/source/security/coreconf/rules.mk#284 (again with nothing else interesting for us). I didn't see anything else included by http://mxr.mozilla.org/seamonkey/source/security/coreconf/config.mk that seemed to look for any existing CFLAGS exports (or otherwise really define them for our purposes), i.e. no "CFLAGS += foo" or "bar = $(CFLAGS)" or "baz += $(CFLAGS)". Unless I'm missing something--and again, I certainly could be; I'm really flailing around in the NSS build system--that seems to be our problem.
(Er, in case anyone decides to go looking at Camino symbol files to compare and gets confused.... In spite of being "-gdwarf-2", Camino on 1.9.0.x still has public NSS symbols just like Firefox 3.0.x on 1.9.0.x [which is "-g -gfull" instead] because Camino has a forked symbol generator script that has fallback support when there are no DWARF symbols. Firefox 3.5.x on 1.9.1.x, which is also "-gdwarf-2", has no NSS symbols precisely because of this difference in the symbol generator. Even more confused? Sorry :( )
Ok, so the reality is that we've *never* had real debug symbols for NSS on Mac/Linux, but we used to have public symbols before the switch to DWARF. Does that sound correct? For Windows, we do have real NSS symbols on 1.9.0.x: http://symbols.mozilla.org/firefox/nss3.pdb/09AA70BD289F42579CB3592DE73ED0161/nss3.sym but not on 1.9.1: http://symbols.mozilla.org/firefox/firefox-3.5b4-WINNT-20090423204732-symbols.txt (note the complete lack of NSS in there) so either something changed in the Makefile where we build NSS: http://mxr.mozilla.org/mozilla-central/source/security/manager/Makefile.in or the difference is due to setting MOZ_DEBUG_SYMBOLS=1 in the mozconfig: http://hg.mozilla.org/build/buildbot-configs/file/d7cd3e71df36/mozilla2/win32/mozilla-1.9.1/release/mozconfig#l10 versus setting it in the environment in tinderbox: http://mxr.mozilla.org/mozilla/source/tools/tinderbox/build-seamonkey-util.pl#460 For the former, it will only be set during the configure phase (although we do propogate it into autoconf.mk: http://mxr.mozilla.org/mozilla-central/source/config/autoconf.mk.in#595 , that won't make it to NSS). For the latter, it will propogate all the way down.
(In reply to comment #3) > Ok, so the reality is that we've *never* had real debug symbols for NSS on > Mac/Linux, but we used to have public symbols before the switch to DWARF. Does > that sound correct? That's definitely correct for Mac OS X. On Linux, it looks like there are no symbols at all on 1.9.0 (with "-gstabs+") and 1.9.1; I assume the problem persists in m-c (I can't actually find the 1.9.1 mozconfig, but m-c is still "-gstabs+"). Today's Linux 1.9.0 nightly shows a complete lack of .so for NSS: http://symbols.mozilla.org/symbols_ffx/firefox-3.0.12pre-Linux-2009052204-symbols.txt and the build log http://tinderbox.mozilla.org/showlog.cgi?log=Firefox3.0/1242990180.1242992461.24023.gz&fulltext=1 has 10 pairs of Stab section not found. Failed to write symbol file. (so unless NSS consists of 10 libs on Linux, Linux is probably missing more symbols) The 3.5pre nightly log shows the same: http://tinderbox.mozilla.org/showlog.cgi?log=Firefox3.5/1242986520.1242989081.17954.gz&fulltext=1 Bug 458553 has now fixed the Windows case, correct?
(In reply to comment #4) > On Linux, it looks like there are no symbols at all on 1.9.0 (with "-gstabs+") > and 1.9.1; I assume the problem persists in m-c (I can't actually find the > 1.9.1 mozconfig, but m-c is still "-gstabs+"). Linux is still using stabs, yes. > Bug 458553 has now fixed the Windows case, correct? It looks so, yes. The Mac/Linux case will need a different fix, though.
When bug 498509 lands in NSS and then on trunk, we'll need to set MOZ_DEBUG_SYMBOLS=1 in the Linux/OS X nightly/release builder mozconfigs.
Component: Build Config → Release Engineering
Product: Core → mozilla.org
QA Contact: build-config → release
Version: Trunk → other
Actually we could just do it proactively, since it's harmless without that bug.
Yes, please do it for Linux and OS/X too
Assignee: nobody → nthomas
Priority: -- → P2
Summary: no debug symbols for NSS libraries → No debug symbols for NSS libraries on Mac & Linux
Created attachment 392421 [details] [diff] [review] Go nuts with MOZ_DEBUG_SYMBOLS This sets MOZ_DEBUG_SYMBOLS on the world, with the exception of codecoverage and xulrunner. We don't need symbols there I don't think. The symlinking might appear to make this an incomplete set, but grep follows the symlinks and says there are no gaps. This patch applies on top of attachment 392198 [details] [diff] [review] in bug 506401.
Attachment #392421 - Flags: review?(bhearsum)
Wouldn't we be better off if security/manager/Makefile.in set MOZ_DEBUG_SYMBOLS automatically based on the --enable-debugger-info-modules configure flag?
Attachment #392421 - Flags: review?(bhearsum) → review+
Comment on attachment 392421 [details] [diff] [review] Go nuts with MOZ_DEBUG_SYMBOLS These ones are fine, we probably want to do this on the tryserver, too though. And what about XULRunner and mobile?
(In reply to comment #12) > Wouldn't we be better off if security/manager/Makefile.in set MOZ_DEBUG_SYMBOLS > automatically based on the --enable-debugger-info-modules configure flag? I don't think that would get us symbols on release builds, but it agree it would be nice to set this globally somewhere.
Comment on attachment 392421 [details] [diff] [review] Go nuts with MOZ_DEBUG_SYMBOLS As checked in: http://hg.mozilla.org/build/buildbot-configs/rev/6ecca6648817 Carried over r+ to make similar changes to */*/mozilla-1.9.2/release and */win32/mozilla-1.9.2/*, which were created since the initial patch. The situation with XULRunner is that we don't process crashes on Socorro, and the symbol server is no good because we're only getting MOZCRT19.pdb currently. I can file a followup bug in Core:Build Config if people want that to get resolved, and we can add the lines to the mozconfigs then. For mobile-browser and desktop mobile builds we don't call 'make buildsymbols' at all, and I'd prefer to not have dead lines in the mozconfig. S'ok ?
Looping around to verify the mozconfig change is working it appears that it's .... not. Today's Mac mozilla-central nightly produced http://symbols.mozilla.org/firefox/firefox-3.7a1pre-Darwin-20090831030956-symbols.txt and refers to http://symbols.mozilla.org/firefox/libnss3.dylib/53AB990CAD209A01FD8478876B84CA0E0/libnss3.dylib.sym which only has PUBLIC declarations. m-c has already merged NSS to pick up bug 498509 http://hg.mozilla.org/mozilla-central/file/beecc75d1001/security/coreconf/Darwin.mk#l116 but there's no "-gdwarf-2 -gfull" in the log, eg gcc-4.0 -arch ppc -o /builds/moz2_slave/mozilla-central-macosx-nightly/build/obj-firefox/ppc/nss/dbm/db.o -c -O2 -fPIC -Dppc -Wmost -fpascal-strings -fno-common -pipe -DDARWIN -DHAVE_STRERROR -DHAVE_BSD_FLOCK -isysroot /Developer/SDKs/MacOSX10.4u.sdk -DXP_UNIX -UDEBUG -DNDEBUG -DNSS_ENABLE_ECC -DUSE_UTIL_DIRECTLY -DSTDC_HEADERS -DHAVE_STRERROR -DHAVE_SNPRINTF -DMEMMOVE -D__DBINTERFACE_PRIVATE -I/builds/moz2_slave/mozilla-central-macosx-nightly/build/obj-firefox/ppc/dist/include/nspr -I/builds/moz2_slave/mozilla-central-macosx-nightly/build/obj-firefox/ppc/dist/include -I/builds/moz2_slave/mozilla-central-macosx-nightly/build/obj-firefox/ppc/dist/public/dbm -I/builds/moz2_slave/mozilla-central-macosx-nightly/build/obj-firefox/ppc/dist/private/dbm -I/builds/moz2_slave/mozilla-central-macosx-nightly/build/obj-firefox/ppc/dist/include -I../../../dbm/include ../../../dbm/src/db.c The equivalent Linux build http://symbols.mozilla.org/firefox/firefox-3.7a1pre-Linux-20090831031028-symbols.txt refers to none of the nss libraries. The call there is /tools/gcc/bin/gcc -o /builds/moz2_slave/mozilla-central-linux-nightly/build/obj-firefox/nss/dbm/db.o -c -O2 -fPIC -DLINUX1_2 -Di386 -D_XOPEN_SOURCE -DLINUX2_1 -ansi -Wall -Werror-implicit-function-declaration -Wno-switch -pipe -DLINUX -Dlinux -D_POSIX_SOURCE -D_BSD_SOURCE -DHAVE_STRERROR -DXP_UNIX -UDEBUG -DNDEBUG -D_REENTRANT -DNSS_ENABLE_ECC -DUSE_UTIL_DIRECTLY -DSTDC_HEADERS -DHAVE_STRERROR -DHAVE_SNPRINTF -DHAVE_SYS_CDEFS_H -DMEMMOVE -D__DBINTERFACE_PRIVATE -I/builds/moz2_slave/mozilla-central-linux-nightly/build/obj-firefox/dist/include/nspr -I/builds/moz2_slave/mozilla-central-linux-nightly/build/obj-firefox/dist/include -I/builds/moz2_slave/mozilla-central-linux-nightly/build/obj-firefox/dist/public/dbm -I/builds/moz2_slave/mozilla-central-linux-nightly/build/obj-firefox/dist/private/dbm -I/builds/moz2_slave/mozilla-central-linux-nightly/build/obj-firefox/dist/include -I../../../dbm/include ../../../dbm/src/db.c
Oh, I think I know why this is: http://mxr.mozilla.org/mozilla-central/source/security/manager/Makefile.in#211 That "export MOZ_DEBUG_SYMBOLS" is inside a WINNT block. It just needs to be moved outside of that.
Created attachment 398017 [details] [diff] [review] Export MOZ_DEBUG_SYMBOLS on all platforms
Attachment #398017 - Flags: review?(ted.mielczarek)
Comment on attachment 398017 [details] [diff] [review] Export MOZ_DEBUG_SYMBOLS on all platforms +# Expose MOZ_DEBUG_SYMBOLS to get NSS symbols in crash stacks from +# released builds I'd leave out "crash stacks" here, just say that setting that makes NSS produce debug symbols.
Attachment #398017 - Flags: review?(ted.mielczarek) → review+
Comment on attachment 398017 [details] [diff] [review] Export MOZ_DEBUG_SYMBOLS on all platforms Tweaked the comment on checkin: http://hg.mozilla.org/mozilla-central/rev/75af94f85a98 Lets make sure this works on m-c and then figure out the other branches.
Attachment #398017 - Flags: approval1.9.2+
Comment on attachment 398017 [details] [diff] [review] Export MOZ_DEBUG_SYMBOLS on all platforms presumptive a=me if today's builds look good. Did that NSS fix land on 1.9.1? If so we should land this there too.
Looking good. Mac m-c nightly: http://symbols.mozilla.org/firefox/firefox-3.7a1pre-Darwin-20090902030655-symbols.txt http://symbols.mozilla.org/firefox/libnss3.dylib/9CD50B5E98BB37A459A0DC000D2EDB0A0/libnss3.dylib.sym Linux m-c nightly: http://symbols.mozilla.org/firefox/firefox-3.7a1pre-Linux-20090902030656-symbols.txt http://symbols.mozilla.org/firefox/libnss3.so/29E7C29C6F749658F5645807A67CD0760/libnss3.so.sym (In reply to comment #21) > Did that NSS fix land on 1.9.1? If so we should land this there too. 1.9.1 still has NSS 3.12.3, one point release too few to pick up bug 498509. There are plans to upgrade it though, see bug 504080.
Comment on attachment 398017 [details] [diff] [review] Export MOZ_DEBUG_SYMBOLS on all platforms This should be harmless to land on 1.9.1 in advance of the NSS upgrade, ensuring that we get good NSS symbols when that happens. I'd just grant you a+, but apparently I don't have that power, so I'll play by the rules and request it.
Attachment #398017 - Flags: approval184.108.40.206?
This caused build bustage on WinCE 5.0 mozilla-central: cd util; make -j1 libs make: Entering directory `/e/builds/moz2_slave/mozilla-central-wince/build/security/nss/lib/util' nsinstall -m 664 /e/builds/moz2_slave/mozilla-central-wince/build/obj-firefox/nss/nssutil/nssutil.lib e:/builds/moz2_slave/mozilla-central-wince/build/obj-firefox/dist/lib nsinstall -m 775 /e/builds/moz2_slave/mozilla-central-wince/build/obj-firefox/nss/nssutil/nssutil3.dll e:/builds/moz2_slave/mozilla-central-wince/build/obj-firefox/dist/lib nsinstall -m 644 /e/builds/moz2_slave/mozilla-central-wince/build/obj-firefox/nss/nssutil/nssutil3.pdb e:/builds/moz2_slave/mozilla-central-wince/build/obj-firefox/dist/lib make: Leaving directory `/e/builds/moz2_slave/mozilla-central-wince/build/security/nss/lib/util' make: Leaving directory `/e/builds/moz2_slave/mozilla-central-wince/build/security/nss/lib' nsinstall: e:\builds\moz2_slave\mozilla-central-wince\build\obj-firefox\nss\nssutil\nssutil3.pdb: No such file or directory make: *** [libs] Error 3 make: *** [libs] Error 2 http://production-master.build.mozilla.org:8010/builders/WINCE%205.0%20mozilla-central%20build/builds/251/steps/compile/logs/stdio
Also, I just noticed that the mozconfig patch in this bug is different than the one that was actually landed (in that the one landed adds MOZ_DEBUG_SYMBOLS to the wince configs) - is that intentional?
I think you went a little too nuts with MOZ_DEBUG_SYMBOLS in the previous patch. Apparently that doesn't work right on the WinCE builds (which probably deserves its own bug), so we should drop it from their mozconfigs. Then it should be safe to reland this.
Did they do a full clobber build on WinCE? If not, then probably no new libs were built, and hence no new pdbs were built.
(In reply to comment #27) > Did they do a full clobber build on WinCE? If not, then probably no new > libs were built, and hence no new pdbs were built. The nightly build failed, too - which was a complete rebuild.
It's possible there's some weird wince-specific bug there, the wince builds are still using their intermediate tools that pretend to be gcc but call visual C++ internally. We should just stop setting MOZ_DEBUG_SYMBOLS in the wince mozconfigs, and spin that error off into a separate bug.
Attachment #398017 - Flags: approval220.127.116.11? → approval18.104.22.168+
Comment on attachment 398017 [details] [diff] [review] Export MOZ_DEBUG_SYMBOLS on all platforms Approved for 22.214.171.124, a=dveditz for release-drivers Please do file the wince bug mentioned in comment 29
Bug 514295 for the WinCE issue. Not setting that as a dep here.
Created attachment 398247 [details] [diff] [review] Comment out MOZ_DEBUG_SYMBOLS for WinCE
Attachment #398247 - Flags: review?(aki)
Comment on attachment 398247 [details] [diff] [review] Comment out MOZ_DEBUG_SYMBOLS for WinCE http://hg.mozilla.org/build/buildbot-configs/rev/a15836392b6e
Comment on attachment 398017 [details] [diff] [review] Export MOZ_DEBUG_SYMBOLS on all platforms Relanded on m-c - http://hg.mozilla.org/mozilla-central/rev/16c141deddba
WinCE is broken in the same way as before, I think because config/autoconf.mk still has MOZ_DEBUG_SYMBOLS=1. I've set a clobber for all slaves.
Comment on attachment 398017 [details] [diff] [review] Export MOZ_DEBUG_SYMBOLS on all platforms Clobber did fix m-c WinCE. http://hg.mozilla.org/releases/mozilla-1.9.2/rev/fffff571cfb0 http://hg.mozilla.org/releases/mozilla-1.9.1/rev/f9e9ccea893d
status1.9.1: --- → .4-fixed
status1.9.2: --- → beta1-fixed
Status: NEW → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → FIXED
Oops, forgot the try server mozconfigs.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Created attachment 400671 [details] [diff] [review] Try server mozconfigs
Attachment #400671 - Flags: review?(bhearsum)
Attachment #400671 - Flags: review?(bhearsum) → review+
Attachment #400671 - Flags: checked-in+
Comment on attachment 400671 [details] [diff] [review] Try server mozconfigs http://hg.mozilla.org/build/buildbot-configs/rev/a32e6149819a and try master updated.
Coop is sorting out XULRunner in bug 514700. Code coverage needs bug 512341 instead.
Status: REOPENED → RESOLVED
Last Resolved: 9 years ago → 9 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
I've seen the following option for output debug symbol ac_add_options --enable-debug-symbols The question is: Why export MOZ_DEBUG_SYMBOLS=1 didn't wipe out?
Sorry, I don't understand the question. The build system probably changed quite a bit in the last 5 years, we're not exporting MOZ_DEBUG_SYMBOLS in mozconfigs any more.
Yeap, that's why I am confusing, because there is still have MOZ_DEBUG_SYMBOLS in some configure files in mozilla source tree, I hope it would be wipe out completely.
This conversation is not relevant to this old, fixed bug.
I need to file a new bug for removing all the things related to MOZ_DEBUG_SYMBOLS?
You need to log in before you can comment on or make changes to this bug.