Closed Bug 1024620 Opened 5 years ago Closed 5 years ago

No source debugging of NSS and NSPR from Firefox build on !linux

Categories

(Firefox Build System :: General, defect, critical)

x86
macOS
defect
Not set
critical

Tracking

(firefox30 wontfix, firefox31 wontfix, firefox32+ fixed, firefox33+ fixed, firefox-esr24 wontfix)

RESOLVED FIXED
mozilla33
Tracking Status
firefox30 --- wontfix
firefox31 --- wontfix
firefox32 + fixed
firefox33 + fixed
firefox-esr24 --- wontfix

People

(Reporter: jryans, Assigned: glandium)

References

Details

Attachments

(2 files, 3 obsolete files)

Attached file Mac Debug mozconfig
With a debug build of Firefox for Mac, both gdb and lldb are unable to locate NSS sources, so while I can step through Firefox / XUL source, if I enter an NSS function, I lose the source view.

I've attached my debug mozconfig.

With a similar debug build for Linux, I am able to source debug NSS, so it's something Mac specific.

After some searching, I learned[1] that Macs rely on the Stabs object file locations in a binary or library to the read further debug info from them.

For NSS, those all seem to point to temporary directories that no longer exist:

$ nm -pa libnss3.dylib | grep OSO

00000000539886ef - 03 0001   OSO /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-debug/security/build/tmplH_nUR/derdec.o
00000000539886ef - 03 0001   OSO /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-debug/security/build/tmplH_nUR/derenc.o
00000000539886ef - 03 0001   OSO /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-debug/security/build/tmplH_nUR/dersubr.o
00000000539886ef - 03 0001   OSO /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-debug/security/build/tmplH_nUR/dertime.o
00000000539886ef - 03 0001   OSO /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-debug/security/build/tmplH_nUR/errstrs.o
[...many more lines...]

For XUL, they point to files that do still exist:

$ nm -pa XUL | grep OSO
00000000539886c9 - 03 0001   OSO /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-debug/toolkit/library/build/../nsStaticXULComponents.o
0000000053988573 - 03 0001   OSO /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-debug/toolkit/library/build/../../../media/kiss_fft/kiss_fft.o
0000000053988573 - 03 0001   OSO /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-debug/toolkit/library/build/../../../media/kiss_fft/kiss_fftr.o
0000000053988574 - 03 0001   OSO /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-debug/toolkit/library/build/../../../security/certverifier/Unified_cpp_certverifier0.o
00000000539886d2 - 03 0001   OSO /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-debug/toolkit/library/build/../../../security/pkix/Unified_cpp_security_pkix0.o

[1]: http://stackoverflow.com/questions/10044697/where-how-does-apples-gcc-store-dwarf-inside-an-executable#comment25100499_12827463
We do some funny things to build NSS, I'm not entirely surprised here. I haven't checked, but I'm pretty sure we have okay debug symbols for them in our crash reports (at least we used to). If you run dsymutil on libnss3.dylib, is it debuggable then?
No, that doesn't seem to change anything.

It does produce the .dSYM directory, but source debugging still fails.  dwarfdump of the .dSYM shows that it only seems to contain info about sqlite symbols.  (After looking at the full output of nm -pa libnss3.dylib | grep OSO again, it seems the path to the one sqlite file is correct, so I'm guessing that's why those symbols do get picked up.)

Also, it produces many warnings, as seen below, about the missing object files, so surely that is why it doesn't contain any of the NSS symbols:

$ dsymutil libnss3.dylib
warning: (x86_64) /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-debug/security/build/tmplH_nUR/derdec.o unable to open object file
warning: (x86_64) /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-debug/security/build/tmplH_nUR/derenc.o unable to open object file
warning: (x86_64) /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-debug/security/build/tmplH_nUR/dersubr.o unable to open object file
warning: (x86_64) /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-debug/security/build/tmplH_nUR/dertime.o unable to open object file
warning: (x86_64) /Users/jryans/projects/mozilla/gecko-dev-2/obj-firefox-debug/security/build/tmplH_nUR/errstrs.o unable to open object file
[...many more lines...]
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #1)
> We do some funny things to build NSS, I'm not entirely surprised here. I
> haven't checked, but I'm pretty sure we have okay debug symbols for them in
> our crash reports (at least we used to).

Actually, you should double check that, because it might not be the case. And if that's not the case, it would have been that way for as long as we've folded libraries.
Well, shit:

[tmielczarek@symbols1.dmz.phx1 symbols_ffx]$ grep libnss3\.dylib\.sym firefox-33.0a1-Darwin-20140612030349-x86-macosx64-symbols.txt
libnss3.dylib/570B8897922F38A7B4974CB950443A490/libnss3.dylib.sym
libnss3.dylib/E5F4EB6CD046311DB263E3B5C8C5ECDF0/libnss3.dylib.sym
libnss3.dylib/570B8897922F38A7B4974CB950443A490/libnss3.dylib.sym
libnss3.dylib/E5F4EB6CD046311DB263E3B5C8C5ECDF0/libnss3.dylib.sym

[tmielczarek@symbols1.dmz.phx1 symbols_ffx]$ head -n4 libnss3.dylib/570B8897922F38A7B4974CB950443A490/libnss3.dylib.sym
MODULE mac x86 570B8897922F38A7B4974CB950443A490 libnss3.dylib
FILE 0 /Developer/SDKs/MacOSX10.6.sdk/usr/include/secure/_string.h
FILE 1 hg:hg.mozilla.org/mozilla-central:db/sqlite3/src/sqlite3.c:9e8e3e903484
FUNC 12dda0 18d 0 sqlite3_compileoption_used

(no NSS source files listed at all)
Severity: normal → critical
Then it's been that way since bug 648407.
Blocks: 648407
So, since Firefox 22. Amazing it wasn't found earlier.
Presumably, this is enough to fix this. It should be thoroughly tested, though, because it changes how the linkage is done.
Assignee: nobody → mh+mozilla
Status: NEW → ASSIGNED
Might not work, though. ISTR the reason i added that is that without, the linker just throws away nss.
Attached file Log from attachment 8439633 (obsolete) —
The patch in attachment 8439633 [details] [diff] [review] makes the linker quite upset, claiming the NSS symbols are undefined.  See attached log.
Attachment #8439641 - Attachment mime type: text/x-log → text/plain
This fixes half the problem. About half of nspr remains broken, though.
Attachment #8439633 - Attachment is obsolete: true
Attachment #8439843 - Attachment is obsolete: true
Attachment #8439641 - Attachment is obsolete: true
Comment on attachment 8439906 [details] [diff] [review]
Use object files next to archives when extracting them at link time, or in a subdirectory

This change fixes NSS source debugging for me.  Thanks for jumping on this so quickly!
Attachment #8439906 - Flags: feedback+
Comment on attachment 8439906 [details] [diff] [review]
Use object files next to archives when extracting them at link time, or in a subdirectory

Review of attachment 8439906 [details] [diff] [review]:
-----------------------------------------------------------------

::: config/expandlibs_exec.py
@@ +108,5 @@
> +                                # If the file extracted from the library also
> +                                # exists in the directory containing the
> +                                # library, or one of its subdirectories, use
> +                                # that instead.
> +                                maybe_obj = lookup(os.path.join(basedir, root[len(tmp)+1:]), f)

I think I'd prefer os.path.relpath(root, tmp).

::: config/tests/unit-expandlibs.py
@@ +249,5 @@
>      def test_extract(self):
>          '''Test library extraction'''
>          # Divert subprocess.call
>          subprocess_call = subprocess.call
> +        subprocess_check_output = subprocess.check_output

We have mock installed in the virtualenv, can you file a followup on using that here instead?

Example:
http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/unit-symbolstore.py#188

Docs:
http://www.voidspace.org.uk/python/mock/

@@ +324,5 @@
> +                        else:
> +                            dirname = os.path.dirname(f[len(self.tmpdir)+1:])
> +                            if base.endswith('f'):
> +                                dirname = os.path.join(dirname, 'foo', 'bar')
> +                            extracted_args.extend([self.tmpfile(dirname, Obj(base)), self.tmpfile(dirname, Obj(base + '2'))])

I kind of wish this was nicer but you wrote a test so I can't really complain.
Attachment #8439906 - Flags: review?(ted) → review+
Note I'm very likely to kill expand_libs soonish. Not sure it's worth filing bugs about its unit test at this point.
Let's give it a few days and uplift then. Flagging me to not forget.
Flags: needinfo?(mh+mozilla)
Flags: needinfo?(mh+mozilla)
Summary: No source debugging of NSS from Firefox build on Mac → No source debugging of NSS and NSPR from Firefox build on !linux
https://hg.mozilla.org/mozilla-central/rev/756d3d9456c3
https://hg.mozilla.org/mozilla-central/rev/1d0eb8fb8d87
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla33
Flags: needinfo?(mh+mozilla)
Comment on attachment 8439906 [details] [diff] [review]
Use object files next to archives when extracting them at link time, or in a subdirectory

Approval Request Comment
[Feature/regressing bug #]: A long time ago in bug 648407
[User impact if declined]: No NSS/NSPR symbols in crash reports.
[Describe test coverage new/current, TBPL]: Landed on m-c a while ago, and verified to add the relevant symbols to debug symbols used by the crash report processing.
[Risks and why]: Only risk is bug 1024620, which hasn't been clearly attributed to this bug. We'd watch perf numbers on aurora if this is a+ed. I'd request approval for beta as well if there wasn't this yet unidentified regression.
[String/UUID change made/needed]: None
Attachment #8439906 - Flags: approval-mozilla-aurora?
Flags: needinfo?(mh+mozilla)
I don't this it's worth the risk of taking this change on beta or ESR 24. Note that 31 is the next ESR train so this issue will exist in ESR 31. We can consider taking this in ESR 31.1.0 if there is significant benefit.
Comment on attachment 8439906 [details] [diff] [review]
Use object files next to archives when extracting them at link time, or in a subdirectory

Aurora+. Please monitor this fix on Aurora over the next two weeks. I would like to identify any fallout before 32 moves to beta.
Attachment #8439906 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.