1470535 - MacOS AddressSanitizer tracebacks don't include source:line information.

Status: NEW

(Firefox Build System :: Toolchains, defect)

Description

[Jesse Schwartzentruber (:truber)]

Source:line debug information is missing in AddressSanitizer tracebacks from TC builds (eg. gecko.v2.mozilla-central.latest.firefox.macosx64-fuzzing-asan-opt); we only get symbol information in the backtrace.

An example is bug 1469328:

    #23 0x113b7952f in void mozilla::dom::Function::Call<nsCOMPtr<nsISupports> >(nsCOMPtr<nsISupports> const&, nsTArray<JS::Value> const&, JS::MutableHandle<JS::Value>, mozilla::ErrorResult&, char const*, mozilla::dom::CallbackObject::ExceptionHandling, JS::Realm*) (/Users/truber/builds/m-c-20180610213656-fuzzing-asan-opt/Nightly.app/Contents/MacOS/XUL:x86_64+0x3f7252f)
    #24 0x113b77c7f in nsGlobalWindowInner::RunTimeoutHandler(mozilla::dom::Timeout*, nsIScriptContext*) (/Users/truber/builds/m-c-20180610213656-fuzzing-asan-opt/Nightly.app/Contents/MacOS/XUL:x86_64+0x3f70c7f)
    #25 0x113e1f2f4 in mozilla::dom::TimeoutManager::RunTimeout(mozilla::TimeStamp const&, mozilla::TimeStamp const&) (/Users/truber/builds/m-c-20180610213656-fuzzing-asan-opt/Nightly.app/Contents/MacOS/XUL:x86_64+0x42182f4)
    #26 0x113e1e305 in mozilla::dom::TimeoutExecutor::MaybeExecute() (/Users/truber/builds/m-c-20180610213656-fuzzing-asan-opt/Nightly.app/Contents/MacOS/XUL:x86_64+0x4217305)
    #27 0x113e210b2 in non-virtual thunk to mozilla::dom::TimeoutExecutor::Notify(nsITimer*) (/Users/truber/builds/m-c-20180610213656-fuzzing-asan-opt/Nightly.app/Contents/MacOS/XUL:x86_64+0x421a0b2)
    #28 0x10ff0dad0 in nsTimerImpl::Fire(int) (/Users/truber/builds/m-c-20180610213656-fuzzing-asan-opt/Nightly.app/Contents/MacOS/XUL:x86_64+0x306ad0)
    #29 0x10fecbdaf in nsTimerEvent::Run() (/Users/truber/builds/m-c-20180610213656-fuzzing-asan-opt/Nightly.app/Contents/MacOS/XUL:x86_64+0x2c4daf)

Comment 1

[Nathan Froyd [:froydnj]]

Presumably this bit:

http://dxr.mozilla.org/mozilla-central/source/browser/config/mozconfigs/macosx64/nightly-asan#4

needs to include -gline-tables-only, like other linux-based asan configs?  Not sure if our Mac builds package up the dwarf debug information with them, though...

Comment 2

[(not currently active) Ted Mielczarek]

I don't really know anything about how llvm-symbolizer works, but presumably the issue here is that the source line info doesn't wind up in the binaries and we'd have to produce and package .dSYM bundles alongside the binaries for this to work properly.

Comment 3

[Andrew McCreight [:mccr8]]

Some documentation I found ( https://clang.llvm.org/docs/AddressSanitizer.html ) says: "Note that on OS X you may need to run dsymutil on your binary to have the file:line info in the AddressSanitizer reports.".

Interestingly, you can also disable the symbolizer with ASAN_OPTIONS=symbolize=0. It sounds like this also makes it work better with sandboxing (though I think there's some other reason sandboxing is disabled for ASan builds). If the format is compatible with fix_macosx_stack.py maybe just using that would be better.

Comment 4

[Eric Rahm [:erahm]]

(In reply to Andrew McCreight [:mccr8] from comment #3)

> Interestingly, you can also disable the symbolizer with
> ASAN_OPTIONS=symbolize=0. It sounds like this also makes it work better with
> sandboxing (though I think there's some other reason sandboxing is disabled
> for ASan builds). If the format is compatible with fix_macosx_stack.py maybe
> just using that would be better.

I'm pretty sure it is. They have |scripts/asan_symbolize.py| [1] which is a more elaborate form of our stack fixer. Either we lifted code from them or they lifted code from us. I'd probably just go with theirs.

[1] https://github.com/google/sanitizers/wiki/AddressSanitizerCallStack

Comment 5

[Christian Holler (:decoder)]

Attachment: WIP test patch

The stack fixup approach helps us in automation, but for the ASan Nightly Project, we would really need to ship dSYM files like we ship PDB files on Windows. Deploying ASan Nightly for Mac without line numbers is not going to be useful because often the exact location is crucial to find and fix the problem.


I tried to hack around in our tree to get it to generate dSYM files and package them (see patch attached, WIP, messy). Generating the dSYM files seems to work (tested locally, hence some MOZ_AUTOMATION commented out), but installing/packaging is not. At first I tried to use the same rules like for PDB (e.g. adding to INSTALL_TARGETS, but that didn't work, got an error that I couldn't resolve).


I think this might need the help of a build peer to get into the right direction, as I don't fully understand all of the logic in these Makefiles involved to generate PDB files and how dSYM might be different.

Comment 6

[Mike Hommey [:glandium]]

dSYM files are massive, you don't want to ship those.

Comment 7

[Mike Hommey [:glandium]]

(pdbs are less massive, but I'd say shipping them is suboptimal as well)

Comment 8

[Mike Hommey [:glandium]]

that said, they are not generated at all on those builds, so we don't have target.crashreporter-full.zip, and that's because the crash reporter is disabled. (there's an explicit --disable-crashreporter in build/unix/mozconfig.asan). What you probably want is to enable the crash reporter code, but disable the crash reporter at runtime, by changing the default behavior in SetExceptionHandler when e.g. MOZ_ASAN is defined.

Comment 9

[Christian Holler (:decoder)]

(In reply to Mike Hommey [:glandium] from comment #8)
> that said, they are not generated at all on those builds, so we don't have
> target.crashreporter-full.zip, and that's because the crash reporter is
> disabled. (there's an explicit --disable-crashreporter in
> build/unix/mozconfig.asan). What you probably want is to enable the crash
> reporter code, but disable the crash reporter at runtime, by changing the
> default behavior in SetExceptionHandler when e.g. MOZ_ASAN is defined.

PDB files are created with --disable-crashreporter (and shipped), so I thought we could mimic the same thing for dSYM.


As for the size, do you have any numbers PDB vs. dSYM? 

Apparently, we can use -gline-tables-only (like on Linux), to significantly reduce dSYM size as well.

Comment 10

[(not currently active) Ted Mielczarek]

Oh, right, `INSTALL_TARGETS` won't work because .dSYMs are bundles (they're a directory), and the build system only handles files there.

glandium: the patch here forces the build to run `dsymutil` where it would normally run dump_syms, using the same workarounds we currently have in place for Windows ASan builds.

I think it might be simpler to just enable the crashreporter at build time at this point (as I've said before).

Comment 11

[Tyson Smith [:tsmith]]

We have started using MOZ_CHAOSMODE with some fuzzers and this has the effect of shaking out bugs that are not easily reproducible. Having this fixed will at least give us better logs in those situation.

Comment 12

[Nathan Froyd [:froydnj]]

It looks like this bug basically has the entire conversation that we had last week: that we should try to ship the dsym files on Mac, just like we ship the pdb files on Windows, and decoder even has a prototype patch for that.  So I think we need to polish that up and ship that, possibly with the -gline-tables-only bit mentioned in comment 9?

Comment 13

[Sylvestre Ledru [:Sylvestre]]

Gabriele, is that something you could help with?
Merci beaucoup!

Comment 14

[Gabriele Svelto [:gsvelto]]

Maybe? If this can be done using Nick's new stack fixer then it should be rather easy.

Comment 15

[Sylvestre Ledru [:Sylvestre]]

Ok, redirecting to nick.

It would be great to fix that for the fuzzing team.

Comment 16

[Gabriele Svelto [:gsvelto]]

Oh sorry, I didn't mean that. What I meant is that I might be able to use Nick's stack fixer to solve this. Taking this.

Comment 17

[Jesse Schwartzentruber (:truber)]

I tried dusting off decoder's WIP patch, but couldn't even get it to call dsymutil for a local build.

As for how big these are vs PDB, when I run dsymutil manually (with -gline-tables-only), the dSYMs are much smaller than PDB files:

32K     ./plugin-container.app/Contents/MacOS/plugin-container.dSYM
208K    ./updater.app/Contents/MacOS/org.mozilla.updater.dSYM
1020K   ./libfreebl3.dylib.dSYM
5.8M    ./libipcclientcerts.dylib.dSYM
76K     ./liblgpllibs.dylib.dSYM
3.2M    ./libmozavcodec.dylib.dSYM
368K    ./libmozavutil.dylib.dSYM
2.1M    ./libmozglue.dylib.dSYM
4.4M    ./libnss3.dylib.dSYM
372K    ./libnssckbi.dylib.dSYM
8.0M    ./libosclientcerts.dylib.dSYM
456K    ./libsoftokn3.dylib.dSYM
1.4M    ./firefox.dSYM
1.4M    ./firefox-bin.dSYM
208K    ./certutil.dSYM
26M     ./http3server.dSYM
856K    ./llvm-symbolizer.dSYM
168K    ./pingsender.dSYM
92K     ./pk12util.dSYM
112K    ./ssltunnel.dSYM
1.3M    ./xpcshell.dSYM
821M    ./XUL.dSYM
878M    total

Comment 18

[BugBot [:suhaib / :marco/ :calixte]]

This bug prevents fuzzing from making progress; however, it has low severity. It is important for fuzz blocker bugs to be addressed in a timely manner (see here why?).
:glandium, could you increase the severity?

For more information, please visit auto_nag documentation.

Comment 19

[Mike Hommey [:glandium]]

Is this still something we want?

Comment 20

[Jesse Schwartzentruber (:truber)]

Yes, without this we can't use fxci builds for macOS fuzzing.

Comment 21

[Mike Hommey [:glandium]]

(In reply to Jesse Schwartzentruber (:truber) from comment #20)

Yes, without this we can't use fxci builds for macOS fuzzing.

Can't you use the target.crashreporter-symbols-full.tar.zst artifact?

Comment 22

[Jesse Schwartzentruber (:truber)]

Crash reporter is disabled for ASan builds.