ASAN silently failing to find symbolizer

RESOLVED FIXED in mozilla32

Status

defect
RESOLVED FIXED
5 years ago
5 years ago

People

(Reporter: mccr8, Assigned: glandium)

Tracking

({regression, sec-want})

unspecified
mozilla32
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox30 unaffected, firefox31 affected, firefox32 affected)

Details

Attachments

(1 attachment)

Ryan said that ASAN hasn't been symbolizing its builds for around a week.  This is affecting both Mochitest and at least reftests.

The code that finds the symbolizer (at least the one in automationutils, there's other ones in other files) is here:

507       # Symbolizer support
508       llvmsym = os.path.join(xrePath, "llvm-symbolizer")
509       if os.path.isfile(llvmsym):
510         env["ASAN_SYMBOLIZER_PATH"] = llvmsym
511         log.info("ASan using symbolizer at %s", llvmsym)

I looked at a random ASAN mochitest log and it contains "ASan running in low-memory configuration", which indicates we didn't throw, but it doesn't contain "ASan using symbolizer".

So, presumably something changed recently so that xrePath is bogus now, which should be fixed.

Additionally, failing to set the symbolizer should turn the tree orange, as ASAN without a symbolize is not very useful, so any change that breaks that code should be backed out.
Blocks: 1019934, 1020584
Keywords: sec-want
Locally, ASAN is still able to find the symbolizer path, so maybe something with the Clang setup changed recently?
Blocks: 988041
I pushed a version to try that prints out the path it is attempting to use for the symbolizer:
  17:09:41     INFO -  Failed to find ASan symbolizer at /builds/slave/test/build/application/firefox/llvm-symbolizer

https://tbpl.mozilla.org/?tree=Try&rev=227105c3b029

Another good thing about enabling LSAN is that it will turn every ASAN test orange if the symbolizer fails, because the suppression list will stop working. ;)
Isolated the regressor, it's bug 1013341. Since that revision, llvm-symbolizer isn't packaged anymore.
I suggest we first either back out the offending commit or the original committers (gk or glandium) provide a fix for issue here such that it still works with GCC. The latter would of course be very much preferable.

As a second step, we should think about how we can make this turn the tree orange in the future. The problem is that when running locally, the symbolizer isn't really a prerequisite, so we would just want this to happen when the llvm-symbolizer binary isn't found in the package that is shipped to the testers.
(In reply to Christian Holler (:decoder) from comment #4)
> I suggest we first either back out the offending commit or the original
> committers (gk or glandium) provide a fix for issue here such that it still
> works with GCC. The latter would of course be very much preferable.

I won't have time to look at this until next Monday. But I wonder how this is possible at all. Don't the builds get packaged on the same machine building them? And if so, then building with Clang should get the symbolizer included. What am I missing here?
Attachment #8435291 - Flags: review?(mshal)
Assignee: nobody → mh+mozilla
Status: NEW → ASSIGNED
Comment on attachment 8435291 [details] [diff] [review]
Fix ASAN builds after bug 1013341

The Makefile is a little hard to read now since MOZ_ICU_DBG_SUFFIX isn't inside any ifdefs, and is in the same group as all of the other ifdef/DEFINES/endif blocks, but meh.
Attachment #8435291 - Flags: review?(mshal) → review+
https://hg.mozilla.org/mozilla-central/rev/a3b45c160d46
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla32
You need to log in before you can comment on or make changes to this bug.