1020590 - ASAN silently failing to find symbolizer

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Andrew McCreight [:mccr8]]

Ryan said that ASAN hasn't been symbolizing its builds for around a week.  This is affecting both Mochitest and at least reftests.

The code that finds the symbolizer (at least the one in automationutils, there's other ones in other files) is here:

507       # Symbolizer support
508       llvmsym = os.path.join(xrePath, "llvm-symbolizer")
509       if os.path.isfile(llvmsym):
510         env["ASAN_SYMBOLIZER_PATH"] = llvmsym
511         log.info("ASan using symbolizer at %s", llvmsym)

I looked at a random ASAN mochitest log and it contains "ASan running in low-memory configuration", which indicates we didn't throw, but it doesn't contain "ASan using symbolizer".

So, presumably something changed recently so that xrePath is bogus now, which should be fixed.

Additionally, failing to set the symbolizer should turn the tree orange, as ASAN without a symbolize is not very useful, so any change that breaks that code should be backed out.

Comment 1

[Andrew McCreight [:mccr8]]

Locally, ASAN is still able to find the symbolizer path, so maybe something with the Clang setup changed recently?

Comment 2

[Andrew McCreight [:mccr8]]

I pushed a version to try that prints out the path it is attempting to use for the symbolizer:
  17:09:41     INFO -  Failed to find ASan symbolizer at /builds/slave/test/build/application/firefox/llvm-symbolizer

https://tbpl.mozilla.org/?tree=Try&rev=227105c3b029

Another good thing about enabling LSAN is that it will turn every ASAN test orange if the symbolizer fails, because the suppression list will stop working. ;)

Comment 3

[Christian Holler (:decoder)]

Isolated the regressor, it's bug 1013341. Since that revision, llvm-symbolizer isn't packaged anymore.

Comment 4

[Christian Holler (:decoder)]

I suggest we first either back out the offending commit or the original committers (gk or glandium) provide a fix for issue here such that it still works with GCC. The latter would of course be very much preferable.

As a second step, we should think about how we can make this turn the tree orange in the future. The problem is that when running locally, the symbolizer isn't really a prerequisite, so we would just want this to happen when the llvm-symbolizer binary isn't found in the package that is shipped to the testers.

Comment 5

[Georg Koppen]

(In reply to Christian Holler (:decoder) from comment #4)
> I suggest we first either back out the offending commit or the original
> committers (gk or glandium) provide a fix for issue here such that it still
> works with GCC. The latter would of course be very much preferable.

I won't have time to look at this until next Monday. But I wonder how this is possible at all. Don't the builds get packaged on the same machine building them? And if so, then building with Clang should get the symbolizer included. What am I missing here?

Comment 6

[Mike Hommey [:glandium]]

Attachment: Fix ASAN builds after bug 1013341

Comment 7

[Michael Shal [:mshal]]

Comment on attachment 8435291 [details] [diff] [review]
Fix ASAN builds after bug 1013341

The Makefile is a little hard to read now since MOZ_ICU_DBG_SUFFIX isn't inside any ifdefs, and is in the same group as all of the other ifdef/DEFINES/endif blocks, but meh.

Comment 8

[Mike Hommey [:glandium]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/a3b45c160d46

Comment 9

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/a3b45c160d46