Ryan said that ASAN hasn't been symbolizing its builds for around a week. This is affecting both Mochitest and at least reftests. The code that finds the symbolizer (at least the one in automationutils, there's other ones in other files) is here: 507 # Symbolizer support 508 llvmsym = os.path.join(xrePath, "llvm-symbolizer") 509 if os.path.isfile(llvmsym): 510 env["ASAN_SYMBOLIZER_PATH"] = llvmsym 511 log.info("ASan using symbolizer at %s", llvmsym) I looked at a random ASAN mochitest log and it contains "ASan running in low-memory configuration", which indicates we didn't throw, but it doesn't contain "ASan using symbolizer". So, presumably something changed recently so that xrePath is bogus now, which should be fixed. Additionally, failing to set the symbolizer should turn the tree orange, as ASAN without a symbolize is not very useful, so any change that breaks that code should be backed out.
Locally, ASAN is still able to find the symbolizer path, so maybe something with the Clang setup changed recently?
I pushed a version to try that prints out the path it is attempting to use for the symbolizer: 17:09:41 INFO - Failed to find ASan symbolizer at /builds/slave/test/build/application/firefox/llvm-symbolizer https://tbpl.mozilla.org/?tree=Try&rev=227105c3b029 Another good thing about enabling LSAN is that it will turn every ASAN test orange if the symbolizer fails, because the suppression list will stop working. ;)
Isolated the regressor, it's bug 1013341. Since that revision, llvm-symbolizer isn't packaged anymore.
I suggest we first either back out the offending commit or the original committers (gk or glandium) provide a fix for issue here such that it still works with GCC. The latter would of course be very much preferable. As a second step, we should think about how we can make this turn the tree orange in the future. The problem is that when running locally, the symbolizer isn't really a prerequisite, so we would just want this to happen when the llvm-symbolizer binary isn't found in the package that is shipped to the testers.
(In reply to Christian Holler (:decoder) from comment #4) > I suggest we first either back out the offending commit or the original > committers (gk or glandium) provide a fix for issue here such that it still > works with GCC. The latter would of course be very much preferable. I won't have time to look at this until next Monday. But I wonder how this is possible at all. Don't the builds get packaged on the same machine building them? And if so, then building with Clang should get the symbolizer included. What am I missing here?
Assignee: nobody → mh+mozilla
Status: NEW → ASSIGNED
Comment on attachment 8435291 [details] [diff] [review] Fix ASAN builds after bug 1013341 The Makefile is a little hard to read now since MOZ_ICU_DBG_SUFFIX isn't inside any ifdefs, and is in the same group as all of the other ifdef/DEFINES/endif blocks, but meh.
Attachment #8435291 - Flags: review?(mshal) → review+
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla32
You need to log in before you can comment on or make changes to this bug.