895845 - Deploy new clang when available to fix ASan: Intermittent crashes [@ NS_IsMainThread] with heap-buffer-overflow

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Christian Holler (:decoder)]

Several people are seeing intermittent crashes like this with the ASan builds:

==21415==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60200001cdb0 at pc 0x7f16c7c39a3d bp 0x7fff8884cc60 sp 0x7fff8884cc58
READ of size 4 at 0x60200001cdb0 thread T0
    #0 0x7f16c7c39a3c in _Z15NS_IsMainThreadv dist/include/nsThreadUtils.h:104
...
[callers vary from run to run]
...
0x60200001cdb0 is located 0 bytes inside of 4-byte region [0x60200001cdb0,0x60200001cdb4)
allocated by thread T0 here:
    #0 0x441f5b in __interceptor_memalign _asan_rtl_
    #1 0x7f16d34a1364 in allocate_and_init /build/buildd/eglibc-2.15/elf/dl-tls.c:526

=>0x0c047fffb9b0: fa fa 00 04 fa fa[04]fa fa fa 00 fa fa fa 00 fa


This seems to be an ASan bug, and it's currently tracked at 

http://code.google.com/p/address-sanitizer/issues/detail?id=204

Once the issue has been diagnosed and fixed, we will likely have to upgrade the Clang version for ASan builds. Filing this now so we can track this, and setting needinfo on myself to provide the necessary fix revision to upgrade.

Comment 1

[Christian Holler (:decoder)]

Attachment: asan-workaround.patch

This patch is a temporary workaround until the actual fix (which needs to happen in LLVM) is done. Jesse, can you test if this patch fixes the problem for you?

Comment 2

[Andrew McCreight [:mccr8]]

Comment on attachment 780112 [details] [diff] [review]
asan-workaround.patch

Review of attachment 780112 [details] [diff] [review]:
-----------------------------------------------------------------

This isn't really a cycle collector thing, but it only affects ASAN builds, so it looks reasonable to me.

Comment 3

[Jesse Ruderman]

Comment on attachment 780112 [details] [diff] [review]
asan-workaround.patch

With this patch, Firefox starts up and shuts down without crashing.  Thanks!

Comment 4

[Christian Holler (:decoder)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/48d3d69fd9b1

Leave open because this is a temporary fix and we're still waiting for the actual upstream fix.

Comment 5

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/48d3d69fd9b1

Comment 6

[Andrew McCreight [:mccr8]]

FYI, NS_IsCycleCollectorThread is going away in bug 901630, but the other one will still be there.

Comment 7

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Found in triage. As this is more a bug about code within an ASan build, not a bug about RelEng automation, I'm moving this to "Other".


(In reply to Christian Holler (:decoder) from comment #4)
> https://hg.mozilla.org/integration/mozilla-inbound/rev/48d3d69fd9b1
> 
> Leave open because this is a temporary fix and we're still waiting for the
> actual upstream fix.
(In reply to Andrew McCreight [:mccr8] from comment #6)
> FYI, NS_IsCycleCollectorThread is going away in bug 901630, but the other
> one will still be there.

Whats status here? What bug is tracking work on "the other one"?

Comment 8

[Andrew McCreight [:mccr8]]

The other assertion in the file, NS_IsMainThread().  There's no work to be done on NS_IsMainThread().  Decoder landed a patch in Gecko to work around the crash.

This bug is a little weird, because what has landed so far is a Gecko patch, but eventually it will be fixed by deploying a new version of Clang, which I guess is why this is in Release Engineering.

Comment 9

[Andrew McCreight [:mccr8]]

So, basically, this bug is waiting on an upstream fix in Clang itself.

Comment 10

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to Andrew McCreight [:mccr8] from comment #9)
> So, basically, this bug is waiting on an upstream fix in Clang itself.

Good to know, thanks. As bug is about upgrading clang when new fixed version is available, I'm moving to RelEng:PlatformSupport, and tweaking summary to match.

mccr8: any ETA on the new version of clang with the fix?

Comment 11

[Andrew McCreight [:mccr8]]

Decoder would know better than me, but just looking at the link above it seems like they haven't fixed the bug in LLVM yet.

Comment 12

[Christian Holler (:decoder)]

I just pinged the ASan developers again in the Google bug. We aim at upgrading Clang for ASan this quarter possibly and it would be nice to include a fix for this issue, if available.

Comment 13

[Christian Holler (:decoder)]

I think this should be gone with the Clang upgrade I made in bug 1415689. If this pops up for anyone, please file a new bug, thanks!