Deploy new clang when available to fix ASan: Intermittent crashes [@ NS_IsMainThread] with heap-buffer-overflow

RESOLVED FIXED

Status

task
RESOLVED FIXED
6 years ago
a year ago

People

(Reporter: decoder, Assigned: decoder)

Tracking

(Blocks 1 bug, {sec-want})

Details

(Whiteboard: [asan][security-assurance-q3][leave open], crash signature)

Attachments

(1 attachment)

Assignee

Description

6 years ago
Several people are seeing intermittent crashes like this with the ASan builds:

==21415==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60200001cdb0 at pc 0x7f16c7c39a3d bp 0x7fff8884cc60 sp 0x7fff8884cc58
READ of size 4 at 0x60200001cdb0 thread T0
    #0 0x7f16c7c39a3c in _Z15NS_IsMainThreadv dist/include/nsThreadUtils.h:104
...
[callers vary from run to run]
...
0x60200001cdb0 is located 0 bytes inside of 4-byte region [0x60200001cdb0,0x60200001cdb4)
allocated by thread T0 here:
    #0 0x441f5b in __interceptor_memalign _asan_rtl_
    #1 0x7f16d34a1364 in allocate_and_init /build/buildd/eglibc-2.15/elf/dl-tls.c:526

=>0x0c047fffb9b0: fa fa 00 04 fa fa[04]fa fa fa 00 fa fa fa 00 fa


This seems to be an ASan bug, and it's currently tracked at 

http://code.google.com/p/address-sanitizer/issues/detail?id=204

Once the issue has been diagnosed and fixed, we will likely have to upgrade the Clang version for ASan builds. Filing this now so we can track this, and setting needinfo on myself to provide the necessary fix revision to upgrade.
Assignee

Updated

6 years ago
Flags: needinfo?(choller)

Updated

6 years ago
Crash Signature: [@ NS_IsMainThread() ]
Assignee

Comment 1

6 years ago
This patch is a temporary workaround until the actual fix (which needs to happen in LLVM) is done. Jesse, can you test if this patch fixes the problem for you?
Assignee: nobody → choller
Status: NEW → ASSIGNED
Attachment #780112 - Flags: feedback?(jruderman)
Flags: needinfo?(choller)
Assignee

Updated

6 years ago
Whiteboard: [asan] → [asan][security-assurance-q3]
Assignee

Updated

6 years ago
Attachment #780112 - Flags: review?(continuation)
Comment on attachment 780112 [details] [diff] [review]
asan-workaround.patch

Review of attachment 780112 [details] [diff] [review]:
-----------------------------------------------------------------

This isn't really a cycle collector thing, but it only affects ASAN builds, so it looks reasonable to me.
Attachment #780112 - Flags: review?(continuation) → review+

Comment 3

6 years ago
Comment on attachment 780112 [details] [diff] [review]
asan-workaround.patch

With this patch, Firefox starts up and shuts down without crashing.  Thanks!
Attachment #780112 - Flags: feedback?(jruderman) → feedback+
Assignee

Comment 4

6 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/48d3d69fd9b1

Leave open because this is a temporary fix and we're still waiting for the actual upstream fix.
Whiteboard: [asan][security-assurance-q3] → [asan][security-assurance-q3][leave open]
FYI, NS_IsCycleCollectorThread is going away in bug 901630, but the other one will still be there.
Product: mozilla.org → Release Engineering
Found in triage. As this is more a bug about code within an ASan build, not a bug about RelEng automation, I'm moving this to "Other".


(In reply to Christian Holler (:decoder) from comment #4)
> https://hg.mozilla.org/integration/mozilla-inbound/rev/48d3d69fd9b1
> 
> Leave open because this is a temporary fix and we're still waiting for the
> actual upstream fix.
(In reply to Andrew McCreight [:mccr8] from comment #6)
> FYI, NS_IsCycleCollectorThread is going away in bug 901630, but the other
> one will still be there.

Whats status here? What bug is tracking work on "the other one"?
Component: General Automation → Other
QA Contact: catlee → joduinn
The other assertion in the file, NS_IsMainThread().  There's no work to be done on NS_IsMainThread().  Decoder landed a patch in Gecko to work around the crash.

This bug is a little weird, because what has landed so far is a Gecko patch, but eventually it will be fixed by deploying a new version of Clang, which I guess is why this is in Release Engineering.
So, basically, this bug is waiting on an upstream fix in Clang itself.
Depends on: 923486
(In reply to Andrew McCreight [:mccr8] from comment #9)
> So, basically, this bug is waiting on an upstream fix in Clang itself.

Good to know, thanks. As bug is about upgrading clang when new fixed version is available, I'm moving to RelEng:PlatformSupport, and tweaking summary to match.

mccr8: any ETA on the new version of clang with the fix?
Component: Other → Platform Support
Flags: needinfo?(continuation)
QA Contact: joduinn → coop
Summary: ASan: Intermittent crashes [@ NS_IsMainThread] with heap-buffer-overflow → Deploy new clang when available to fix ASan: Intermittent crashes [@ NS_IsMainThread] with heap-buffer-overflow
Decoder would know better than me, but just looking at the link above it seems like they haven't fixed the bug in LLVM yet.
Assignee

Comment 12

5 years ago
I just pinged the ASan developers again in the Google bug. We aim at upgrading Clang for ASan this quarter possibly and it would be nice to include a fix for this issue, if available.
Assignee

Updated

5 years ago
Depends on: 957865

Updated

4 years ago
Crash Signature: [@ NS_IsMainThread() ] → [@ NS_IsMainThread() ] [@ NS_IsMainThread ]
Assignee

Comment 13

a year ago
I think this should be gone with the Clang upgrade I made in bug 1415689. If this pops up for anyone, please file a new bug, thanks!
Assignee

Updated

a year ago
Status: ASSIGNED → RESOLVED
Last Resolved: a year ago
Resolution: --- → FIXED
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.