XPCOM NS_InvokeByIndex is incompatible with AddressSanitizer LLVM 3.8+

RESOLVED FIXED in Firefox 47

Status

()

--
major
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: decoder, Assigned: decoder)

Tracking

(Blocks: 1 bug, {sec-want})

Trunk
mozilla47
All
Linux
sec-want
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox47 fixed)

Details

(Whiteboard: [adv-main47-])

Attachments

(1 attachment)

(Assignee)

Description

3 years ago
XPCOM's NS_InvokeByIndex method on x86_64 makes assumptions about the stack, in particular about the behavior and alignment of __builtin_alloca that do not hold with AddressSanitizer's dynamic stack instrumentation feature turned on. This feature has been turned on by default in LLVM 3.8+. With that LLVM version, Firefox crashes on startup with stack-overflow/underflow errors.

Perma-link to the problematic function: https://dxr.mozilla.org/mozilla-central/rev/5e0140b6d11821e0c2a2de25bc5431783f03380a/xpcom/reflect/xptcall/md/unix/xptcinvoke_x86_64_unix.cpp#117

I've discussed this with the ASan developers:

> Yes. This code looks like a portability nightmare, and IMHO should be 
> deleted or rewritten in assembly as this comment suggests [...]
> You can't depend on the behavior or alignment of alloca() function as this
> code does, and ASan certainly changes the stack frame layout significantly.
> It used to ignore alloca() calls, so you were able to get away w/o disabling
> this function under ASan, but it's not expected to work appropriately.
> Blacklisting it (or, better, adding __attribute__((no_sanitize("address")))
> seems like a right thing to do.

I suggest adding MOZ_ASAN_BLACKLIST to this function for this and all other platforms that use __builtin_alloca for stack space.

Patch coming.
(Assignee)

Comment 1

3 years ago
Created attachment 8724898 [details]
MozReview Request: Bug 1252072 - Prevent ASan instrumentation for unsafe xpcom functions. r=froydnj

Review commit: https://reviewboard.mozilla.org/r/37217/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/37217/
Attachment #8724898 - Flags: review?(nfroyd)
Comment on attachment 8724898 [details]
MozReview Request: Bug 1252072 - Prevent ASan instrumentation for unsafe xpcom functions. r=froydnj

https://reviewboard.mozilla.org/r/37217/#review33763

r=me with the change below.

::: xpcom/reflect/xptcall/md/unix/xptcinvoke_linux_s390.cpp:173
(Diff revision 1)
> +MOZ_ASAN_BLACKLIST

All those ASan runs we do on s390... ;)

::: xpcom/reflect/xptcall/md/unix/xptcinvoke_x86_64_unix.cpp:116
(Diff revision 1)
> +MOZ_ASAN_BLACKLIST

If you could at least comment this particular annotation and explain why we're blacklisting this function, that would be most helpful.  I don't think the other ones need annotations at this time, but if you're feeling motivated, you could comment them too.
Attachment #8724898 - Flags: review?(nfroyd) → review+

Comment 4

3 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/1af72489ebdc
Status: NEW → RESOLVED
Last Resolved: 3 years ago
status-firefox47: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla47
(Assignee)

Updated

3 years ago
Blocks: 1267650

Updated

3 years ago
Depends on: 1272498
Whiteboard: [adv-main47-]
You need to log in before you can comment on or make changes to this bug.