1252072 - XPCOM NS_InvokeByIndex is incompatible with AddressSanitizer LLVM 3.8+

Status: RESOLVED FIXED

(Core :: XPCOM, defect)

Description

[Christian Holler (:decoder)]

XPCOM's NS_InvokeByIndex method on x86_64 makes assumptions about the stack, in particular about the behavior and alignment of __builtin_alloca that do not hold with AddressSanitizer's dynamic stack instrumentation feature turned on. This feature has been turned on by default in LLVM 3.8+. With that LLVM version, Firefox crashes on startup with stack-overflow/underflow errors.

Perma-link to the problematic function: https://dxr.mozilla.org/mozilla-central/rev/5e0140b6d11821e0c2a2de25bc5431783f03380a/xpcom/reflect/xptcall/md/unix/xptcinvoke_x86_64_unix.cpp#117

I've discussed this with the ASan developers:

> Yes. This code looks like a portability nightmare, and IMHO should be 
> deleted or rewritten in assembly as this comment suggests [...]
> You can't depend on the behavior or alignment of alloca() function as this
> code does, and ASan certainly changes the stack frame layout significantly.
> It used to ignore alloca() calls, so you were able to get away w/o disabling
> this function under ASan, but it's not expected to work appropriately.
> Blacklisting it (or, better, adding __attribute__((no_sanitize("address")))
> seems like a right thing to do.

I suggest adding MOZ_ASAN_BLACKLIST to this function for this and all other platforms that use __builtin_alloca for stack space.

Patch coming.

Comment 1

[Christian Holler (:decoder)]

Attachment: MozReview Request: Bug 1252072 - Prevent ASan instrumentation for unsafe xpcom functions. r=froydnj

Review commit: https://reviewboard.mozilla.org/r/37217/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/37217/

Comment 2

[Nathan Froyd [:froydnj]]

Comment on attachment 8724898 [details]
MozReview Request: Bug 1252072 - Prevent ASan instrumentation for unsafe xpcom functions. r=froydnj

https://reviewboard.mozilla.org/r/37217/#review33763

r=me with the change below.

::: xpcom/reflect/xptcall/md/unix/xptcinvoke_linux_s390.cpp:173
(Diff revision 1)
> +MOZ_ASAN_BLACKLIST

All those ASan runs we do on s390... ;)

::: xpcom/reflect/xptcall/md/unix/xptcinvoke_x86_64_unix.cpp:116
(Diff revision 1)
> +MOZ_ASAN_BLACKLIST

If you could at least comment this particular annotation and explain why we're blacklisting this function, that would be most helpful.  I don't think the other ones need annotations at this time, but if you're feeling motivated, you could comment them too.

Comment 3

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/1af72489ebdc

Comment 4

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/1af72489ebdc