Closed Bug 1252072 Opened 6 years ago Closed 6 years ago

XPCOM NS_InvokeByIndex is incompatible with AddressSanitizer LLVM 3.8+


(Core :: XPCOM, defect)

Not set



Tracking Status
firefox47 --- fixed


(Reporter: decoder, Assigned: decoder)


(Blocks 1 open bug)


(Keywords: sec-want, Whiteboard: [adv-main47-])


(1 file)

XPCOM's NS_InvokeByIndex method on x86_64 makes assumptions about the stack, in particular about the behavior and alignment of __builtin_alloca that do not hold with AddressSanitizer's dynamic stack instrumentation feature turned on. This feature has been turned on by default in LLVM 3.8+. With that LLVM version, Firefox crashes on startup with stack-overflow/underflow errors.

Perma-link to the problematic function:

I've discussed this with the ASan developers:

> Yes. This code looks like a portability nightmare, and IMHO should be 
> deleted or rewritten in assembly as this comment suggests [...]
> You can't depend on the behavior or alignment of alloca() function as this
> code does, and ASan certainly changes the stack frame layout significantly.
> It used to ignore alloca() calls, so you were able to get away w/o disabling
> this function under ASan, but it's not expected to work appropriately.
> Blacklisting it (or, better, adding __attribute__((no_sanitize("address")))
> seems like a right thing to do.

I suggest adding MOZ_ASAN_BLACKLIST to this function for this and all other platforms that use __builtin_alloca for stack space.

Patch coming.
Comment on attachment 8724898 [details]
MozReview Request: Bug 1252072 - Prevent ASan instrumentation for unsafe xpcom functions. r=froydnj

r=me with the change below.

::: xpcom/reflect/xptcall/md/unix/xptcinvoke_linux_s390.cpp:173
(Diff revision 1)

All those ASan runs we do on s390... ;)

::: xpcom/reflect/xptcall/md/unix/xptcinvoke_x86_64_unix.cpp:116
(Diff revision 1)

If you could at least comment this particular annotation and explain why we're blacklisting this function, that would be most helpful.  I don't think the other ones need annotations at this time, but if you're feeling motivated, you could comment them too.
Attachment #8724898 - Flags: review?(nfroyd) → review+
Closed: 6 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla47
Blocks: 1267650
Depends on: 1272498
Whiteboard: [adv-main47-]
You need to log in before you can comment on or make changes to this bug.