Closed Bug 1252072 Opened 4 years ago Closed 4 years ago

XPCOM NS_InvokeByIndex is incompatible with AddressSanitizer LLVM 3.8+

Categories

(Core :: XPCOM, defect, major)

All
Linux
defect
Not set
major

Tracking

()

RESOLVED FIXED
mozilla47
Tracking Status
firefox47 --- fixed

People

(Reporter: decoder, Assigned: decoder)

References

(Blocks 1 open bug)

Details

(Keywords: sec-want, Whiteboard: [adv-main47-])

Attachments

(1 file)

XPCOM's NS_InvokeByIndex method on x86_64 makes assumptions about the stack, in particular about the behavior and alignment of __builtin_alloca that do not hold with AddressSanitizer's dynamic stack instrumentation feature turned on. This feature has been turned on by default in LLVM 3.8+. With that LLVM version, Firefox crashes on startup with stack-overflow/underflow errors.

Perma-link to the problematic function: https://dxr.mozilla.org/mozilla-central/rev/5e0140b6d11821e0c2a2de25bc5431783f03380a/xpcom/reflect/xptcall/md/unix/xptcinvoke_x86_64_unix.cpp#117

I've discussed this with the ASan developers:

> Yes. This code looks like a portability nightmare, and IMHO should be 
> deleted or rewritten in assembly as this comment suggests [...]
> You can't depend on the behavior or alignment of alloca() function as this
> code does, and ASan certainly changes the stack frame layout significantly.
> It used to ignore alloca() calls, so you were able to get away w/o disabling
> this function under ASan, but it's not expected to work appropriately.
> Blacklisting it (or, better, adding __attribute__((no_sanitize("address")))
> seems like a right thing to do.

I suggest adding MOZ_ASAN_BLACKLIST to this function for this and all other platforms that use __builtin_alloca for stack space.

Patch coming.
Comment on attachment 8724898 [details]
MozReview Request: Bug 1252072 - Prevent ASan instrumentation for unsafe xpcom functions. r=froydnj

https://reviewboard.mozilla.org/r/37217/#review33763

r=me with the change below.

::: xpcom/reflect/xptcall/md/unix/xptcinvoke_linux_s390.cpp:173
(Diff revision 1)
> +MOZ_ASAN_BLACKLIST

All those ASan runs we do on s390... ;)

::: xpcom/reflect/xptcall/md/unix/xptcinvoke_x86_64_unix.cpp:116
(Diff revision 1)
> +MOZ_ASAN_BLACKLIST

If you could at least comment this particular annotation and explain why we're blacklisting this function, that would be most helpful.  I don't think the other ones need annotations at this time, but if you're feeling motivated, you could comment them too.
Attachment #8724898 - Flags: review?(nfroyd) → review+
https://hg.mozilla.org/mozilla-central/rev/1af72489ebdc
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla47
Blocks: 1267650
Depends on: 1272498
Whiteboard: [adv-main47-]
You need to log in before you can comment on or make changes to this bug.