1492845 - Crash [@ js::jit::AutoWritableJitCode::AutoWritableJitCode]

Status: RESOLVED FIXED

(Core :: JavaScript Engine, defect)

Description

[Christian Holler (:decoder)]

The following crash happened on mozilla-central revision 08592337ced1 (build with --enable-posix-nspr-emulation --enable-valgrind --enable-gczeal --disable-tests --disable-profiling --enable-debug --enable-optimize, run with --fuzzing-safe --cpu-count=2 --disable-oom-functions --disable-oom-functions --ion-eager --ion-offthread-compile=off --baseline-eager:

Backtrace:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000055d7b038b2d5 in js::jit::AutoWritableJitCode::AutoWritableJitCode (size=4920, addr=0x1d10326dfd38, rt=0x7efddd484000, this=0x7efdd8bfac18) at js/src/jit/JitRealm.h:645
#1  js::jit::AutoWritableJitCode::AutoWritableJitCode (size=4920, addr=0x1d10326dfd38, this=0x7efdd8bfac18) at js/src/jit/JitRealm.h:649
#2  mozilla::Maybe<js::jit::AutoWritableJitCode>::emplace<unsigned char*&, unsigned long&> (this=this@entry=0x7efdd8bfac18, aArgs#0=@0x7efdd8bfaa10: 0x1d10326dfd38 "", aArgs#1=@0x7efdd8bfaa00: 4920) at dist/include/mozilla/Maybe.h:599
#3  0x000055d7b037c0db in js::jit::Linker::newCode (this=this@entry=0x7efdd8bfac10, cx=cx@entry=0x7efddee16000, kind=kind@entry=js::jit::CodeKind::Ion) at js/src/jit/Linker.cpp:63
#4  0x000055d7b0250a4e in js::jit::CodeGenerator::link (this=this@entry=0x7efdce8fc000, cx=0x7efddee16000, constraints=<optimized out>) at js/src/jit/CodeGenerator.cpp:10599
#5  0x000055d7b02e014f in LinkCodeGen (cx=<optimized out>, builder=builder@entry=0x7efdcee2a220, codegen=0x7efdce8fc000) at js/src/jit/Ion.cpp:511
#6  0x000055d7b0338396 in js::jit::IonCompile (cx=<optimized out>, cx@entry=0x7efddee16000, script=<optimized out>, baselineFrame=baselineFrame@entry=0x0, osrPc=osrPc@entry=0x0, recompile=<optimized out>, optimizationLevel=<optimized out>) at js/src/jit/Ion.cpp:2221
#7  0x000055d7b0338b0c in js::jit::Compile (cx=cx@entry=0x7efddee16000, script=script@entry=..., osrFrame=osrFrame@entry=0x0, osrPc=osrPc@entry=0x0, forceRecompile=forceRecompile@entry=false) at js/src/jit/Ion.cpp:2437
#8  0x000055d7b0338c3c in js::jit::CanEnterIon (cx=cx@entry=0x7efddee16000, state=...) at js/src/jit/Ion.cpp:2528
#9  0x000055d7b0363099 in js::jit::MaybeEnterJit (cx=cx@entry=0x7efddee16000, state=...) at js/src/jit/Jit.cpp:145
#10 0x000055d7b00cb444 in js::RunScript (cx=0x7efddee16000, state=...) at js/src/vm/Interpreter.cpp:425
#11 0x000055d7b00ce9bd in js::ExecuteKernel (cx=<optimized out>, script=..., script@entry=..., envChainArg=..., newTargetValue=..., evalInFrame=..., evalInFrame@entry=..., result=result@entry=0x0) at js/src/vm/Interpreter.cpp:806
#12 0x000055d7b00cedb9 in js::Execute (cx=<optimized out>, script=script@entry=..., envChainArg=..., rval=0x0) at js/src/vm/Interpreter.cpp:839
#13 0x000055d7b06267e1 in Evaluate (cx=<optimized out>, scopeKind=scopeKind@entry=js::ScopeKind::Global, env=env@entry=..., optionsArg=..., srcBuf=..., rval=...) at js/src/vm/CompilationAndEvaluation.cpp:501
#14 0x000055d7b0627ec9 in JS::EvaluateUtf8 (cx=<optimized out>, cx@entry=0x7efddee16000, options=..., bytes=0x7efddee16020 "x\262\277\330\375~", length=<optimized out>, rval=rval@entry=...) at js/src/vm/CompilationAndEvaluation.cpp:529
#15 0x000055d7b062808b in JS::EvaluateUtf8Path (cx=0x7efddee16000, optionsArg=..., filename=0x7efdce743560 "/srv/repos/mozilla-central/js/src/jit-test/lib/../../tests/non262/shell.js", rval=...) at js/src/vm/CompilationAndEvaluation.cpp:577
#16 0x000055d7aff5b879 in LoadScript (cx=0x7efddee16000, argc=<optimized out>, vp=<optimized out>, scriptRelative=false) at js/src/shell/js.cpp:1643
#17 0x00001d10326d73a4 in ?? ()
[...]
#21 0x0000000000000000 in ?? ()
rax	0x0	0
rbx	0x7efdd8bfac18	139628728265752
rcx	0x7efe02be22dd	139629432808157
rdx	0x0	0
rsi	0x7efe02eb1770	139629435754352
rdi	0x7efe02eb0540	139629435749696
rbp	0x7efdd8bfa9d0	139628728265168
rsp	0x7efdd8bfa9b0	139628728265136
r8	0x7efe02eb1770	139629435754352
r9	0x7efdd8bff700	139628728284928
r10	0x0	0
r11	0x0	0
r12	0x7efddd484000	139628804325376
r13	0x1338	4920
r14	0x1d10326dfd38	31955402751288
r15	0x7efddd81bc00	139628808092672
rip	0x55d7b038b2d5 <mozilla::Maybe<js::jit::AutoWritableJitCode>::emplace<unsigned char*&, unsigned long&>(unsigned char*&, unsigned long&)+293>
=> 0x55d7b038b2d5 <mozilla::Maybe<js::jit::AutoWritableJitCode>::emplace<unsigned char*&, unsigned long&>(unsigned char*&, unsigned long&)+293>:	movl   $0x0,0x0
   0x55d7b038b2e0 <mozilla::Maybe<js::jit::AutoWritableJitCode>::emplace<unsigned char*&, unsigned long&>(unsigned char*&, unsigned long&)+304>:	ud2


This is a topcrasher in fuzzing and has been for a long time, but we cannot find a single reproducible testcase for this. It is highly likely that it is related to OOM situations because limiting the JS shell's memory consumption makes the issue go away.

Filing without a testcase on request of jandem and mgaudet.

Comment 1

[Matthew Gaudet (he/him) [:mgaudet]]

My read on this is that the infallible assumption of AutoWritableJitCode::AutoWritableJitCode doesn't really hold in fuzzing scenarios. 

It seems like we could build a relatively small patch to make the Linker::newCode case fallible (potentially by removing Linker::awjc, and doing the protection manually, or adding a success field to AutoWritableJitCode). 

Other uses of AutoWritableJitCode may be more challenging to handle- i.e. it's not clear how we could handle a failure of AutoWriteableJitCode in Baseline's toggle* functions, Ion's invalidation action, or WASM debug states. Those cases might be best handled with their own unhandleable oom assertion.

Comment 2

[Jan de Mooij [:jandem]]

Did this get worse at some point, decoder? In that case I think there's an underlying issue of ending up with too many memory mappings again and I'd like to understand what triggers it..

Comment 3

[Matthew Gaudet (he/him) [:mgaudet]]

At least on Linux, reading the mprotect manpage, it seems like we always need to handle the case where the kernel returns -1 and signals ENOMEM, and from discussion on IRC it seems like this may be a side effect of multiple heavily loaded processes which would stress the kernel's page mappings.

Comment 4

[Christian Holler (:decoder)]

Yes, I think this got worse at some point, but I cannot really tell you when. We have about roughly 10k crashes on file and we could group them by day, but I don't know if it goes back far enough. We throw crashes away after some months. The oldest crash we have is from End of July 2018, so those 10k are only for 2 months. But it has been around for a lot longer, I remember that.

Comment 5

[Matthew Gaudet (he/him) [:mgaudet]]

Attachment: Bug 1492845 - Create AutoWriteableJitCodeFallible to allow graceful handling of writable mapping failure in circumstances where recovery is possible r?jandem

Some creators of AutoWriteJitCode have are already fallible functions, and so we
should be able to safely fail in those functions. This will reduce the cases where
we crash due to kernel mapping failures

Comment 6

[Matthew Gaudet (he/him) [:mgaudet]]

(I think there's also potentially some value in doing some work to smuggle errno back on non-windows platforms, so that at the very lease we could crash with strerror(errno))

Comment 7

[Matthew Gaudet (he/him) [:mgaudet]]

Looking at CrashStats, it's perhaps interesting to note that the constructor failure (failure to mark writable) [1] is relatively rare, and the destructor failure is more frequent (failure to mark executable) [2] (Bug 1474214) 



[1]: https://crash-stats.mozilla.com/signature/?signature=js%3A%3Ajit%3A%3AAutoWritableJitCode%3A%3AAutoWritableJitCode&date=%3E%3D2018-06-21T12%3A15%3A25.000Z&date=%3C2018-09-21T12%3A15%3A25.000Z
[2]: https://crash-stats.mozilla.com/signature/?signature=js%3A%3Ajit%3A%3AAutoWritableJitCode%3A%3A~AutoWritableJitCode&date=%3E%3D2018-06-21T12%3A16%3A45.000Z&date=%3C2018-09-21T12%3A16%3A45.000Z

Comment 8

[Jan de Mooij [:jandem]]

Comment on attachment 9010645 [details]
Bug 1492845 - Create AutoWriteableJitCodeFallible to allow graceful handling of writable mapping failure in circumstances where recovery is possible r?jandem

Jan de Mooij [:jandem] has approved the revision.

Comment 9

[Matthew Gaudet (he/him) [:mgaudet]]

Pulsebot seems to be down: 

Pushed https://hg.mozilla.org/integration/mozilla-inbound/rev/52a3cb6a67af

Comment 10

[Eliza Balazs [:ebalazs_]]

https://hg.mozilla.org/mozilla-central/rev/52a3cb6a67af

Comment 11

[Ryan VanderMeulen [:RyanVM]]

I assume this doesn't need to be backported, but feel free to nominate if that's not the case.