1903709 - Crash in [@ OOM | unknown | js::AutoEnterOOMUnsafeRegion::crash_impl | js::AutoEnterOOMUnsafeRegion::crash | V8::FatalProcessOutOfMemory]

Status: NEW

(Core :: JavaScript Engine, defect, P3)

Description

[Chris Peterson [:cpeterson]]

This is not a new crash signature, but its volume appears to have been spiked in Fx 126. Did we update the V8 irregexp in Fx 126?

About 84% of these RegExpCompiler OOM crash reports are from Android.

Crash report: https://crash-stats.mozilla.org/report/index/b575eb31-9f22-4493-a7a9-ad93e0240618

Reason: SIGSEGV / SEGV_MAPERR

Top 10 frames:

0  libxul.so  MOZ_Crash(char const*, int, char const*)  mfbt/Assertions.h:317
0  libxul.so  js::AutoEnterOOMUnsafeRegion::crash_impl(char const*)  js/src/vm/JSContext.cpp:1344
1  libxul.so  js::AutoEnterOOMUnsafeRegion::crash(char const*)  js/public/Utility.h:309
1  libxul.so  V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*)  js/src/irregexp/RegExpShim.h:1338
1  libxul.so  v8::internal::RegExpCompiler::ToNodeCheckForStackOverflow()  js/src/irregexp/imported/regexp-compiler.cc:3982
2  libxul.so  v8::internal::RegExpCompiler::ToNodeMaybeCheckForStackOverflow()  js/src/irregexp/imported/regexp-compiler.h:557
2  libxul.so  v8::internal::RegExpAlternative::ToNode(v8::internal::RegExpCompiler*, v8::in...  js/src/irregexp/imported/regexp-compiler-tonode.cc:1336
3  libxul.so  v8::internal::RegExpCapture::ToNode(v8::internal::RegExpTree*, int, v8::inter...  js/src/irregexp/imported/regexp-compiler-tonode.cc:1244
4  libxul.so  v8::internal::RegExpCompiler::PreprocessRegExp(v8::internal::RegExpCompileDat...  js/src/irregexp/imported/regexp-compiler.cc:3941
5  libxul.so  js::irregexp::CompilePattern(JSContext*, JS::MutableHandle<js::RegExpShared*>...  js/src/irregexp/RegExpAPI.cpp:848

Comment 1

[Chris Peterson [:cpeterson]]

irregexp was updated in Fx 125 by bug 1882696. Maybe this OOM increase in Fx 126 is a regression from that update?

irregexp was updated again in Fx 128 by bug 1895871, so we can watch to see if that update resolved this OOM.

Comment 2

[Iain Ireland [:iain]]

Looking at the graph over the last three months by major version, it seems like the increase started in early May with Fx 125, then picked up a week or so later. I also see a lot of crashes in ESR115. There have always been crashes in ESR115, but it seems like the rate has gone up: looking at crashes in the last six months, and picking a dividing line of May 4 (which is around when we started seeing the increase in release), I see about 6.5 crashes/day before that point, but 19 crashes/day after. We haven't uplifted

It looks like we don't have any URL information for Android crashes. Looking only at desktop Firefox, in the last 3 months, there have been 1196 crashes, of which 1115 have no URL information. Of the remaining 84, 63 are on bild.de (a German newspaper). That's only a small fraction of the overall crashes, but it seems significant.

The stack trace in comment 0 is actually slightly unusual: I see 162 crashes in the last 3 months under RegExpAlternative, compared to 10,161 over the same time period under RegExpDisjunction. The irregexp changes linked in comment 1 shouldn't have affected either of these. Bug 1895871 did add a new stack overflow check here, but that doesn't show up in a single crash, so I think it's unlikely to be related.

Notably, none of the crash reports I've looked at is recursing particularly deeply inside the regexp parser. Instead, it looks like these crashes happen when we are already deeply recursed, and then parse a regexp.

Putting this all together, my best guess is that the change is external. Possibly bild.de has shipped some code that makes them more likely to trigger a regexp with a very deep stack, and maybe that hits Android hardest because is has less stack space overall.

Fixing this would be tricky. The problem is in code that we import from V8. There's a TODO in the upstream codebase to rewrite this code to add the ability to abort regexp compilation for stack overflow. In the meantime, I would generally expect Chrome to have similar problems in this case, although it's possible that they have different stack limits / stack frame sizes that would shift the crash rate higher or lower.

Unfortunately I don't think these crashes give us enough information to go to bild.de and recommend a particular change to their website.