Closed Bug 558860 Opened 15 years ago Closed 14 years ago

Firefox 3.6.3 Crash Report [@ js_math_max | NPSWF32.dll@0xab785]

Categories

(External Software Affecting Firefox Graveyard :: Flash (Adobe), defect)

Product:
External Software Affecting Firefox Graveyard
Component:
Flash (Adobe)
Platform:
x86
Windows XP
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: chofmann, Unassigned)

Details

(Keywords: crash, Whiteboard: [sg:vector-critical (Adobe Flash)], Flash Player 10.1.53.38)

Crash Data

#143 top crash that seems to appear only on firefox 3.6.3 at this point and its only on WinNT 5.1 (winXP) a lot of the crashes seem to be on youtube domains of sites 97 http://www.youtube.com 8 \N// 5 http://badoo.com 4 http://www.google.com 4 http://www.facebook.com 3 http://video.mail.ru 2 http://www.iraqgoals.tv 2 http://www.google.co.in stack looks like below with flash then js at the top of the stack. http://crash-stats.mozilla.com/report/index/7eb5cb71-50e0-4994-b48b-824802100410 Frame Module Signature [Expand] Source 0 js3250.dll js_math_max 1 NPSWF32.dll NPSWF32.dll@0xab785 2 NPSWF32.dll NPSWF32.dll@0xedcf1 3 NPSWF32.dll NPSWF32.dll@0x139bdb 4 NPSWF32.dll NPSWF32.dll@0x13a16b 5 xul.dll nsNPAPIPluginInstance::Stop modules/plugin/base/src/nsNPAPIPluginInstance.cpp:1178 6 xul.dll DoStopPlugin layout/generic/nsObjectFrame.cpp:2248 7 xul.dll nsStopPluginRunnable::Run layout/generic/nsObjectFrame.cpp:2298 8 xul.dll nsThread::ProcessNextEvent xpcom/threads/nsThread.cpp:527 9 xul.dll nsBaseAppShell::Run widget/src/xpwidgets/nsBaseAppShell.cpp:170 10 xul.dll nsAppStartup::Run toolkit/components/startup/src/nsAppStartup.cpp:183 11 nspr4.dll nspr4.dll@0xbdcf 12 firefox.exe wmain toolkit/xre/nsWindowsWMain.cpp:120 13 firefox.exe __tmainCRTStartup obj-firefox/memory/jemalloc/crtsrc/crtexe.c:591 14 kernel32.dll BaseProcessStart more reports at http://crash-stats.mozilla.com/report/list?range_value=2&range_unit=weeks&signature=js_math_max&version=Firefox%3A3.6.3
Could be flash jumping off into a bad address, which happens to lie in that range. Are the EIP values all the same? Same Flash addresses or versions? Flash *could* be calling js_math_max, since it's exported, but I would be pretty shocked.
Are you sure we export that symbol? It wouldn't be intentional.
Pretty sure; it's not static, and we don't do visibility magic on the JS engine, afaik.
Why do we go through all the JS_PUBLIC_API trickery then if everything is exported anyway? I am pretty sure I get linker errors for everything thats not JS_PUBLIC_API or JS_FRIEND_API.
It varies by platform and how we compile and link stuff. Back to this bug: didn't we have a similar bug in the last year where Flash was jumping into some code in the JS engine it could not possibly have intended to call? Can someone find that bug? I hope the valgrind probs where Flash was reading uninitialized memory, which may have been bugs in our plugin code IIRC, aren't to blame. /be
I checked out the dump. js_math_max is being called with arguments all zero. Thus: JSBool js_math_max(JSContext *cx, uintN argc, jsval *vp) { jsdouble x, z = js_NegativeInfinity; jsval *argv; uintN i; if (argc == 0) { ### NPE on the following line because vp == NULL. *vp = cx->runtime->negativeInfinityValue; return JS_TRUE; } This is being called from somewhere in Flash, so it looks like it must be a bad jump/call from Flash. I found bug 535550 as an earlier instance of this kind of thing.
Assignee: general → nobody
Severity: normal → critical
Component: JavaScript Engine → Flash (Adobe)
Keywords: crash
Product: Core → Plugins
QA Contact: general → adobe-flash
Summary: Firefox 3.6.3 Crash Report [@ js_math_max ] → Firefox 3.6.3 Crash Report [@ js_math_max | NPSWF32.dll@0xab785]
Version: Trunk → 10.x
looks like there might be something similar going on with crashes like this. http://crash-stats.mozilla.com/report/index/e29283f9-7fca-48da-921b-6eb752100407 Frame Module Signature [Expand] Source 0 ssl3.dll ssl3.dll@0x20fff 1 NPSWF32.dll NPSWF32.dll@0xab785 2 NPSWF32.dll NPSWF32.dll@0xedcf1 3 NPSWF32.dll NPSWF32.dll@0x139bdb 4 NPSWF32.dll NPSWF32.dll@0x13a16b 5 xul.dll nsNPAPIPluginInstance::Stop modules/plugin/base/src/nsNPAPIPluginInstance.cpp:1178 6 xul.dll DoStopPlugin layout/generic/nsObjectFrame.cpp:2248 7 xul.dll nsStopPluginRunnable::Run layout/generic/nsObjectFrame.cpp:2298 8 xul.dll nsThread::ProcessNextEvent xpcom/threads/nsThread.cpp:527 9 xul.dll nsBaseAppShell::Run widget/src/xpwidgets/nsBaseAppShell.cpp:170 10 xul.dll nsAppStartup::Run toolkit/components/startup/src/nsAppStartup.cpp:183 11 nspr4.dll nspr4.dll@0xbdcf 12 firefox.exe wmain toolkit/xre/nsWindowsWMain.cpp:120 13 firefox.exe __tmainCRTStartup obj-firefox/memory/jemalloc/crtsrc/crtexe.c:591 14 kernel32.dll BaseProcessStart should this get moved over to plugins:flash or are there defenses on both sides of the npapi?
(I'm investigating this from the Flash point of view...) (In reply to comment #1) > Flash *could* be calling js_math_max, since it's exported, but I would be > pretty shocked. Flash is definitely not calling it deliberately. > Could be flash jumping off into a bad address, which happens to lie in that > range. Are the EIP values all the same? Same Flash addresses or versions? It's possible -- sure seems weird that we'd just happen to call an exported function precisely at its entry point. (In reply to comment #8) > looks like there might be something similar going on with crashes like this. Yeah, evidence looks compelling. Any idea how easy to repro this is -- have we seen it happen in captivity, or do we only know of it via crash log analysis? Also -- do crash logs of this sort have any indication of the version(s) of Flash in use?
Yep, if you click the crash report and then the "modules" tab, you'll see the Flash and other library versions. For the one in comment 8 it's 10.0.45.2.
> > (In reply to comment #8) > > looks like there might be something similar going on with crashes like this. > > Yeah, evidence looks compelling. Any idea how easy to repro this is -- have we > seen it happen in captivity, or do we only know of it via crash log analysis? > only by crash log analysis so far. here are some user comments. not much useful. js_math_max I'm getting sick and tired of Firefox crashing everytime I watch a video on YouTube!!! js_math_max Youtube again.. | Clicked a video. js_math_max not sure what happened i clicked on a link on the iowalottery.com and it crashed js_math_max Peegel AA Trojan ssl3.dll@0x20fff I got a notice from you that mozilla has crashed....yahoo.com ssl3.dll@0x20fff I pressed a tab, and it just crashed ssl3.dll@0x20fff "Crashed when entering the ""back"" arrow after opening a PDF stored in an online database." ssl3.dll@0x20fff clicked a link that went to a youtube channle and hit the back button in my browser qucikly. This casused the error. ssl3.dll@0x20fff estava no programa Surpreenda Mastercard ssl3.dll@0x20fff On a Video sharing site and the BBC and then Firefox crashed... ssl3.dll@0x20fff Firefox crashes when I open a pdf inside the browser and then hit the back button. It goes back to the original page and hangs. ssl3.dll@0x20fff "This crash sometimes occurs when I use the ""back"" button. It has happened a few times already." > Also -- do crash logs of this sort have any indication of the version(s) of > Flash in use? still waiting on some new ways to get at that data. hopefully soon. until then I try to dig some info out looking at a bunch of reports.
(In reply to comment #10) > Yep, if you click the crash report and then the "modules" tab, you'll see the > Flash and other library versions. For the one in comment 8 it's 10.0.45.2. From a random sampling of the crashes, most seem to be running 10.0.45.2 (the most recent release) but a handful are running earlier versions of 10. Haven't seen any running 9.x yet. The thing that seems odd is that we'd be calling a "real" function -- if it's just a random memory overwrite I wouldn't expect so many crashes to end up at the same call, with the same (zeroed) args. Is js_math_max exposed in the NPAPI in some way?
Looks like the comment stacktrace is that Flash is called from nsNPAPIPluginInstance::Stop -- I presume this (most likely) corresponds to calling into NPP_Destroy; perhaps we have some problematic condition with our teardown code. Investigating...
for a sample of 100 of the js_math_max reports here are the version numbers 63 NPSWF32.dll 10.0.45.2 11 NPSWF32.dll 10.0.32.18 10 NPSWF32.dll 10.0.42.34 10 NPSWF32.dll 10.0.22.87 2 NPSWF32.dll 10.0.12.36 1 NPSWF32.dll 10.1.53.7 1 NPSWF32.dll 10.0.32.18 ssl3 reports are about the same distribution.
FWIW, there's a handful of the crash reports that as crashing in js_math_max without NPSWF32.dll on the call stack -- a tiny percentage so it seems likely that Flash is somehow to blame, but still puzzling...
0x00300000 is the address that mozjs32.dll is loaded at (and it doesn't have ASLR, whoops), and js_math_max is early in that DLL (EIP here is at 0x78 from the base), so I would look for a case where you call a virtual method at 0x78 from a pointer, which is itself getting corrupted somewhere. Possibly because it's already been shut down and reallocated as something else? Could well be that people are watching youtube and then another instance is getting shutdown while this one is running (since there are a lot of complaints about youtube, and we sure aren't shutting that instance down while it's playing!). Might even be that it's a previous youtube page being expired from the bfcache, but could be any number of flash ads or games. Do you see anything if you run under Purify or similar with some of the patterns above?
Hey look, I just sketched an exploit direction, possibly. Closing for consideration of whether that's a real risk from my comment 16.
Group: core-security
(In reply to comment #16) > I would look for a case where you call a virtual method at 0x78 > from a pointer, which is itself getting corrupted somewhere. Possibly because > it's already been shut down and reallocated as something else? Yeah, that's what I had figured on too. I'm going to get my XP dev system working again (investigation so far was on Win7, because that's what I had hand) and dig deeper -- the tracebacks are remarkably stable, so if I can get an identical memory layout I should be able to figure out the badness.
Whiteboard: [sg:vector-critical (Adobe Flash)]
This is looking a lot like a bug that was recently fixed in FP10.1 -- it will be hard to confirm definitively though.
here is the one crash found from 10.1.53.7 that was in that sample from comment 14 http://crash-stats.mozilla.com/report/index/41daee28-b396-4ba8-af15-f65f82100411
(In reply to comment #20) > here is the one crash found from 10.1.53.7 that was in that sample from comment That's odd -- I don't think that build was ever sent out, must have been a crash from internal testing. (Unfortunately, I think that build should have had the fix that I was hoping addressed it)
(In reply to comment #20) > here is the one crash found from 10.1.53.7 FWIW, that stacktrace looks quite different from the 10.0 ones I've looked at -- 10.0 seems to be a particular trace originating from NPP_Destroy. We may have a different crasher with a similar result. Ugh.
OK, so the good news here is it looks like the "fix" in 10.1 is incomplete, but we're pretty sure it's still the same bug, and we think we know how to fix the fix.
Update: -- definitely a bug in Flash, not Firefox. -- existing builds of 10.0 and 10.1 all have the bug in some form -- a fix is being applied internally and should be in the final shipping version of 10.1 -- it's extremely unlikely that this could be used for a useful security exploit, as it would require predicting certain GC behaviors to an unlikely degree; that said, it's theoretically possible, so let's keep this bug marked as security-sensitive for now. Given all this, not sure what you want the disposition of this bug to be -- close it out? Keep it open to track against 10.1? (The bug isn't a bug in Tamarin, but in another, non-open-sourced part of Flash, so it doesn't make sense to transfer it to the Tamarin bugbase.)
one option would be to mark it fixed some time around when a beta or release version appears, then we could verify by trying to watch the crash volume numbers go down and it starts to have some positive impact on firefox users. we haven't really figured out a good time to mark these plugin:other_vendor bugs as fixed so suggestions welcome.
a few (possibly dumb) questions are still floating around in my head. still wondering about my question in comment 8 about if there are defenses against this crash on the gecko side. now that there is a better understanding about what flash was doing is it possible other plugins could be doing the same unintentionally (or maliciously) and would it be possible for problems like this to be lurking elsewhere. How could we look for such things in the crash data? Steven mentioned the ability to make this exploitable if predicting GC got easier or more understandable (turning his words around). I'm guessing flash GC, but is it GC on the flash side or the gecko side? Sounds like a possible calling card for nils at next year's pwn2own so it would be good to understand the details of that better.
(In reply to comment #26) > still wondering about my question in comment 8 about if there are defenses > against this crash on the gecko side. Moving plugins to a separate process (which is underway IIUC) would do the trick, but otherwise I don't think there's much you can do on your side. Mike nailed the cause: a deleted object is having a virtual function called on it. Where you land depends on what the deleted memory happens to point to when/if it gets reused for something else. > Steven mentioned the ability to make this exploitable if predicting GC got > easier or more understandable (turning his words around). I'm guessing flash > GC, but is it GC on the flash side or the gecko side? On the Flash side.
The expectation is this fix should be available in our latest RC release of Flash Player 10.1 (version 10.1.53.38) released last night
Hey Chris. How is the crash volume on this after 10.1.53.38?
these are the only flash 10.1 crashes we have seen durning May on this signature. looks like all 10.1.53.7 or older. looks like this can be marked fixed, and we could probably open the bug up at some point when you get users moving to 10.1 and/or we get our users on 3.6.4 with out of process plugins. want to race? ;-) $ awk -F\t '$1 ~ /js_math_max/ && $22 ~ /^10\.1/ {print FILENAME,$22,$3}' 201005* 20100505-crashdata.csv 10.1.53.7 http://crash-stats.mozilla.com/report/index/e698808f-d0fa-47de-a61d-468232100505 20100509-crashdata.csv 10.1.53.7 http://crash-stats.mozilla.com/report/index/57b674f8-0c16-48c8-84e6-678872100509 20100516-crashdata.csv 10.1.53.7 http://crash-stats.mozilla.com/report/index/fef9eadf-bdb7-46c5-a55e-107f92100516 20100524-crashdata.csv 10.1.51.95 http://crash-stats.mozilla.com/report/index/1873ccc5-06f9-4e61-8bf3-989ce2100524 20100526-crashdata.csv 10.1.53.7 http://crash-stats.mozilla.com/report/index/5a12d56a-9679-486d-8f0b-acb422100526 20100527-crashdata.csv 10.1.53.21 http://crash-stats.mozilla.com/report/index/301bf2db-095f-4c7f-8b2b-129242100527
there is one report of the ssl3.dll signature on 10.1.53.38 but flash is not lower on the stack like in comment 8. awk -F\t '$1 ~ /ssl3.dll@0x20fff/ && $22 ~ /^10\.1/ {print FILENAME,$22,$3}' 201005* 20100511-crashdata.csv 10.1.53.7 http://crash-stats.mozilla.com/report/index/d1478fc3-478f-4067-be63-2f4652100511 20100513-crashdata.csv 10.1.53.7 http://crash-stats.mozilla.com/report/index/ab96aba7-4fff-4e1c-ae99-71f5b2100513 20100524-crashdata.csv 10.1.53.7 http://crash-stats.mozilla.com/report/index/eb55b6f6-04e3-4062-8184-f7dc22100524 20100524-crashdata.csv 10.1.53.7 http://crash-stats.mozilla.com/report/index/c54ea54d-4af5-4c1a-9ae3-bb73b2100524 20100526-crashdata.csv 10.1.53.7 http://crash-stats.mozilla.com/report/index/0ab3dc69-0e0c-4923-a59a-5cb7d2100526 20100526-crashdata.csv 10.1.53.38 http://crash-stats.mozilla.com/report/index/f056ab59-91eb-40a5-827f-5efe42100526
I think we have enough to close this at this point. Thanks for providing the data.
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
Whiteboard: [sg:vector-critical (Adobe Flash)] → [sg:vector-critical (Adobe Flash)], Flash Player 10.1.53.38
Crash Signature: [@ js_math_max | NPSWF32.dll@0xab785]
Group: core-security → core-security-release
Group: core-security-release
Version and milestone values are being reset to defaults as part of product refactoring.
Version: 10.x → unspecified
Product: External Software Affecting Firefox → External Software Affecting Firefox Graveyard
You need to log in before you can comment on or make changes to this bug.