840284 - (odinfuzz) OdinMonkey: fuzz

Status: RESOLVED FIXED

(Core :: JavaScript: WebAssembly, defect)

Description

[Luke Wagner [:luke]]

OdinMonkey is currently receiving a lot of testing for valid asm.js code from the Emscripten test suite and all these big beefy Emscripten apps.  What I'm hoping to get out of fuzzing is testing for the type-checker when code *doesn't* validate.  Here, Jesse pointed out that decoder's mutation fuzzer should work well since it'll randomly inject non-asm.js into the middle of valid asm.js.

Comment 1

[Luke Wagner [:luke]]

Note: we'll have to wait for bug 840280 since, atm, out-of-bounds access isn't checking which means it's quite easy to crash odin.

Comment 2

[Andrew McCreight [:mccr8]]

Isn't there going to be code that's valid but not Emscripten-y?

Comment 3

[Luke Wagner [:luke]]

Good question.  Roughly speaking, asm.js is the image of the Emscripten function.

Comment 4

[Alon Zakai (:azakai)]

(In reply to Andrew McCreight [:mccr8] from comment #2)
> Isn't there going to be code that's valid but not Emscripten-y?

I think so. Would be interesting to have a random code generator based on the type system itself, and/or to mutate asm.js from emscripten in ways that *do* keep it asm.js, not just break out of it. Both would test things just emscripten output does not.

Comment 5

[Luke Wagner [:luke]]

Additionally, it would be interesting to throw a C fuzzer in front of Emscripten :)

Comment 6

[Sam Tobin-Hochstadt [:samth]]

Csmith (http://embed.cs.utah.edu/csmith/) is a high-quality C fuzzer.  They say that they've found bugs in every tool they've tested with it.  The same research group also produces a test case reducer: http://embed.cs.utah.edu/creduce/

Comment 7

[Christian Holler (:decoder)]

As Sam already said, Csmith is the tool of choice for C mutation. LangFuzz can also mutate C with an appropriate grammar, but is not aware of any type system logic in the language, therefore unsuitable for strongly typed languages.

Comment 8

[Alon Zakai (:azakai)]

Thanks, I'll take a look at Csmith in emscripten. We should still do fuzzing on the emscripten output though, because the space of outputs generated from clang+emscripten is still constrained, even with random input to it.

Comment 9

[Luke Wagner [:luke]]

Agreed.  Between Csmith+Emscripten and decoder's mutation-based fuzzer on Emscripten output, it seems like we'll have good coverage of validating and non-validating inputs.

Comment 10

[Luke Wagner [:luke]]

Attachment: fuzzing material

Linux64 and OSX64 are done (except for the bugs, of course), so we are ready for fuzzing.  You should be able to pull http://hg.mozilla.org/users/lwagner_mozilla.com/odinmonkey/ and build.  A threadsafe shell would be useful since it allows more realistic operation-callback testing (just add a 'timeout(X)' call at the beginning and the watchdog thread will start firing interrupts every 1/10 of a second).

Attached is the Emscripten benchmark suite; just run each file individually from the shell (no args).  You should see "warning: Successfully compiled asm.js code" printed to the console.

A few notes/ideas for fuzzing:
 - It would be good to direct mutations to *inside* function containing the "use asm" directive.
 - Some of the benchmarks are tiny (corrections, memopsnjn, primes, copy) and short-running, so it would be fruitful to inject probably-valid asm.js into these scripts to see if you can get a crash or divergent behavior (compared to running either on a trunk shell or an odin shell with the "use asm" directive removed).  Note: different recursion limits are expected.
 - Other benchmarks are quite large (bullet.js, fannkuch.js, zlib) and take longer to run (esp if the code doesn't asm.js-type-check; several of these benchmarks are 5-10x slower w/o odin).  However, they compile relatively quickly (<1s) so these would be good files to mutate randomly to test for errors in the typechecker/compiler.  If you set warnings-as-errors (options("werror") toggles this option), you'll get a clean exception when compilation succeeds/fails and not have to wait for the actual script to run.

Let me know if you have any problems or ideas.  Alon said he started looking at Csmith and it's already caught an Emscripten bug; I think he'll turn the guns on Odin after he gets out from under the MWC work.

Comment 11

[Jesse Ruderman]

Can you elaborate on how to use timeout() for fuzzing?  Should I just have the fuzzer call "timeout(1800)" sometimes?  Or do you want something like bug 831046?

Comment 12

[Luke Wagner [:luke]]

(In reply to Jesse Ruderman from comment #11)
> Can you elaborate on how to use timeout() for fuzzing?

Sure.  In bug 842411, I changed the threadsafe shell so that, when you call "timeout(X)", the watchdog thread will JS_TriggerOperationCallback every 10th of a second.  Before X seconds have elapsed, the operation callback will return "true", allowing execution to continue.

> Should I just have the fuzzer call "timeout(1800)" sometimes?

That'd work.  An improvement would be to add a parameter that lets you configure how often the interrupt occurs.  I was using 10,000 times a second earlier to really stress the Odin path.

Comment 13

[Jesse Ruderman]

I'll do that, but I'm worried the testcases will depend on timing, so they won't get reduced / bisected (and they might get lost altogether).  Please fix bug 831046 :)

Comment 14

[Christian Holler (:decoder)]

I agree, using timeout for fuzzing won't work. Especially with LangFuzz, the "replay" timing is different to the timing while the fuzzer is actually running. Tests are unlikely to reproduce/reduce later on.

Comment 15

[Luke Wagner [:luke]]

That's fine.  It's easy enough for me to test this feature by cranking up frequency and running these large Emscripten codes (which I've been doing).

Comment 16

[Alon Zakai (:azakai)]

Here's the whole emscripten test suite (asm.js part, so no C++ exceptions etc.), 252 tests, hope it's useful for fuzzing.

http://dl.dropbox.com/u/80664946/emscripten_test_suite.tar.bz2

Comment 17

[Luke Wagner [:luke]]

Attachment: fuzzing material v.2

We just made a fix in the asm.js spec (http://hg.mozilla.org/users/lwagner_mozilla.com/odinmonkey/rev/f61b4dacc3d1) and type system that will reject one of the benchmarks in the previous zip (bullet).  Here is the new version of all the benchmark files.  Let the assault continue.  (Thanks!)

Comment 18

[Christian Holler (:decoder)]

Is there a reason not to use the full testsuite that Alon suggested?

Comment 19

[Alon Zakai (:azakai)]

I'll prepare another version of that updated for the new spec.

Comment 20

[Alon Zakai (:azakai)]

Here it is: http://dl.dropbox.com/u/80664946/emscripten_test_suite2.zip

Comment 21

[Christian Holler (:decoder)]

(In reply to Alon Zakai (:azakai) from comment #20)
> Here it is: http://dl.dropbox.com/u/80664946/emscripten_test_suite2.zip

Thanks Alon :)

Comment 22

[Alon Zakai (:azakai)]

I've been running the csmith fuzzer, which generates C code, then compiling that to asm.js. So far I've not found a single bug in OdinMonkey, but I've found several in emscripten and LLVM.

decoder, if it would be useful, I can provide a bunch (let me know how many) of randomly-generated asm.js code from csmith for JS-level fuzzing.

Comment 23

[Alon Zakai (:azakai)]

Attachment: csmith fuzzing material

Here are ~100 csmith reduced testcases. I turned on full debug logging in them so they do more ffi calls.

Comment 24

[Luke Wagner [:luke]]

Christian: Linux32 and OSX32 should work now, please feel free to fuzz them.  They use a different protection mechanism (segmentation) and the IonMonkey load/store codegen is different, so it'd be valuable to give one of these a go.

Also: should I take the pause in fuzz bugs to indicate you've aimed your barrels elsewhere for the time being or that Odin is surviving?

Comment 25

[Christian Holler (:decoder)]

(In reply to Luke Wagner [:luke] from comment #24)
> Christian: Linux32 and OSX32 should work now, please feel free to fuzz them.
> They use a different protection mechanism (segmentation) and the IonMonkey
> load/store codegen is different, so it'd be valuable to give one of these a
> go.

Alright, I'll give it a try on Monday for Linux.

> 
> Also: should I take the pause in fuzz bugs to indicate you've aimed your
> barrels elsewhere for the time being or that Odin is surviving?

Both. I have one more issue with Odin I think, but I currently have to take two days of PTO per week to get all my vacation from last year spent :) So I'm not working as intensively as I am normally.

Comment 26

[BMO Automation]

Per policy at https://wiki.mozilla.org/Bug_Triage/Projects/Bug_Handling/Bug_Husbandry#Inactive_Bugs. If this bug is not an enhancement request or a bug not present in a supported release of Firefox, then it may be reopened.

Comment 27

[Benjamin Bouvier [:bbouvier] (inactive)]

Probably fixed nowadays :)