Closed Bug 840284 (odinfuzz) Opened 8 years ago Closed 10 months ago

OdinMonkey: fuzz

Categories

(Core :: Javascript: WebAssembly, defect)

defect
Not set
normal

Tracking

()

RESOLVED FIXED

People

(Reporter: luke, Assigned: luke)

References

Details

Attachments

(2 files, 1 obsolete file)

OdinMonkey is currently receiving a lot of testing for valid asm.js code from the Emscripten test suite and all these big beefy Emscripten apps.  What I'm hoping to get out of fuzzing is testing for the type-checker when code *doesn't* validate.  Here, Jesse pointed out that decoder's mutation fuzzer should work well since it'll randomly inject non-asm.js into the middle of valid asm.js.
Blocks: 840282
Note: we'll have to wait for bug 840280 since, atm, out-of-bounds access isn't checking which means it's quite easy to crash odin.
Isn't there going to be code that's valid but not Emscripten-y?
Good question.  Roughly speaking, asm.js is the image of the Emscripten function.
(In reply to Andrew McCreight [:mccr8] from comment #2)
> Isn't there going to be code that's valid but not Emscripten-y?

I think so. Would be interesting to have a random code generator based on the type system itself, and/or to mutate asm.js from emscripten in ways that *do* keep it asm.js, not just break out of it. Both would test things just emscripten output does not.
Additionally, it would be interesting to throw a C fuzzer in front of Emscripten :)
Csmith (http://embed.cs.utah.edu/csmith/) is a high-quality C fuzzer.  They say that they've found bugs in every tool they've tested with it.  The same research group also produces a test case reducer: http://embed.cs.utah.edu/creduce/
As Sam already said, Csmith is the tool of choice for C mutation. LangFuzz can also mutate C with an appropriate grammar, but is not aware of any type system logic in the language, therefore unsuitable for strongly typed languages.
Thanks, I'll take a look at Csmith in emscripten. We should still do fuzzing on the emscripten output though, because the space of outputs generated from clang+emscripten is still constrained, even with random input to it.
Agreed.  Between Csmith+Emscripten and decoder's mutation-based fuzzer on Emscripten output, it seems like we'll have good coverage of validating and non-validating inputs.
Attached file fuzzing material (obsolete) —
Linux64 and OSX64 are done (except for the bugs, of course), so we are ready for fuzzing.  You should be able to pull http://hg.mozilla.org/users/lwagner_mozilla.com/odinmonkey/ and build.  A threadsafe shell would be useful since it allows more realistic operation-callback testing (just add a 'timeout(X)' call at the beginning and the watchdog thread will start firing interrupts every 1/10 of a second).

Attached is the Emscripten benchmark suite; just run each file individually from the shell (no args).  You should see "warning: Successfully compiled asm.js code" printed to the console.

A few notes/ideas for fuzzing:
 - It would be good to direct mutations to *inside* function containing the "use asm" directive.
 - Some of the benchmarks are tiny (corrections, memopsnjn, primes, copy) and short-running, so it would be fruitful to inject probably-valid asm.js into these scripts to see if you can get a crash or divergent behavior (compared to running either on a trunk shell or an odin shell with the "use asm" directive removed).  Note: different recursion limits are expected.
 - Other benchmarks are quite large (bullet.js, fannkuch.js, zlib) and take longer to run (esp if the code doesn't asm.js-type-check; several of these benchmarks are 5-10x slower w/o odin).  However, they compile relatively quickly (<1s) so these would be good files to mutate randomly to test for errors in the typechecker/compiler.  If you set warnings-as-errors (options("werror") toggles this option), you'll get a clean exception when compilation succeeds/fails and not have to wait for the actual script to run.

Let me know if you have any problems or ideas.  Alon said he started looking at Csmith and it's already caught an Emscripten bug; I think he'll turn the guns on Odin after he gets out from under the MWC work.
Can you elaborate on how to use timeout() for fuzzing?  Should I just have the fuzzer call "timeout(1800)" sometimes?  Or do you want something like bug 831046?
(In reply to Jesse Ruderman from comment #11)
> Can you elaborate on how to use timeout() for fuzzing?

Sure.  In bug 842411, I changed the threadsafe shell so that, when you call "timeout(X)", the watchdog thread will JS_TriggerOperationCallback every 10th of a second.  Before X seconds have elapsed, the operation callback will return "true", allowing execution to continue.

> Should I just have the fuzzer call "timeout(1800)" sometimes?

That'd work.  An improvement would be to add a parameter that lets you configure how often the interrupt occurs.  I was using 10,000 times a second earlier to really stress the Odin path.
I'll do that, but I'm worried the testcases will depend on timing, so they won't get reduced / bisected (and they might get lost altogether).  Please fix bug 831046 :)
I agree, using timeout for fuzzing won't work. Especially with LangFuzz, the "replay" timing is different to the timing while the fuzzer is actually running. Tests are unlikely to reproduce/reduce later on.
That's fine.  It's easy enough for me to test this feature by cranking up frequency and running these large Emscripten codes (which I've been doing).
Attachment #716296 - Attachment is patch: false
Attachment #716296 - Attachment mime type: text/plain → application/zip
Here's the whole emscripten test suite (asm.js part, so no C++ exceptions etc.), 252 tests, hope it's useful for fuzzing.

http://dl.dropbox.com/u/80664946/emscripten_test_suite.tar.bz2
Depends on: 843934
Depends on: 844334
Depends on: 844335
Depends on: 844355
Depends on: 844360
Depends on: 844365
Depends on: 844372
Depends on: 844476
Depends on: 844477
Depends on: 844480
Depends on: 844483
Depends on: 844633
Depends on: 844645
Depends on: 844665
Depends on: 844836
Depends on: 845046
Attached file fuzzing material v.2
We just made a fix in the asm.js spec (http://hg.mozilla.org/users/lwagner_mozilla.com/odinmonkey/rev/f61b4dacc3d1) and type system that will reject one of the benchmarks in the previous zip (bullet).  Here is the new version of all the benchmark files.  Let the assault continue.  (Thanks!)
Attachment #716296 - Attachment is obsolete: true
Is there a reason not to use the full testsuite that Alon suggested?
I'll prepare another version of that updated for the new spec.
(In reply to Alon Zakai (:azakai) from comment #20)
> Here it is: http://dl.dropbox.com/u/80664946/emscripten_test_suite2.zip

Thanks Alon :)
Depends on: 846304
Depends on: 846317
Depends on: 847387
No longer depends on: 847387
I've been running the csmith fuzzer, which generates C code, then compiling that to asm.js. So far I've not found a single bug in OdinMonkey, but I've found several in emscripten and LLVM.

decoder, if it would be useful, I can provide a bunch (let me know how many) of randomly-generated asm.js code from csmith for JS-level fuzzing.
Here are ~100 csmith reduced testcases. I turned on full debug logging in them so they do more ffi calls.
Christian: Linux32 and OSX32 should work now, please feel free to fuzz them.  They use a different protection mechanism (segmentation) and the IonMonkey load/store codegen is different, so it'd be valuable to give one of these a go.

Also: should I take the pause in fuzz bugs to indicate you've aimed your barrels elsewhere for the time being or that Odin is surviving?
(In reply to Luke Wagner [:luke] from comment #24)
> Christian: Linux32 and OSX32 should work now, please feel free to fuzz them.
> They use a different protection mechanism (segmentation) and the IonMonkey
> load/store codegen is different, so it'd be valuable to give one of these a
> go.

Alright, I'll give it a try on Monday for Linux.

> 
> Also: should I take the pause in fuzz bugs to indicate you've aimed your
> barrels elsewhere for the time being or that Odin is surviving?

Both. I have one more issue with Odin I think, but I currently have to take two days of PTO per week to get all my vacation from last year spent :) So I'm not working as intensively as I am normally.
Depends on: 854448
Depends on: 855442
Depends on: 855526
Depends on: 878417
Alias: odinfuzz
Depends on: 878429
Depends on: 878433
Depends on: 878435
Depends on: 878444
Depends on: 878495
Depends on: 878505
Depends on: 878520
No longer depends on: 878429
No longer depends on: 878444
Depends on: 881608
Depends on: 882008
Depends on: 882012
Depends on: 883490
Depends on: 886246
Depends on: 886248
Depends on: 886285
Depends on: 888568
Depends on: 893368
No longer depends on: 927389
No longer depends on: 951527
Depends on: 970643
No longer depends on: 1005590
Per policy at https://wiki.mozilla.org/Bug_Triage/Projects/Bug_Handling/Bug_Husbandry#Inactive_Bugs. If this bug is not an enhancement request or a bug not present in a supported release of Firefox, then it may be reopened.
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → INACTIVE
Status: RESOLVED → REOPENED
Resolution: INACTIVE → ---
Component: JavaScript Engine → Javascript: WebAssembly

Probably fixed nowadays :)

Status: REOPENED → RESOLVED
Closed: 2 years ago10 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.