Status

()

Core
JavaScript Engine
5 years ago
3 years ago

People

(Reporter: luke, Assigned: luke)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments, 1 obsolete attachment)

1.04 MB, application/zip
Details
930.00 KB, application/octet-stream
Details
(Assignee)

Description

5 years ago
OdinMonkey is currently receiving a lot of testing for valid asm.js code from the Emscripten test suite and all these big beefy Emscripten apps.  What I'm hoping to get out of fuzzing is testing for the type-checker when code *doesn't* validate.  Here, Jesse pointed out that decoder's mutation fuzzer should work well since it'll randomly inject non-asm.js into the middle of valid asm.js.
(Assignee)

Updated

5 years ago
Blocks: 840282
(Assignee)

Comment 1

5 years ago
Note: we'll have to wait for bug 840280 since, atm, out-of-bounds access isn't checking which means it's quite easy to crash odin.
Isn't there going to be code that's valid but not Emscripten-y?
(Assignee)

Comment 3

5 years ago
Good question.  Roughly speaking, asm.js is the image of the Emscripten function.

Comment 4

5 years ago
(In reply to Andrew McCreight [:mccr8] from comment #2)
> Isn't there going to be code that's valid but not Emscripten-y?

I think so. Would be interesting to have a random code generator based on the type system itself, and/or to mutate asm.js from emscripten in ways that *do* keep it asm.js, not just break out of it. Both would test things just emscripten output does not.
(Assignee)

Comment 5

5 years ago
Additionally, it would be interesting to throw a C fuzzer in front of Emscripten :)
Csmith (http://embed.cs.utah.edu/csmith/) is a high-quality C fuzzer.  They say that they've found bugs in every tool they've tested with it.  The same research group also produces a test case reducer: http://embed.cs.utah.edu/creduce/
As Sam already said, Csmith is the tool of choice for C mutation. LangFuzz can also mutate C with an appropriate grammar, but is not aware of any type system logic in the language, therefore unsuitable for strongly typed languages.

Comment 8

5 years ago
Thanks, I'll take a look at Csmith in emscripten. We should still do fuzzing on the emscripten output though, because the space of outputs generated from clang+emscripten is still constrained, even with random input to it.
(Assignee)

Comment 9

5 years ago
Agreed.  Between Csmith+Emscripten and decoder's mutation-based fuzzer on Emscripten output, it seems like we'll have good coverage of validating and non-validating inputs.
(Assignee)

Comment 10

5 years ago
Created attachment 716296 [details]
fuzzing material

Linux64 and OSX64 are done (except for the bugs, of course), so we are ready for fuzzing.  You should be able to pull http://hg.mozilla.org/users/lwagner_mozilla.com/odinmonkey/ and build.  A threadsafe shell would be useful since it allows more realistic operation-callback testing (just add a 'timeout(X)' call at the beginning and the watchdog thread will start firing interrupts every 1/10 of a second).

Attached is the Emscripten benchmark suite; just run each file individually from the shell (no args).  You should see "warning: Successfully compiled asm.js code" printed to the console.

A few notes/ideas for fuzzing:
 - It would be good to direct mutations to *inside* function containing the "use asm" directive.
 - Some of the benchmarks are tiny (corrections, memopsnjn, primes, copy) and short-running, so it would be fruitful to inject probably-valid asm.js into these scripts to see if you can get a crash or divergent behavior (compared to running either on a trunk shell or an odin shell with the "use asm" directive removed).  Note: different recursion limits are expected.
 - Other benchmarks are quite large (bullet.js, fannkuch.js, zlib) and take longer to run (esp if the code doesn't asm.js-type-check; several of these benchmarks are 5-10x slower w/o odin).  However, they compile relatively quickly (<1s) so these would be good files to mutate randomly to test for errors in the typechecker/compiler.  If you set warnings-as-errors (options("werror") toggles this option), you'll get a clean exception when compilation succeeds/fails and not have to wait for the actual script to run.

Let me know if you have any problems or ideas.  Alon said he started looking at Csmith and it's already caught an Emscripten bug; I think he'll turn the guns on Odin after he gets out from under the MWC work.

Comment 11

5 years ago
Can you elaborate on how to use timeout() for fuzzing?  Should I just have the fuzzer call "timeout(1800)" sometimes?  Or do you want something like bug 831046?
(Assignee)

Comment 12

5 years ago
(In reply to Jesse Ruderman from comment #11)
> Can you elaborate on how to use timeout() for fuzzing?

Sure.  In bug 842411, I changed the threadsafe shell so that, when you call "timeout(X)", the watchdog thread will JS_TriggerOperationCallback every 10th of a second.  Before X seconds have elapsed, the operation callback will return "true", allowing execution to continue.

> Should I just have the fuzzer call "timeout(1800)" sometimes?

That'd work.  An improvement would be to add a parameter that lets you configure how often the interrupt occurs.  I was using 10,000 times a second earlier to really stress the Odin path.

Comment 13

5 years ago
I'll do that, but I'm worried the testcases will depend on timing, so they won't get reduced / bisected (and they might get lost altogether).  Please fix bug 831046 :)
I agree, using timeout for fuzzing won't work. Especially with LangFuzz, the "replay" timing is different to the timing while the fuzzer is actually running. Tests are unlikely to reproduce/reduce later on.
(Assignee)

Comment 15

5 years ago
That's fine.  It's easy enough for me to test this feature by cranking up frequency and running these large Emscripten codes (which I've been doing).
Attachment #716296 - Attachment is patch: false
Attachment #716296 - Attachment mime type: text/plain → application/zip
Here's the whole emscripten test suite (asm.js part, so no C++ exceptions etc.), 252 tests, hope it's useful for fuzzing.

http://dl.dropbox.com/u/80664946/emscripten_test_suite.tar.bz2
Depends on: 843934
Depends on: 844334
Depends on: 844335
Depends on: 844355
Depends on: 844360
Depends on: 844365
Depends on: 844372
Depends on: 844476
Depends on: 844477
Depends on: 844480
Depends on: 844483
Depends on: 844633
Depends on: 844645
Depends on: 844665
Depends on: 844836
Depends on: 845046
(Assignee)

Comment 17

5 years ago
Created attachment 718174 [details]
fuzzing material v.2

We just made a fix in the asm.js spec (http://hg.mozilla.org/users/lwagner_mozilla.com/odinmonkey/rev/f61b4dacc3d1) and type system that will reject one of the benchmarks in the previous zip (bullet).  Here is the new version of all the benchmark files.  Let the assault continue.  (Thanks!)
(Assignee)

Updated

5 years ago
Attachment #716296 - Attachment is obsolete: true
Is there a reason not to use the full testsuite that Alon suggested?
I'll prepare another version of that updated for the new spec.
Here it is: http://dl.dropbox.com/u/80664946/emscripten_test_suite2.zip
(In reply to Alon Zakai (:azakai) from comment #20)
> Here it is: http://dl.dropbox.com/u/80664946/emscripten_test_suite2.zip

Thanks Alon :)
Depends on: 846304
Depends on: 846317
Depends on: 847387
(Assignee)

Updated

5 years ago
No longer depends on: 847387
I've been running the csmith fuzzer, which generates C code, then compiling that to asm.js. So far I've not found a single bug in OdinMonkey, but I've found several in emscripten and LLVM.

decoder, if it would be useful, I can provide a bunch (let me know how many) of randomly-generated asm.js code from csmith for JS-level fuzzing.
Created attachment 720856 [details]
csmith fuzzing material

Here are ~100 csmith reduced testcases. I turned on full debug logging in them so they do more ffi calls.
(Assignee)

Comment 24

4 years ago
Christian: Linux32 and OSX32 should work now, please feel free to fuzz them.  They use a different protection mechanism (segmentation) and the IonMonkey load/store codegen is different, so it'd be valuable to give one of these a go.

Also: should I take the pause in fuzz bugs to indicate you've aimed your barrels elsewhere for the time being or that Odin is surviving?
(In reply to Luke Wagner [:luke] from comment #24)
> Christian: Linux32 and OSX32 should work now, please feel free to fuzz them.
> They use a different protection mechanism (segmentation) and the IonMonkey
> load/store codegen is different, so it'd be valuable to give one of these a
> go.

Alright, I'll give it a try on Monday for Linux.

> 
> Also: should I take the pause in fuzz bugs to indicate you've aimed your
> barrels elsewhere for the time being or that Odin is surviving?

Both. I have one more issue with Odin I think, but I currently have to take two days of PTO per week to get all my vacation from last year spent :) So I'm not working as intensively as I am normally.
Depends on: 854448
Depends on: 855442
Depends on: 855526

Updated

4 years ago
Depends on: 878417

Updated

4 years ago
Alias: odinfuzz

Updated

4 years ago
Depends on: 878429

Updated

4 years ago
Depends on: 878433

Updated

4 years ago
Depends on: 878435

Updated

4 years ago
Depends on: 878444

Updated

4 years ago
Depends on: 878495

Updated

4 years ago
Depends on: 878505

Updated

4 years ago
Depends on: 878520
Depends on: 879132
(Assignee)

Updated

4 years ago
No longer depends on: 878429
(Assignee)

Updated

4 years ago
No longer depends on: 878444

Updated

4 years ago
Depends on: 881608

Updated

4 years ago
Depends on: 882008

Updated

4 years ago
Depends on: 882012

Updated

4 years ago
Depends on: 883490

Updated

4 years ago
Depends on: 886246

Updated

4 years ago
Depends on: 886248

Updated

4 years ago
Depends on: 886285

Updated

4 years ago
Depends on: 888568

Updated

4 years ago
Depends on: 893368
Depends on: 908608
Depends on: 908813
Depends on: 911368
Depends on: 913867
Depends on: 913876
Depends on: 913977
Depends on: 927389
(Assignee)

Updated

4 years ago
No longer depends on: 927389
Depends on: 944278
Depends on: 951527
(Assignee)

Updated

4 years ago
No longer depends on: 951527

Updated

4 years ago
Depends on: 970643
Depends on: 975182
Depends on: 975335
Depends on: 978714
Depends on: 986313
Depends on: 1007512
(Assignee)

Updated

3 years ago
No longer depends on: 1005590
Depends on: 1008818
You need to log in before you can comment on or make changes to this bug.