630996 - A tool for stress testing objects and PIC's

Status: RESOLVED FIXED

(Core :: JavaScript Engine, defect)

Description

[Jan de Mooij [:jandem]]

Attachment: Script

A few months ago I wrote a simple script for stress-testing operations on objects. It's by no means a fuzzer like the ones from Jesse and decoder. It performs only these steps:

1) generate random objects
2) generate functions (using a pre-defined list of statements)
3) generate calls (pass a random object to a random function)

Since I wrote it in a few days and it's a very simple script (< 400 loc), I will just post it here. Maybe other people are interested in this or know about other operations to add. And I only tested this on my MBP, maybe others can run it on more powerful machines and test on Windows, ARM etc.

Marking security sensitive because there are still (possible) security-sensitive bugs open and I haven't tested this against FF 3.6. After these issues are fixed I will just move it to github or somewhere else.

Comment 1

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

jandem should own his fuzzer bug. :)

Comment 2

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Attachment: script with rudimentary additions to simple objects

Here's a rudimentary hack to the fuzzer. I taught some e4x objects (porting over hackishly from jsfunfuzz) among others to this fuzzer, and it at least found bug 636421 as well.

Comment 3

[Jan de Mooij [:jandem]]

(In reply to comment #2)
> I taught some e4x objects (porting
> over hackishly from jsfunfuzz) among others to this fuzzer,

That's a nice list, thanks again. (we talked on IRC, I have some local changes as well, we may try to create a repository somewhere)

Comment 4

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Attachment: script with comprehensive additions to simple objects

I wrote a objectTester.js script which uses the js shell to test if certain words and their combinations (e.g. "Array" and ".prototype") are objects. After removing duplicates, I added them to test.py.

This is a way more comprehensive list of simple objects, btw.

Comment 5

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Attachment: objectTester.js

This script bridges the gap between jsfunfuzz and this fuzzer. It uses words taught to jsfunfuzz (e.g. e4x and gang), puts them in combination with each other (e.g. Iterator and .prototype) as described in the previous comment, and tests if they are objects, printing them if they are.

To remove dupes, I piped the stdout through `sort` and then `uniq`.

Thanks to jorendorff for helping me out - his advice was invaluable. This has found bug 636818 very quickly.

Comment 6

[Jan de Mooij [:jandem]]

It might be interesting to use the output of this script as input to decoder's adaptive fuzzer. This can be done easily by saving every file instead of files which cause problems.

Comment 7

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

(In reply to comment #6)
> It might be interesting to use the output of this script as input to decoder's
> adaptive fuzzer. This can be done easily by saving every file instead of files
> which cause problems.

I was thinking exactly the same thing for jsfunfuzz just yesterday before I plonked down on bed.

Or we could think about combining all 3 js fuzzers in some way, this one, jsfunfuzz and decoder's one. What language is decoder's fuzzer in?

Comment 8

[Jan de Mooij [:jandem]]

We talked about this on IRC, would love to hear what it finds :)

About the license, MPL is probably easiest if jsfunfuzz uses it too. So I'll stick with MPL for now, but let me know if another license would make things easier.

Comment 9

[Christian Holler (:decoder)]

Both ideas (using jsfunfuzz and jandem's script as LangFuzz input) will surely work and sound good. As soon as you can write the source to a file instead of running it, you can either tell LangFuzz to learn script fragments from it (which will be put into existing regression tests), or we would mutate the tests that those tools generate themselves.

Both should work out of the box, we only have to think about how to get the source into single files for langfuzz.

Note though that E4X is unsupported as the grammar I am using does not support it.

Comment 10

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

(In reply to comment #9)
> Both ideas (using jsfunfuzz and jandem's script as LangFuzz input) will surely
> work and sound good. As soon as you can write the source to a file instead of
> running it, you can either tell LangFuzz to learn script fragments from it
> (which will be put into existing regression tests), or we would mutate the
> tests that those tools generate themselves.
> 
> Both should work out of the box, we only have to think about how to get the
> source into single files for langfuzz.
> 

decoder, I have it working (splicing jandem's generated files into jsfunfuzz scripts prior to running the latter) in a private Mozilla repository, would you like access? (jandem's getting access soon - this will make things much easier)

I'm not sure how langfuzz will fit in but here's how it's done:

jandem's fuzzer generates files which then get appended into jsfunfuzz. The script is then run by a js shell, where jsfunfuzz begins its random generation of fuzz data. If a crash is found by jandem's fuzzer, the edited jsfunfuzz script will display the error before it even generates its random data.

Sidenote: we could even splice in existing regression tests from the TM source code repository, but it's not done yet.

> Note though that E4X is unsupported as the grammar I am using does not support
> it.

Could it be trained for E4X? jandem's fuzzer didn't support E4X too till it got trained.

Comment 11

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

(In reply to comment #8)
> About the license, MPL is probably easiest if jsfunfuzz uses it too. So I'll
> stick with MPL for now, but let me know if another license would make things
> easier.

jsfunfuzz is on tri-license (MPL/ LGPL/ GPL) so I'm guessing that's the one you want for your fuzzer.

ref http://www.mozilla.org/MPL/boilerplate-1.1/mpl-tri-license-c

Comment 12

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Attachment: 4test.py

This was my modification to test.py to generate a script (current.js) at a time (I merely commented out some file generation code near the bottom).

Through startjsfunfuzz.py, jsfunfuzz will then take care of the rest of the file generation. The fuzzing harness will then run it continuously. It has a ton of assumptions, including the fuzzing repo must lie in ~/fuzzing/ and tracemonkey is assumed to lie in ~/tracemonkey/

Go to fuzzing/jsfunfuzz and run `python startjsfunfuzz.py 32 dbg tm" to get a about-to-be-fuzzed 32-bit js debug shell TM build on the Desktop (in its folder) - correct as of today, might overhaul in the future.

jandem, in the fuzzing repo as long as you modify the stuff your fuzzer generates, the rest of us will pick it up just by `hg pull`-ing to tip, assuming you pushed.

Comment 13

[Christian Holler (:decoder)]

(In reply to comment #10)

> decoder, I have it working (splicing jandem's generated files into jsfunfuzz
> scripts prior to running the latter) in a private Mozilla repository, would you
> like access? (jandem's getting access soon - this will make things much easier)

Sure :)

> 
> I'm not sure how langfuzz will fit in but here's how it's done:
> 
> jandem's fuzzer generates files which then get appended into jsfunfuzz. The
> script is then run by a js shell, where jsfunfuzz begins its random generation
> of fuzz data. If a crash is found by jandem's fuzzer, the edited jsfunfuzz
> script will display the error before it even generates its random data.

To integrate LangFuzz here, all I need is a set of files, one "test" per file that can be run. This would suffice for both learning and mutating. For learning only, the file doesn't need to be runnable standalone, I only need to be able to parse it.

> > Note though that E4X is unsupported as the grammar I am using does not support
> > it.
> 
> Could it be trained for E4X? jandem's fuzzer didn't support E4X too till it got
> trained.

I doubt that because it requires to modify my ES3 grammars to generically understand E4X. In order to parse E4X (required to learn/mutate scripts with E4X), you would need to modify my ES3 ANTLR grammar to parse E4X. The fuzzer itself does nothing more than parsing using a grammar/parser class generated from grammar (and that's intended to keep it language independent).

Comment 14

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

(In reply to comment #13)
> To integrate LangFuzz here, all I need is a set of files, one "test" per file
> that can be run. This would suffice for both learning and mutating. For
> learning only, the file doesn't need to be runnable standalone, I only need to
> be able to parse it.

To run 4test.py alone for your purposes, try:

python -u 4test.py ./js .

This will output a current.js file in the current working directory.

python -u <fuzz script> <js shell> <location where current.js gets output>

I don't think the location works other then current working directory, but the original code that jandem wrote to make it this way is still present in 4test.py, albeit commented out.


> I doubt that because it requires to modify my ES3 grammars to generically
> understand E4X. In order to parse E4X (required to learn/mutate scripts with
> E4X), you would need to modify my ES3 ANTLR grammar to parse E4X. The fuzzer
> itself does nothing more than parsing using a grammar/parser class generated
> from grammar (and that's intended to keep it language independent).

Ok, just clarifying.

Comment 15

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

(In reply to Gary Kwong [:gkw, :nth10sd] from comment #5)
> Created attachment 515159 [details]
> objectTester.js

Braindump of command to run objectTester.js :

./js objTester.js 2>&1 | grep -v "strict warning" | grep -v "warning:" | sort | uniq | sort > objectTResults.txt

Comment 16

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

> > Created attachment 515159 [details]
> > objectTester.js

Second braindump for xpcshell:

LD_LIBRARY_PATH=. ./xpcshell objectTester.js 2>&1 | grep -v "strict warning" | grep -v "warning:" | sort | uniq | sort | grep "\"," > resultsObjectTester.txt

Comment 17

[Jan de Mooij [:jandem]]

Closing this bug as it's not really used anymore and shows up in my bz dashboard. jsfunfuzz now does this afaik.