Closed Bug 630996 Opened 13 years ago Closed 9 years ago

A tool for stress testing objects and PIC's

Categories

(Core :: JavaScript Engine, defect)

defect
Not set
normal

Tracking

()

RESOLVED FIXED

People

(Reporter: jandem, Assigned: jandem)

References

Details

(Keywords: meta, sec-other, Whiteboard: [sg:nse meta])

Attachments

(3 files, 2 obsolete files)

Attached file Script —
A few months ago I wrote a simple script for stress-testing operations on objects. It's by no means a fuzzer like the ones from Jesse and decoder. It performs only these steps:

1) generate random objects
2) generate functions (using a pre-defined list of statements)
3) generate calls (pass a random object to a random function)

Since I wrote it in a few days and it's a very simple script (< 400 loc), I will just post it here. Maybe other people are interested in this or know about other operations to add. And I only tested this on my MBP, maybe others can run it on more powerful machines and test on Windows, ARM etc.

Marking security sensitive because there are still (possible) security-sensitive bugs open and I haven't tested this against FF 3.6. After these issues are fixed I will just move it to github or somewhere else.
Depends on: 630865
Depends on: 625718
Depends on: 627984
Depends on: 628334
Depends on: 630366
Keywords: meta
Whiteboard: [sg:nse meta]
Depends on: 631219
Depends on: 632778
Depends on: 632901
Depends on: 632924
Depends on: 633525
Depends on: 633741
Depends on: 633752
Depends on: 634210
Depends on: 635200
Depends on: 635531
jandem should own his fuzzer bug. :)
Assignee: general → jandemooij
Here's a rudimentary hack to the fuzzer. I taught some e4x objects (porting over hackishly from jsfunfuzz) among others to this fuzzer, and it at least found bug 636421 as well.
Attachment #514769 - Attachment mime type: application/octet-stream → text/x-python-script
Depends on: 636487
(In reply to comment #2)
> I taught some e4x objects (porting
> over hackishly from jsfunfuzz) among others to this fuzzer,

That's a nice list, thanks again. (we talked on IRC, I have some local changes as well, we may try to create a repository somewhere)
I wrote a objectTester.js script which uses the js shell to test if certain words and their combinations (e.g. "Array" and ".prototype") are objects. After removing duplicates, I added them to test.py.

This is a way more comprehensive list of simple objects, btw.
Attachment #514769 - Attachment is obsolete: true
Attached file objectTester.js —
This script bridges the gap between jsfunfuzz and this fuzzer. It uses words taught to jsfunfuzz (e.g. e4x and gang), puts them in combination with each other (e.g. Iterator and .prototype) as described in the previous comment, and tests if they are objects, printing them if they are.

To remove dupes, I piped the stdout through `sort` and then `uniq`.

Thanks to jorendorff for helping me out - his advice was invaluable. This has found bug 636818 very quickly.
It might be interesting to use the output of this script as input to decoder's adaptive fuzzer. This can be done easily by saving every file instead of files which cause problems.
(In reply to comment #6)
> It might be interesting to use the output of this script as input to decoder's
> adaptive fuzzer. This can be done easily by saving every file instead of files
> which cause problems.

I was thinking exactly the same thing for jsfunfuzz just yesterday before I plonked down on bed.

Or we could think about combining all 3 js fuzzers in some way, this one, jsfunfuzz and decoder's one. What language is decoder's fuzzer in?
We talked about this on IRC, would love to hear what it finds :)

About the license, MPL is probably easiest if jsfunfuzz uses it too. So I'll stick with MPL for now, but let me know if another license would make things easier.
Both ideas (using jsfunfuzz and jandem's script as LangFuzz input) will surely work and sound good. As soon as you can write the source to a file instead of running it, you can either tell LangFuzz to learn script fragments from it (which will be put into existing regression tests), or we would mutate the tests that those tools generate themselves.

Both should work out of the box, we only have to think about how to get the source into single files for langfuzz.

Note though that E4X is unsupported as the grammar I am using does not support it.
(In reply to comment #9)
> Both ideas (using jsfunfuzz and jandem's script as LangFuzz input) will surely
> work and sound good. As soon as you can write the source to a file instead of
> running it, you can either tell LangFuzz to learn script fragments from it
> (which will be put into existing regression tests), or we would mutate the
> tests that those tools generate themselves.
> 
> Both should work out of the box, we only have to think about how to get the
> source into single files for langfuzz.
> 

decoder, I have it working (splicing jandem's generated files into jsfunfuzz scripts prior to running the latter) in a private Mozilla repository, would you like access? (jandem's getting access soon - this will make things much easier)

I'm not sure how langfuzz will fit in but here's how it's done:

jandem's fuzzer generates files which then get appended into jsfunfuzz. The script is then run by a js shell, where jsfunfuzz begins its random generation of fuzz data. If a crash is found by jandem's fuzzer, the edited jsfunfuzz script will display the error before it even generates its random data.

Sidenote: we could even splice in existing regression tests from the TM source code repository, but it's not done yet.

> Note though that E4X is unsupported as the grammar I am using does not support
> it.

Could it be trained for E4X? jandem's fuzzer didn't support E4X too till it got trained.
(In reply to comment #8)
> About the license, MPL is probably easiest if jsfunfuzz uses it too. So I'll
> stick with MPL for now, but let me know if another license would make things
> easier.

jsfunfuzz is on tri-license (MPL/ LGPL/ GPL) so I'm guessing that's the one you want for your fuzzer.

ref http://www.mozilla.org/MPL/boilerplate-1.1/mpl-tri-license-c
Attached file 4test.py —
This was my modification to test.py to generate a script (current.js) at a time (I merely commented out some file generation code near the bottom).

Through startjsfunfuzz.py, jsfunfuzz will then take care of the rest of the file generation. The fuzzing harness will then run it continuously. It has a ton of assumptions, including the fuzzing repo must lie in ~/fuzzing/ and tracemonkey is assumed to lie in ~/tracemonkey/

Go to fuzzing/jsfunfuzz and run `python startjsfunfuzz.py 32 dbg tm" to get a about-to-be-fuzzed 32-bit js debug shell TM build on the Desktop (in its folder) - correct as of today, might overhaul in the future.

jandem, in the fuzzing repo as long as you modify the stuff your fuzzer generates, the rest of us will pick it up just by `hg pull`-ing to tip, assuming you pushed.
Attachment #515154 - Attachment is obsolete: true
(In reply to comment #10)

> decoder, I have it working (splicing jandem's generated files into jsfunfuzz
> scripts prior to running the latter) in a private Mozilla repository, would you
> like access? (jandem's getting access soon - this will make things much easier)

Sure :)

> 
> I'm not sure how langfuzz will fit in but here's how it's done:
> 
> jandem's fuzzer generates files which then get appended into jsfunfuzz. The
> script is then run by a js shell, where jsfunfuzz begins its random generation
> of fuzz data. If a crash is found by jandem's fuzzer, the edited jsfunfuzz
> script will display the error before it even generates its random data.

To integrate LangFuzz here, all I need is a set of files, one "test" per file that can be run. This would suffice for both learning and mutating. For learning only, the file doesn't need to be runnable standalone, I only need to be able to parse it.

> > Note though that E4X is unsupported as the grammar I am using does not support
> > it.
> 
> Could it be trained for E4X? jandem's fuzzer didn't support E4X too till it got
> trained.

I doubt that because it requires to modify my ES3 grammars to generically understand E4X. In order to parse E4X (required to learn/mutate scripts with E4X), you would need to modify my ES3 ANTLR grammar to parse E4X. The fuzzer itself does nothing more than parsing using a grammar/parser class generated from grammar (and that's intended to keep it language independent).
(In reply to comment #13)
> To integrate LangFuzz here, all I need is a set of files, one "test" per file
> that can be run. This would suffice for both learning and mutating. For
> learning only, the file doesn't need to be runnable standalone, I only need to
> be able to parse it.

To run 4test.py alone for your purposes, try:

python -u 4test.py ./js .

This will output a current.js file in the current working directory.

python -u <fuzz script> <js shell> <location where current.js gets output>

I don't think the location works other then current working directory, but the original code that jandem wrote to make it this way is still present in 4test.py, albeit commented out.


> I doubt that because it requires to modify my ES3 grammars to generically
> understand E4X. In order to parse E4X (required to learn/mutate scripts with
> E4X), you would need to modify my ES3 ANTLR grammar to parse E4X. The fuzzer
> itself does nothing more than parsing using a grammar/parser class generated
> from grammar (and that's intended to keep it language independent).

Ok, just clarifying.
Depends on: 637014
Depends on: 637397
Depends on: 637403
Depends on: 641479
Depends on: 641491
Depends on: 641525
Depends on: 641535
Depends on: 642894
Depends on: 643182
Depends on: 645629
Depends on: 645632
Depends on: 645645
Depends on: 645666
Depends on: 645985
Depends on: 649439
Depends on: 649939
Depends on: 652806
Depends on: 666003
(In reply to Gary Kwong [:gkw, :nth10sd] from comment #5)
> Created attachment 515159 [details]
> objectTester.js

Braindump of command to run objectTester.js :

./js objTester.js 2>&1 | grep -v "strict warning" | grep -v "warning:" | sort | uniq | sort > objectTResults.txt
> > Created attachment 515159 [details]
> > objectTester.js

Second braindump for xpcshell:

LD_LIBRARY_PATH=. ./xpcshell objectTester.js 2>&1 | grep -v "strict warning" | grep -v "warning:" | sort | uniq | sort | grep "\"," > resultsObjectTester.txt
Closing this bug as it's not really used anymore and shows up in my bz dashboard. jsfunfuzz now does this afaik.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Group: core-security → core-security-release
Group: core-security-release
You need to log in before you can comment on or make changes to this bug.