476221 - Krakow Labs Browser Fuzzer 2

Status: RESOLVED FIXED

(Core :: Fuzzing, defect)

Description

[Brandon Sterne (:bsterne)]

Attachment: Browser fuzzer Perl script

Krakow Labs released their Browser Fuzzer 2 this month:
http://www.krakowlabs.com/dev.html

I've started running a trunk nightly build against Phase 2 of the fuzzer (DOM):
/bf2.pl -o `pwd` -p2

So far it's gone through ~2800 of the 3365 testcases with no crashes or hangs.  I know chofmann is testing phase 3 (HTML).  We can report any findings here.

Comment 1

[Brandon Sterne (:bsterne)]

The nightly trunk build passed all of the phase 2 testcases with no hangs or crashes.

Comment 2

[chris hofmann]

phase 3 is going to take a lot longer.  I've been running  all afternoon but are only at 4500 out of 60,000 test cases.  More news next week ;-)

Comment 3

[chris hofmann]

er, phase three -> Final Count: 70760 tests

Comment 4

[Jesse Ruderman]

Any assertion failures or memory leaks?

Comment 5

[chris hofmann]

I think we need to figure out a way to parallelize the testing.  I've set up all the testcases at http://people.mozilla.com/~chofmann/security/bf2/

we should recruit a bunch of people to drop in and run tests, and report back which builds were run, which tests were run, if they saw problems, and which test cases where the problems were observed.

Comment 6

[chris hofmann]

looks like you can start the test by dropping into any directory and loading any file.   

eg.  http://people.mozilla.com/~chofmann/security/bf2/phase3/html4000.html

and the testing will pick up from there.

Comment 7

[chris hofmann]

I had a lot of other stuff going on with many windows and tabs opened, but I notice cpu use has gone to 100% and test stop updating when I hit 
http://people.mozilla.com/~chofmann/security/bf2/phase3/html4304.html

another tab with phase1 test running stalled out for a bit around
http://people.mozilla.com/~chofmann/security/bf2/phase1/css162.html
but now it seems like those tests are on the move and running again.

running: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b3pre) Gecko/20090120 Shiretoko/3.1b3pre Ubiquity/0.1.5

Comment 8

[chris hofmann]

also notice on the error console

Error: [Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIWebNavigation.sessionHistory]"  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: chrome://global/content/bindings/browser.xml ::  :: line 641"  data: no]
Source File: chrome://global/content/bindings/browser.xml
Line: 647

but now phase3 test are chugging along again and up to http://people.mozilla.com/~chofmann/security/bf2/phase3/html4338.html

cpu use is still 90%, the browser is running a but slow but is still responding.

Comment 9

[chris hofmann]

made it through all the phase1 css tests (1-348) and cpu use has dropped again.  need to rerun these in a little cleaner environment to see if they were behind the high cpu use, and the expception.

Comment 10

[chris hofmann]

phase3 tests stopped running at
http://people.mozilla.com/~chofmann/security/bf2/phase3/html7919.html
for some reason.  I'll try running overnight again.

Comment 11

[chris hofmann]

ran out of diskspace setting up the phase3 tests.  trying to get them setup correctly

Comment 12

[Aravind Gottipati [:aravind]]

I added more space for now, but there is no more to give on the netapp... we have to come up with a different web space or get bigger disk volumes for stuff like this.

Comment 13

[Johnathan Nightingale [:johnath]]

Would any subset of these (particularly the ones that don't take forever) be appropriate to include as unit tests?

Comment 14

[chris hofmann]

re: comment 13.  haven't seen anything yet worth integrating into regular tests, but there is a lot to look at.

phase3 tests up to html45155.html are available at 
http://people.mozilla.com/~chofmann/security/bf2/phase3/

if interested in helping take 5k tests and report back if you see anything interesting.

starting points are:
http://people.mozilla.com/~chofmann/security/bf2/phase3/10000.html
http://people.mozilla.com/~chofmann/security/bf2/phase3/15000.html
http://people.mozilla.com/~chofmann/security/bf2/phase3/20000.html
http://people.mozilla.com/~chofmann/security/bf2/phase3/25000.html
http://people.mozilla.com/~chofmann/security/bf2/phase3/30000.html
http://people.mozilla.com/~chofmann/security/bf2/phase3/35000.html
http://people.mozilla.com/~chofmann/security/bf2/phase3/40000.html

Comment 15

[chris hofmann]

er,

starting points are:
http://people.mozilla.com/~chofmann/security/bf2/phase3/html10000.html
http://people.mozilla.com/~chofmann/security/bf2/phase3/html15000.html
http://people.mozilla.com/~chofmann/security/bf2/phase3/html20000.html
http://people.mozilla.com/~chofmann/security/bf2/phase3/html25000.html
http://people.mozilla.com/~chofmann/security/bf2/phase3/html30000.html
http://people.mozilla.com/~chofmann/security/bf2/phase3/html35000.html
http://people.mozilla.com/~chofmann/security/bf2/phase3/html40000.html

Comment 16

[chris hofmann]

made to phase3/html9542.html while letting it run overnight. appeared to have stop loading.  memory (919MB) and cpu use was high, but the browser was still responding.  This is turning out to be more stress test than security bug finder.

Comment 17

[Brandon Sterne (:bsterne)]

(In reply to comment #16)
> This is turning out to be more stress test than security bug
> finder.

FWIW, my IE 7 is hanging on the 10th phase 2 testcase so there may be some valid security bugs yielded.

Comment 18

[chris hofmann]

staging for this has moved to http://63.245.208.177/~chofmann/bf2/