468331 - (mmfuzz) mozmill script fuzzer

Status: RESOLVED WONTFIX

(Testing Graveyard :: Mozmill, defect)

Description

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Attachment: mozmillfuzz 0.1 - proof of concept

I gained the inspiration of this while walking along the streets of Mongkok, Hong Kong. :)

mozmillfuzz generates mozmill scripts that output different content randomly. It is a proof of concept and not meant to be comprehensive in any way.

To use it, run |python mozmillfuzz.py| and stdout output will be a randomly generated .js file that can be used in mozmill automated testing.

The idea is that such random behaviours can be used to generate scripts for Firefox / Thunderbird user actions, be able to detect motortrend.com crashes (see bug 435047 ) automatically and also ultimately to banish manual Top 100 site testing (except when involving visual inspection).

The proof of concept code also accepts URLs with unicode characters, as shown in the sample generated script that will be attached later. It isn't clean, as I'm not a mozmill script expert, so probably assert functions can be used to detect if the browser exits cleanly or certain things render correctly or stuff like that. Randomly opening and closing preferences / add-on windows in a random order while viewing random websites is also possible, just like random actions in Thunderbird wrt. opening the Write or Address Book pane.

Hopefully this can be run by (something)_timed_run.py for automatic reopening Firefox / Thunderbird after they have been closed by the script, and also to generate Lithium-compatible output for automated reduced test cases.

I'd love to hear thoughts by the mozmill folks on whether such a mozmill script generator plays a part in automated testing.

(Security-fied because fuzzers are notorious for generating exploitable code)

Comment 1

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Attachment: sample randomly generated .js mozmill script

(so far only Python 2.5.x is required)

Comment 2

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Attachment: another sample randomly generated .js mozmill script

Comment 3

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Just randomly setting as assignee.

Comment 4

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

(In reply to comment #3)
> Just randomly setting as assignee.

The fuzzer is planned to be created as part of my final-year school project - pre-alpha code is in a hg repository till it's ready. :)

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Sounds great. Can you point us to the repository?

Comment 6

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

(In reply to comment #5)
> Sounds great. Can you point us to the repository?

It's for the moment totally non-usable, so until it can be sensibly run at some point, it'll be on a private repository in Bitbucket.

However, once I get the legal stuff (shouldn't be much) sorted out, it should hopefully eventually live in the private fuzzing repository that all the other Mozilla fuzzers (jsfunfuzz, etc.) live in.

Comment 7

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Gary, when used with UI, this sort of thing is sometimes called a test monkey or a monkey tester (depending on who you ask).

http://en.wikipedia.org/wiki/Monkey_test

I've been talking here about doing similar things, so this sounds awesome. My thought was more towards something we could just leave constantly running across a semi-random state machine, but that requires some pretty intelligent logging so that found problems are even possibly reproduceable later.

I actually kind of like your route of generating scripts and then chaining into mozmill.  It has the advantage that on failure, you can just save off the script that's currently running.

One thing to think about is dumb monkey vs. brilliant monkey--iow, doing fully random actions vs. doing weighted chaining so that you get a "semi-intelligent" exporatory test.  IOW, if it creates a bookmark, maybe it's more likely to use that bookmark within the next N tests.

At any rate, run with it! I'll be taking a look at this with interest.

BTW, I'd be a little surprised if you found security bugs this way (more likely to find memory leaks and other performance issues) so keeping this confidential may be overkill.  I'll let someone else make that call for sure though.

Comment 8

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

After a f2f discussion and demonstration on how this works, I'm giving access to Henrik to the code repository.

mmfuzz is likely to be tri-licensed.

Let me know if anyone else requires access.

Comment 9

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

mmfuzz has now been transferred from academia into open source dominion, and is now present in the private fuzzing repository. Further development will likely occur there.

:)

Comment 10

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Gary, I'd really like to see it. How can I make that happen?

Comment 11

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

(In reply to comment #10)
> Gary, I'd really like to see it. How can I make that happen?

Filed bug 662502.

Comment 12

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

:gkw, :jesse:

Found in triage: what are the next steps here?

Comment 13

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

(In reply to John O'Duinn [:joduinn] from comment #12)
> :gkw, :jesse:
> 
> Found in triage: what are the next steps here?

Mozmill is not supported on Mobile, so it's been placed on hold.

I'm thinking of working on one for Mobile, same UI fuzzing concept, but based on Robotium to be able to work on NativeUI Fennec. I may file a new bug for that though.

Comment 14

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

No progress, deassigning, won't have time to work on this anytime soon.

Comment 15

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Opening up.

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Mozmill will reach its end of life soon. We are currently working on getting all the tests for Firefox ported to Marionette. For status updates please see bug 1080766.

Gary, if you think that this is something we still need, maybe file a new bug for Marionette.