846225 - Run monkey/orangutan tests in automated infrastructure

Status: RESOLVED INCOMPLETE

(Core :: General, defect)

Description

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Bug 832328 provides an orangutan harness to send random input gestures to content.  This kind of infrastructure is like UI fuzzing, and turns up tons of valuable bugs.  We should run this on automation.

The problem to be solved here is
 - what's a pass/fail?  Maybe pass means "X hours without 'problems'", see below.
 - what's a "problem"?  Crashes are always problems.  Hangs are too.  It's a bit hard to define "hang" from within the orangutan harness, but we should be able to figure something out.

Once we get these issues sorted out, we just need to install this code on infra.

Comment 1

[(not currently active) Ted Mielczarek]

I don't know that pass/fail is useful terminology here. I think you want this to act just like our existing fuzzing setups, where you simply get a report of crashes and other bad things that it turns up.

Comment 2

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Crashes are definitely the first priority here. Next up would be assertions and then hangs. We should clear most of these out first then worry about the rest later (misc. UI issues, or incorrect JS error messages, maybe?)

I know Christoph was trying out marionette stuff too via Peach, so cc'ing some other fuzzer folks.

Comment 3

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

We caught up briefly on IRC, but you can run the monkey tests with a script that's part of the b2g "distribution" checked out

$ ./scripts/run-monkey.sh

Comment 4

[Christoph Diehl [:posidron]]

Q: Is it already possible to get crash dumps on B2G other than monitoring logcat?
If yes, where do I find those?

Comment 5

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Yes, they're saved to /data/b2g/mozilla/Crash Reports/pending.