1182298 - [meta] Start running OrrangeHunterr

Status: NEW

(Testing :: General, defect)

Description

[Benoit Girard (:BenWa)]

This bug will track running RROrangeHunter (https://github.com/bgirard/RROrangeHunter). The goal of this project is to generate RR replays to debug intermittent test failures

Comment 1

[Martijn Wargers (dead)]

RROrangeHunter is linux only?

Comment 2

[(not currently active) Ted Mielczarek]

Yes, rr only works on Linux.

jmaher: BenWa is doing this rr work on DigitalOcean VMs. He and I sat down in Whistler and hacked up the basics of "fetch a build + test package, run tests from test package under rr" in a little shell script in the repo in comment 0.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

oh cool- my time invested into reusing my existing hardware didn't go anywhere.

Is the DigitalOcean VMs similar to our environment that we test upon?

Some questions surrounding the expectations of this:
* Are we looking for intermittent oranges to fix, or other problems that might show up?
* How long would we run rr on a single build?  1 day, 7 days, ?
* Assuming we find a rr session to reproduce an issue, do we expect somebody to use rr to fix the issue?
** if so, how much time is reasonable to wait given that a lot of code could change.

Automation costs money to run and maintain, so I would like to ensure we understand what we are really getting from this rather than a developer running a script locally overnight and catching an error in rr for personal hacking.

Comment 4

[(not currently active) Ted Mielczarek]

(In reply to Joel Maher (:jmaher) from comment #3)
> Automation costs money to run and maintain, so I would like to ensure we
> understand what we are really getting from this rather than a developer
> running a script locally overnight and catching an error in rr for personal
> hacking.

Yeah, we discussed some of this in person and in an email thread. We want to keep the automation investment minimal here--this is intended to be a prototype to prove that we can find and fix real bugs in our automated tests with rr.

If we can fix a few actual bugs with the prototype then I think we can easily justify spending the time to write automation around it.

Comment 5

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

I'm renaming this to be clever ;) change it back if you disagree.

Comment 6

[(not currently active) Ted Mielczarek]

Related: https://mail.mozilla.org/pipermail/rr-dev/2016-January/000320.html

Comment 7

[Benoit Girard (:BenWa)]

I've been looking for oranges a bit with this setup. It allowed me to diagnose bug 1226748 however now I keep reproducing it and it's preventing me from finding a replay for another bug. Tracking as a dependency.

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #6)
> Related: https://mail.mozilla.org/pipermail/rr-dev/2016-January/000320.html

I'm already running a branch of RR and I'd running not run with the merge of two feature branches. I'll turn this once it's in master.

Comment 8

[Benoit Girard (:BenWa)]

A bit of progress has been made here. Targeting RR+Chaos at high value intermittent test has been a lot more fruitful then investigating anything that fails. There's a whitelist on github that I'm currently fuzzing. I'll accept any working PR if people have tests they want fuzzed:

https://github.com/bgirard/RROrangeHunter/blob/master/tests.sh