Last Comment Bug 482536 - (autobisect) Create script to do automatic hg bisect given a testcase
(autobisect)
: Create script to do automatic hg bisect given a testcase
Status: RESOLVED FIXED
:
Product: Core
Classification: Components
Component: JavaScript Engine (show other bugs)
: Trunk
: All All
: -- enhancement (vote)
: ---
Assigned To: Gary Kwong [:gkw] [:nth10sd]
:
Mentors:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2009-03-10 10:40 PDT by Gary Kwong [:gkw] [:nth10sd]
Modified: 2015-07-08 15:10 PDT (History)
19 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
autoBisect-exitCode.sh (1.32 KB, text/plain)
2009-03-11 06:57 PDT, Gary Kwong [:gkw] [:nth10sd]
no flags Details
autoBisect 0.1 (5.64 KB, text/plain)
2009-03-14 11:49 PDT, Gary Kwong [:gkw] [:nth10sd]
no flags Details
run.sh - Sample run script (907 bytes, text/plain)
2009-03-14 11:57 PDT, Gary Kwong [:gkw] [:nth10sd]
no flags Details
autoBisect 0.2 (6.82 KB, text/plain)
2009-03-23 01:22 PDT, Gary Kwong [:gkw] [:nth10sd]
no flags Details

Description Gary Kwong [:gkw] [:nth10sd] 2009-03-10 10:40:34 PDT
I had a sort-of-maybe working script below to do automatic hg bisections of testcases. jorendorff mentioned he's probably getting something working on his side.

The general idea should also incorporate: 

- copying the js/src/ directory to, say, Desktop/ and running autoconf from there to avoid contamination of original directory (I don't know if these initial steps are really necessary), 
- compiling the js shells themselves in a fresh directory,
- testing them with the testcase,
- labelling them bad/good
- bisecting them and repeating process till the first bad changeset is found. (There's some reversal of bad / good bisection labeling when checking when bugs became WFM vs the bad changeset, elsewhere off the script.)

Ultimately, the script should be able to:

1 - bisect looking for a specific assertion message.
2 - bisect looking for a specific crash (exit return code?).
3 - else return 127 (for say, another assertion that occurred, but which we would do a |hg bisect --skip| to skip it when bisecting manually)

If this works perfectly in js shells, it can possibly be ported over for Fx / TB QA folks to use for faster-regression-window-discovery-times. Also, in case it becomes horribly complex to be written in Bash, feel free to do Python-style or whatever's simpler, the code below is just to show how it would probably work.


Put this in a ./myscript:
===
compileType=$1
branchType=$2
testcaseFile=$3
requiredOutput=$4

./js-$compileType-$branchType-intelmac -j $testcaseFile > tempResult 2>&1

if [ "$?" == 0 ]; then exit 0; fi
if grep -q "$requiredOutput" tempResult; then exit 1; fi
exit 127
===

then run it in the command here:

hg bisect --command './myscript dbg tm guilty.js'
Comment 1 Bob Clary [:bc:] 2009-03-10 11:16:55 PDT
this is specific to the Sisyphus framework which allows specifications of different trees and branches, but you might find some of the issues it deals with helpful. You will have to deal with changed, build commands, broken builds and other problems to keep the bisection working across the range of builds you are bisecting. Note, it also does CVS.

See http://mxr.mozilla.org/mozilla/source/js/tests/bisect.sh

I use it as follows:

 G=9b2a99adc05e && B=e99eeca71505  && t=js1_5/extensions/regress-434837-01.js && tests/mozilla.org/js/bisect.sh -p js -b 1.9.1 -e tracemonkey-test -T debug -G "$G" -B "$B"  -t $t
Comment 2 Gary Kwong [:gkw] [:nth10sd] 2009-03-11 01:53:43 PDT
Sort of, the idea is indeed similar, though I was thinking to have a simple script that does the compilation then hg bisection (as we look into the future, cvs support isn't really needed anymore) without requiring Sisyphus support.

This will enable developers to bisect really really quickly, automatically, cutting down the times needed for finding regression windows.

Thoughts, anyone?
Comment 3 Bob Clary [:bc:] 2009-03-11 02:58:18 PDT
Understood Sisyphus is a dead end and no one wants to use it but you'll have to deal with all of the issues that the hg portion of the bisect.sh script deals with: changes to the build process over time, broken builds, changes to the source directory by rogue parts of the build process (nsprpub/configure is the main one)...
Comment 4 Gary Kwong [:gkw] [:nth10sd] 2009-03-11 03:05:29 PDT
(In reply to comment #3)
> Understood Sisyphus is a dead end and no one wants to use it but you'll have to
> deal with all of the issues that the hg portion of the bisect.sh script deals
> with: changes to the build process over time, broken builds, changes to the
> source directory by rogue parts of the build process (nsprpub/configure is the
> main one)...

I don't think the script should be that ambitious for the moment - I would like it to work with TM from changeset 21500 onwards (that's about the time when autoconf support in hg got implemented).

While it's true that the exact command to compile js shells in the future might change (i.e. if pymake does replace make and autoconf), I'm thinking short term here. And oh, if builds are broken in that changeset, just do a |hg bisect --skip|.
Comment 5 Gary Kwong [:gkw] [:nth10sd] 2009-03-11 06:57:37 PDT
Created attachment 366797 [details]
autoBisect-exitCode.sh

Here's a work-in-progress script that is supposed to work using this command:

hg bisect --command "./autoBisect-exitCode.sh dbg tm guilty.js"

(for debug builds, use "dbg" and opt builds use "opt")

Unfortunately, due to an issue in Mercurial I keep bumping into, it does not work for the moment: http://www.selenic.com/mercurial/bts/issue1512

I may revert to not use |hg bisect -c| in another script, let's see how it goes.
Comment 6 Gary Kwong [:gkw] [:nth10sd] 2009-03-11 12:06:19 PDT
(In reply to comment #5)
> I may revert to not use |hg bisect -c| in another script, let's see how it
> goes.

ok, after much effort, I have succeeded in coming up with a script (or two) that does auto-bisect (it needs autoconf for js shell though, so that's after changeset 21500) for both new bugs (to find the changeset that caused the bug) and WFM bugs (to find the changeset that fixes the bug), depending on the command line parameters without making use of |hg bisect -c|. Hurray!

It's too long, ugly, unwieldy yet hardcodes some paths and makes some assumptions, and I should be testing it out over the next week or two, all the while tidying code, now I should be sleeping too, so I'll attach another day.

Huge thanks to jorendorff, Wes, jimb, and anyone else I forgot to mention.
Comment 7 Gary Kwong [:gkw] [:nth10sd] 2009-03-14 11:49:32 PDT
Created attachment 367413 [details]
autoBisect 0.1

Release notes:
- Works only with Spidermonkey js shells.
- Automatically bisects for a regression window. Use "bug" to trigger bisecting for a regression window, "wfm" to trigger bisecting for a fix window. (wfm = worksforme)
- Choose to test debug or opt js shells.
- Differentiates between different assertion messages, replacing STRINGTOLOOKOUTFOR as required. Use "" for crashes without assertion messages.
- Does not differentiate between crashes at different bad exit codes.
- "-j" is turned on by default.

Usage instructions:
bash /path/to/autoBisect.sh /path/to/testcase.js [dbg|opt] [bug|wfm] "STRINGTOLOOKOUTFOR"

Known issues:
- So far, works only on Mac Leopard. Linux and Windows (MozillaBuild) are known to break.
- Assumes your TM hg repo directory is at ~/tracemonkey/ and will be the directory where the `hg bisect -b` and `hg bisect -g` commands will be run.
- Works only with "-j" on.
- Assumes earliest working shell is at changeset 21500. That is about the time when TM shifted to autoconf compilation away from Makefiles.

Steps to use:
0 - Delete all existing autoBisect-created directories on the Desktop.
1 - Make sure you're in the TM repo directory, e.g. ~/tracemonkey/
2 - Run `hg bisect -r`.
3 - Give a starting point `hg bisect -g`. (A changeset where it doesn't show the bug for "bug", and where it shows the bug for "wfm")
4 - Give an ending point `hg bisect -b` first. (A changeset where it shows the bug for "bug", and where it doesn't show the bug for "wfm")
5 - Run command, such as a debug wfm assertion (adapting to your needs):

bash ~/Desktop/autoBisect.sh ~/Desktop/476655wfmA.js dbg wfm "StackBase"

or for a debug crash (regression window):

bash ~/Desktop/autoBisect.sh ~/Desktop/482271guilty.js dbg bug ""

6 - Run command repeatedly until regression window is found.
Comment 8 Gary Kwong [:gkw] [:nth10sd] 2009-03-14 11:57:58 PDT
Created attachment 367415 [details]
run.sh - Sample run script

To be used in conjunction with autoBisect.sh - this automates the whole routine of each bisect instance until the smallest regression window is found. Be sure to EDIT run.sh first.

- Will WIPE AWAY ALL of your local changes - beware! (because it runs `hg up -C tip`)
- Assumes autoBisect.sh is on Desktop.
- 4 sets of REPLACEMEs, one for TM repo directory, one for the starting point, one for the ending point, and last one for the command. These should be changed if you want to automate each iteration of autoBisect.sh
- Once run.sh is run, it will do the first bisection, then you will have to key in the number of estimated tests required - hg shows you e.g. ~10 tests or something, just key that number in.
- run.sh will go through an extra iteration just to be sure.

Be warned, if you don't end up with a window, but remain stuck at, say, 8 tests, chances are the assertion morphed. autoBisech.sh will skip this changeset by default. The regression window should be further manually narrowed and clarified to only show the required assertion throughout the regression window, before re-running autoBisect.sh (or run.sh for that matter)
Comment 9 Gary Kwong [:gkw] [:nth10sd] 2009-03-14 12:01:11 PDT
(In reply to comment #7)
> - Works only with Spidermonkey js shells.

Or Tracemonkey js shells or whatever name it's at - as long as it supports the "-j" option.

> - Assumes your TM hg repo directory is at ~/tracemonkey/ and will be the
> directory where the `hg bisect -b` and `hg bisect -g` commands will be run.

Feel free to edit autoBisect.sh locally to point to your own hg repo location.

> - Works only with "-j" on.

This works for the case of bugs that occur without -j too.
Comment 10 Gary Kwong [:gkw] [:nth10sd] 2009-03-15 02:50:20 PDT
Another known issue with 0.1:

If a compile at a particular changeset fails with a compile error, the script aborts and autoBisect.sh stops. Ideally, this should result in `hg bisect -s` to skip the changeset and move on.
Comment 11 Gary Kwong [:gkw] [:nth10sd] 2009-03-19 02:57:30 PDT
Also known:

autoBisect 0.1 works from approximately changeset 25800 (the local number on my machine - which is a range of Feb - March 09) onwards - between 21500 and 25800 autoconf couldn't work from another directory that was not a subdirectory of the source. (i.e. on the same level as the source directory)

I think I have it fixed in an earlier alpha version that eventually broke in 0.1.
Comment 12 Gary Kwong [:gkw] [:nth10sd] 2009-03-23 01:22:40 PDT
Created attachment 368857 [details]
autoBisect 0.2

Fixes only comment #11 by reverting to the old working behaviour.

I attempted to tackle comment #10 as well, but for some reason if make fails, it didn't generate an exit code I could rely on. I had to comment out this attempt at a fix.
Comment 13 Gary Kwong [:gkw] [:nth10sd] 2009-04-22 18:15:36 PDT
bug 488963 comment 0 is an example of assertion(s) morphing - autoBisect would skip them by doing `hg bisect -s` if it doesn't detect an identical assertion message or exit code for crashes.
Comment 14 Gary Kwong [:gkw] [:nth10sd] 2009-05-06 09:47:30 PDT
Bug with 0.2:

When there is a debug-only crash at present, the parameters `dbg bug ""` are set, however if in the past there was an assertion with the same testcase, autoBisect 0.2 cannot differentiate between the two.

Some form of check that the exitCode is identical should be added, since a SEGFAULT (SIGSEGV) definitely has a different exit code from an assertion.
Comment 15 Gary Kwong [:gkw] [:nth10sd] 2009-06-01 20:39:57 PDT
Another 0.2 bug:

autoBisect doesn't work if the "good" revision is an infinite loop, and the "bad" revision is an assertion or a crash.

Probably there should be a time function that cuts off the infinite loop since it is working as expected.
Comment 16 Gary Kwong [:gkw] [:nth10sd] 2009-06-01 20:56:04 PDT
(In reply to comment #15)
> Another 0.2 bug:
> 
> autoBisect doesn't work if the "good" revision is an infinite loop, and the
> "bad" revision is an assertion or a crash.
> 
> Probably there should be a time function that cuts off the infinite loop since
> it is working as expected.

Partial workaround outside of autoBisect is to add a "break;" function within the infinite loop, but that would be slightly tantamount to modifying the testcase.
Comment 17 Andreas Gal :gal 2009-06-01 21:05:58 PDT
Gary, the my patch contains a test case that doesn't iloop. You can probably transform most loops like that.
Comment 18 Gary Kwong [:gkw] [:nth10sd] 2009-06-03 07:38:57 PDT
Also, see bug 496113 comment 4 for a way to make use of autoBisect on testcases with different output, but which do not crash / assert.
Comment 19 Gary Kwong [:gkw] [:nth10sd] 2009-06-03 07:56:19 PDT
(In reply to comment #18)
> Also, see bug 496113 comment 4 for a way to make use of autoBisect on testcases
> with different output, but which do not crash / assert.

xref http://mxr.mozilla.org/mozilla-central/source/js/src/shell/js.cpp#103

Snapshot:

103 typedef enum JSShellExitCode {
104     EXITCODE_RUNTIME_ERROR      = 3,
105     EXITCODE_FILE_NOT_FOUND     = 4,
106     EXITCODE_OUT_OF_MEMORY      = 5,
107     EXITCODE_TIMEOUT            = 6
108 } JSShellExitCode;
Comment 20 Gary Kwong [:gkw] [:nth10sd] 2010-05-14 12:24:14 PDT
A Python version of autoBisect, autoBisect.py is now in private repositories. It only works with js shells and does away with the run.sh script. :)
Comment 21 Gervase Markham [:gerv] 2013-03-04 09:22:14 PST
Does this now exist?
https://wiki.mozilla.org/Auto-tools/Projects/RegressionHunter

Gerv
Comment 22 Gary Kwong [:gkw] [:nth10sd] 2013-03-04 10:27:05 PST
(No, it doesn't exist as RegressionHunter)

Yes, it exists as autoBisect.py but only for the JavaScript spidermonkey shell. I'll be looking to open source it whenever I have the time to do it.
Comment 23 Gary Kwong [:gkw] [:nth10sd] 2013-03-04 10:28:19 PST
Taking. This will be fixed once released as OSS.
Comment 24 Ting-Yu Chou [:ting] 2015-07-07 01:50:31 PDT
Gary, I learnt from Thinker that you're going to release the tool soon, do you have any schedule? Thanks.
Comment 25 Ting-Yu Chou [:ting] 2015-07-07 19:30:19 PDT
Gary sent me an email offline, and now the code is right here: https://github.com/MozillaSecurity/funfuzz
Comment 26 Gary Kwong [:gkw] [:nth10sd] 2015-07-08 15:10:33 PDT
FIXED by the open sourcing of code in the funfuzz repository as per comment 25.

Note You need to log in before you can comment on or make changes to this bug.