Bug 482536 (autobisect)

Create script to do automatic hg bisect given a testcase

RESOLVED FIXED

Status

()

Core
JavaScript Engine
--
enhancement
RESOLVED FIXED
8 years ago
2 years ago

People

(Reporter: gkw, Assigned: gkw)

Tracking

Trunk
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(4 obsolete attachments)

(Assignee)

Description

8 years ago
I had a sort-of-maybe working script below to do automatic hg bisections of testcases. jorendorff mentioned he's probably getting something working on his side.

The general idea should also incorporate: 

- copying the js/src/ directory to, say, Desktop/ and running autoconf from there to avoid contamination of original directory (I don't know if these initial steps are really necessary), 
- compiling the js shells themselves in a fresh directory,
- testing them with the testcase,
- labelling them bad/good
- bisecting them and repeating process till the first bad changeset is found. (There's some reversal of bad / good bisection labeling when checking when bugs became WFM vs the bad changeset, elsewhere off the script.)

Ultimately, the script should be able to:

1 - bisect looking for a specific assertion message.
2 - bisect looking for a specific crash (exit return code?).
3 - else return 127 (for say, another assertion that occurred, but which we would do a |hg bisect --skip| to skip it when bisecting manually)

If this works perfectly in js shells, it can possibly be ported over for Fx / TB QA folks to use for faster-regression-window-discovery-times. Also, in case it becomes horribly complex to be written in Bash, feel free to do Python-style or whatever's simpler, the code below is just to show how it would probably work.


Put this in a ./myscript:
===
compileType=$1
branchType=$2
testcaseFile=$3
requiredOutput=$4

./js-$compileType-$branchType-intelmac -j $testcaseFile > tempResult 2>&1

if [ "$?" == 0 ]; then exit 0; fi
if grep -q "$requiredOutput" tempResult; then exit 1; fi
exit 127
===

then run it in the command here:

hg bisect --command './myscript dbg tm guilty.js'

Comment 1

8 years ago
this is specific to the Sisyphus framework which allows specifications of different trees and branches, but you might find some of the issues it deals with helpful. You will have to deal with changed, build commands, broken builds and other problems to keep the bisection working across the range of builds you are bisecting. Note, it also does CVS.

See http://mxr.mozilla.org/mozilla/source/js/tests/bisect.sh

I use it as follows:

 G=9b2a99adc05e && B=e99eeca71505  && t=js1_5/extensions/regress-434837-01.js && tests/mozilla.org/js/bisect.sh -p js -b 1.9.1 -e tracemonkey-test -T debug -G "$G" -B "$B"  -t $t
(Assignee)

Comment 2

8 years ago
Sort of, the idea is indeed similar, though I was thinking to have a simple script that does the compilation then hg bisection (as we look into the future, cvs support isn't really needed anymore) without requiring Sisyphus support.

This will enable developers to bisect really really quickly, automatically, cutting down the times needed for finding regression windows.

Thoughts, anyone?

Comment 3

8 years ago
Understood Sisyphus is a dead end and no one wants to use it but you'll have to deal with all of the issues that the hg portion of the bisect.sh script deals with: changes to the build process over time, broken builds, changes to the source directory by rogue parts of the build process (nsprpub/configure is the main one)...
(Assignee)

Comment 4

8 years ago
(In reply to comment #3)
> Understood Sisyphus is a dead end and no one wants to use it but you'll have to
> deal with all of the issues that the hg portion of the bisect.sh script deals
> with: changes to the build process over time, broken builds, changes to the
> source directory by rogue parts of the build process (nsprpub/configure is the
> main one)...

I don't think the script should be that ambitious for the moment - I would like it to work with TM from changeset 21500 onwards (that's about the time when autoconf support in hg got implemented).

While it's true that the exact command to compile js shells in the future might change (i.e. if pymake does replace make and autoconf), I'm thinking short term here. And oh, if builds are broken in that changeset, just do a |hg bisect --skip|.
(Assignee)

Comment 5

8 years ago
Created attachment 366797 [details]
autoBisect-exitCode.sh

Here's a work-in-progress script that is supposed to work using this command:

hg bisect --command "./autoBisect-exitCode.sh dbg tm guilty.js"

(for debug builds, use "dbg" and opt builds use "opt")

Unfortunately, due to an issue in Mercurial I keep bumping into, it does not work for the moment: http://www.selenic.com/mercurial/bts/issue1512

I may revert to not use |hg bisect -c| in another script, let's see how it goes.
(Assignee)

Comment 6

8 years ago
(In reply to comment #5)
> I may revert to not use |hg bisect -c| in another script, let's see how it
> goes.

ok, after much effort, I have succeeded in coming up with a script (or two) that does auto-bisect (it needs autoconf for js shell though, so that's after changeset 21500) for both new bugs (to find the changeset that caused the bug) and WFM bugs (to find the changeset that fixes the bug), depending on the command line parameters without making use of |hg bisect -c|. Hurray!

It's too long, ugly, unwieldy yet hardcodes some paths and makes some assumptions, and I should be testing it out over the next week or two, all the while tidying code, now I should be sleeping too, so I'll attach another day.

Huge thanks to jorendorff, Wes, jimb, and anyone else I forgot to mention.
(Assignee)

Updated

8 years ago
Alias: autobisect
(Assignee)

Comment 7

8 years ago
Created attachment 367413 [details]
autoBisect 0.1

Release notes:
- Works only with Spidermonkey js shells.
- Automatically bisects for a regression window. Use "bug" to trigger bisecting for a regression window, "wfm" to trigger bisecting for a fix window. (wfm = worksforme)
- Choose to test debug or opt js shells.
- Differentiates between different assertion messages, replacing STRINGTOLOOKOUTFOR as required. Use "" for crashes without assertion messages.
- Does not differentiate between crashes at different bad exit codes.
- "-j" is turned on by default.

Usage instructions:
bash /path/to/autoBisect.sh /path/to/testcase.js [dbg|opt] [bug|wfm] "STRINGTOLOOKOUTFOR"

Known issues:
- So far, works only on Mac Leopard. Linux and Windows (MozillaBuild) are known to break.
- Assumes your TM hg repo directory is at ~/tracemonkey/ and will be the directory where the `hg bisect -b` and `hg bisect -g` commands will be run.
- Works only with "-j" on.
- Assumes earliest working shell is at changeset 21500. That is about the time when TM shifted to autoconf compilation away from Makefiles.

Steps to use:
0 - Delete all existing autoBisect-created directories on the Desktop.
1 - Make sure you're in the TM repo directory, e.g. ~/tracemonkey/
2 - Run `hg bisect -r`.
3 - Give a starting point `hg bisect -g`. (A changeset where it doesn't show the bug for "bug", and where it shows the bug for "wfm")
4 - Give an ending point `hg bisect -b` first. (A changeset where it shows the bug for "bug", and where it doesn't show the bug for "wfm")
5 - Run command, such as a debug wfm assertion (adapting to your needs):

bash ~/Desktop/autoBisect.sh ~/Desktop/476655wfmA.js dbg wfm "StackBase"

or for a debug crash (regression window):

bash ~/Desktop/autoBisect.sh ~/Desktop/482271guilty.js dbg bug ""

6 - Run command repeatedly until regression window is found.
Attachment #366797 - Attachment is obsolete: true
(Assignee)

Updated

8 years ago
Attachment #367413 - Attachment mime type: application/octet-stream → text/plain
(Assignee)

Comment 8

8 years ago
Created attachment 367415 [details]
run.sh - Sample run script

To be used in conjunction with autoBisect.sh - this automates the whole routine of each bisect instance until the smallest regression window is found. Be sure to EDIT run.sh first.

- Will WIPE AWAY ALL of your local changes - beware! (because it runs `hg up -C tip`)
- Assumes autoBisect.sh is on Desktop.
- 4 sets of REPLACEMEs, one for TM repo directory, one for the starting point, one for the ending point, and last one for the command. These should be changed if you want to automate each iteration of autoBisect.sh
- Once run.sh is run, it will do the first bisection, then you will have to key in the number of estimated tests required - hg shows you e.g. ~10 tests or something, just key that number in.
- run.sh will go through an extra iteration just to be sure.

Be warned, if you don't end up with a window, but remain stuck at, say, 8 tests, chances are the assertion morphed. autoBisech.sh will skip this changeset by default. The regression window should be further manually narrowed and clarified to only show the required assertion throughout the regression window, before re-running autoBisect.sh (or run.sh for that matter)
(Assignee)

Comment 9

8 years ago
(In reply to comment #7)
> - Works only with Spidermonkey js shells.

Or Tracemonkey js shells or whatever name it's at - as long as it supports the "-j" option.

> - Assumes your TM hg repo directory is at ~/tracemonkey/ and will be the
> directory where the `hg bisect -b` and `hg bisect -g` commands will be run.

Feel free to edit autoBisect.sh locally to point to your own hg repo location.

> - Works only with "-j" on.

This works for the case of bugs that occur without -j too.
(Assignee)

Comment 10

8 years ago
Another known issue with 0.1:

If a compile at a particular changeset fails with a compile error, the script aborts and autoBisect.sh stops. Ideally, this should result in `hg bisect -s` to skip the changeset and move on.
(Assignee)

Comment 11

8 years ago
Also known:

autoBisect 0.1 works from approximately changeset 25800 (the local number on my machine - which is a range of Feb - March 09) onwards - between 21500 and 25800 autoconf couldn't work from another directory that was not a subdirectory of the source. (i.e. on the same level as the source directory)

I think I have it fixed in an earlier alpha version that eventually broke in 0.1.
(Assignee)

Comment 12

8 years ago
Created attachment 368857 [details]
autoBisect 0.2

Fixes only comment #11 by reverting to the old working behaviour.

I attempted to tackle comment #10 as well, but for some reason if make fails, it didn't generate an exit code I could rely on. I had to comment out this attempt at a fix.
Attachment #367413 - Attachment is obsolete: true
(Assignee)

Updated

8 years ago
Attachment #368857 - Attachment mime type: application/octet-stream → text/plain
(Assignee)

Comment 13

8 years ago
bug 488963 comment 0 is an example of assertion(s) morphing - autoBisect would skip them by doing `hg bisect -s` if it doesn't detect an identical assertion message or exit code for crashes.
(Assignee)

Comment 14

8 years ago
Bug with 0.2:

When there is a debug-only crash at present, the parameters `dbg bug ""` are set, however if in the past there was an assertion with the same testcase, autoBisect 0.2 cannot differentiate between the two.

Some form of check that the exitCode is identical should be added, since a SEGFAULT (SIGSEGV) definitely has a different exit code from an assertion.
(Assignee)

Comment 15

8 years ago
Another 0.2 bug:

autoBisect doesn't work if the "good" revision is an infinite loop, and the "bad" revision is an assertion or a crash.

Probably there should be a time function that cuts off the infinite loop since it is working as expected.
(Assignee)

Comment 16

8 years ago
(In reply to comment #15)
> Another 0.2 bug:
> 
> autoBisect doesn't work if the "good" revision is an infinite loop, and the
> "bad" revision is an assertion or a crash.
> 
> Probably there should be a time function that cuts off the infinite loop since
> it is working as expected.

Partial workaround outside of autoBisect is to add a "break;" function within the infinite loop, but that would be slightly tantamount to modifying the testcase.

Comment 17

8 years ago
Gary, the my patch contains a test case that doesn't iloop. You can probably transform most loops like that.
(Assignee)

Comment 18

8 years ago
Also, see bug 496113 comment 4 for a way to make use of autoBisect on testcases with different output, but which do not crash / assert.
(Assignee)

Comment 19

8 years ago
(In reply to comment #18)
> Also, see bug 496113 comment 4 for a way to make use of autoBisect on testcases
> with different output, but which do not crash / assert.

xref http://mxr.mozilla.org/mozilla-central/source/js/src/shell/js.cpp#103

Snapshot:

103 typedef enum JSShellExitCode {
104     EXITCODE_RUNTIME_ERROR      = 3,
105     EXITCODE_FILE_NOT_FOUND     = 4,
106     EXITCODE_OUT_OF_MEMORY      = 5,
107     EXITCODE_TIMEOUT            = 6
108 } JSShellExitCode;
(Assignee)

Updated

7 years ago
Attachment #367415 - Attachment is obsolete: true
(Assignee)

Updated

7 years ago
Attachment #368857 - Attachment is obsolete: true
(Assignee)

Comment 20

7 years ago
A Python version of autoBisect, autoBisect.py is now in private repositories. It only works with js shells and does away with the run.sh script. :)
Does this now exist?
https://wiki.mozilla.org/Auto-tools/Projects/RegressionHunter

Gerv
(Assignee)

Comment 22

4 years ago
(No, it doesn't exist as RegressionHunter)

Yes, it exists as autoBisect.py but only for the JavaScript spidermonkey shell. I'll be looking to open source it whenever I have the time to do it.
(Assignee)

Comment 23

4 years ago
Taking. This will be fixed once released as OSS.
Assignee: general → gary
Status: NEW → ASSIGNED
Gary, I learnt from Thinker that you're going to release the tool soon, do you have any schedule? Thanks.
Flags: needinfo?(gary)
Gary sent me an email offline, and now the code is right here: https://github.com/MozillaSecurity/funfuzz
Flags: needinfo?(gary)
(Assignee)

Comment 26

2 years ago
FIXED by the open sourcing of code in the funfuzz repository as per comment 25.
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.