456935 - Sisyphus - JavaScript Tests - add bisection script

Status: RESOLVED FIXED

(Testing Graveyard :: Sisyphus, enhancement)

Description

[Bob Clary [:bc] (inactive)]

Attachment: bisect.sh patch

finding fixes and regressions on hg is a pita, but hg bisect does make things nice. This is a working script I have started using to test new javascript tests and to find fixes and regressions on mozilla-central and tracemonkey. It doesn't yet support CVS but that should be possible as well.

Comment 1

[Bob Clary [:bc] (inactive)]

Attachment: clobber.sh patch

the js shell clobber does not always work. this forces it to succeed.

Comment 2

[Bob Clary [:bc] (inactive)]

An example session follows:

$ bisect.sh 
usage: bisect.sh -p product -b branch -e extra\
                   -T  buildtype \
                   -t test \
                   -S string \
                   -G good -B bad \
                   [-J javascriptoptions]

    variable            description
    ===============     ============================================================
    -p bisect_product     one of js, firefox
    -b bisect_branches    one of branches 1.8.0, 1.8.1, 1.9.0, 1.9.1
    -e bisect_extra       optional. extra qualifier to pick build tree and mozconfig.
    -T bisect_buildtype   one of build types opt debug
    -t bisect_test        Test to be bisected.
    -S bisect_string      optional. String containing a regular expression which 
                          can be used to distinguish different failures for the given
                          test. The script will search the failure log for the pattern 
                          "$bisect_test.*$string" to determine if a test failure
                          matches the expected failure.
                          bisect_string can contain any extended regular expressions 
                          supported by egrep.
    -G bisect_good        For branches 1.8.0, 1.8.1, 1.9.0, date test passed
                          For branch 1.9.1, revision test passed
    -B bisect_bad         For branches, 1.8.0, 1.8.1, 1.9.0 date test failed
                          For branch 1.9.1, revision test failed. 

       If the good revision (test passed) occurred  prior to the bad revision 
       (test failed), the script will search for the first bad revision which 
       caused the test to regress.

       If the bad revision (test failed) occurred prior to the good revision 
       (test passed), the script will search for the first good revision which 
       fixed the failing test.
       
    -J javascriptoptions  optional. Set JavaScript options:
         -Z n Set gczeal to n. Currently, only valid for 
              debug builds of Gecko 1.8.1.15, 1.9.0 and later.
         -z optional. use split objects in the shell.
         -j optional. use JIT in the shell. Only available on 1.9.1 and later



$ G=a82accd282b4 && B=tip  && t=js1_5/Array/regress-456845.js  && tests/mozilla.org/js/bisect.sh -p js -b 1.9.1 -e test -T debug -G $G -B $B  -t $t
pulling from http://hg.mozilla.org/mozilla-central/
searching for changes
no changes found
pulling from /work/mozilla/builds/hg.mozilla.org/mozilla-central
searching for changes
adding changesets
adding manifests
adding file changes
added 39 changesets with 96 changes to 83 files
(run 'hg update' to get a working copy)

searching for a regression between 19520:a82accd282b4 and 19645:tip
the result is considered good when the test result does not appear in the failure log, bad otherwise.
the bisection is searching for the transition from test failure not found, to test failure found.

checking that the test fails in the bad revision 19645:tip
81 files updated, 0 files merged, 2 files removed, 0 files unresolved
test failure js1_5/Array/regress-456845.js.* found, bad revision 19645:tip confirmed
checking that the test passes in the good revision 19520:a82accd282b4
91 files updated, 0 files merged, 31 files removed, 0 files unresolved
test failure js1_5/Array/regress-456845.js.* not found, good revision 19520:a82accd282b4 confirmed
bisecting /work/mozilla/builds/1.9.1-test/mozilla a82accd282b4 to tip
Testing changeset 19594:c0364f5e0a84 (125 changesets remaining, ~6 tests)
Testing changeset 19594:c0364f5e0a84 (125 changesets remaining, ~6 tests)
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
test failure js1_5/Array/regress-456845.js.* found, marking revision bad
Testing changeset 19563:66a76c8c7346 (62 changesets remaining, ~5 tests)
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
test failure js1_5/Array/regress-456845.js.* not found, marking revision good
Testing changeset 19578:034e02267810 (31 changesets remaining, ~4 tests)
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
test failure js1_5/Array/regress-456845.js.* not found, marking revision good
Testing changeset 19586:f780338381fe (16 changesets remaining, ~4 tests)
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
test failure js1_5/Array/regress-456845.js.* found, marking revision bad
Testing changeset 19582:2b795ca33b0a (8 changesets remaining, ~3 tests)
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
test failure js1_5/Array/regress-456845.js.* found, marking revision bad
Testing changeset 19579:b38dc7abba98 (4 changesets remaining, ~2 tests)
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
test failure js1_5/Array/regress-456845.js.* not found, marking revision good
Testing changeset 19580:4d02662a0cd2 (3 changesets remaining, ~1 tests)
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
test failure js1_5/Array/regress-456845.js.* not found, marking revision good
Testing changeset 19581:77a0720282f7 (2 changesets remaining, ~1 tests)
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
test failure js1_5/Array/regress-456845.js.* not found, marking revision good
regression changeset: 19582:2b795ca33b0a user: Andreas Gal <[e-mail]> date: Thu Sep 18 16:53:09 2008 -0700 summary: Add support for fast native that return jsval and wire up push/pop (453734, r=mrbkap).
*** revision 2b795ca33b0a found ***

Comment 3

[Bob Clary [:bc] (inactive)]

/cvsroot/mozilla/testing/sisyphus/bin/clobber.sh,v  <--  clobber.sh
new revision: 1.2; previous revision: 1.1

/cvsroot/mozilla/js/tests/bisect.sh,v  <--  bisect.sh
initial revision: 1.1

http://hg.mozilla.org/mozilla-central/rev/0108d45cfd8e

Comment 4

[(not currently active) Ted Mielczarek]

Mass move of Sisyphus bugs to Testing : Sisyphus. Filter on SisyphusMassMove to ignore.