406196 - JavaScript Tests - run tinderbox browser tests without restarting browser for each test

Status: RESOLVED FIXED

(Testing Graveyard :: Sisyphus, defect)

Description

[Bob Clary [:bc] (inactive)]

Running the JS tests in the browser currently takes multiple hours mostly due to the time spent starting and stopping the browser for each test. 

crowder suggests that loading the tests in a newly created iframe will prevent modifications to the JS native objects from affecting later tests. 

This will mean that the test runs using this approach will need to exclude crashers from the tests being executed. I think this should be in addition to and not a replacement to the current approach of running each test in a fresh invocation of the browser.

Comment 1

[Bob Clary [:bc] (inactive)]

Attachment: patch

This patch adds the ability for the browser tests to be run without restarting the browser for each test. In order to do so, it must know which tests would crash or timeout and exclude them or otherwise the test run would fail as soon as one of the offending tests was run. 

The known failure file is used to determine which tests crash or time out for the branch and build type. By default, the browser tests are run in non-restart mode and tests which crash or timeout are excluded.

New options have been added to runtests.sh to support the new run modes:

-R              run in restart mode. useful for testing crashes and/or timeouts

-X excludelist  excludelist can be one or more individual tests rooted in the
                test directory, e.g. js1_5/Regress/regress-123456.js, or or one
                or more files containing lists of tests to be excluded, or
                one or more directories to be excluded.
                Note the file is assumed to live in the js tests directory. 
                If more than one item is listed, they must be quoted.

                If not specified, the branch specific 
                spidermonkey-n-$branch.tests and performance-$branch.tests will
                be excluded. To specify no excluded tests, use /dev/null 

-I includelist  can be one or more individual tests rooted in the
                test directory, e.g. js1_5/Regress/regress-123456.js, or or one
                or more files containing lists of tests to be included, or
                one or more directories to be included.
                Note the file is assumed to live in the js tests directory. 
                If more than one item is listed, they must be quoted.

                If not specified, the branch specific set of directories
                are included.
                
-c              by default, tests which crash for this branch, buildtype, 
                operating system, etc. are excluded. To include them, 
                specify -c.

-t              by default, tests which timeout for this branch, buildtype,
                operating system, etc. are excluded. To include them,
                specify -t.

The patch also simplifies the format of the test output, test summary files
to help improve consistency between shell and browser test output. 

The known failure processing which detects possible fixes or possible regressions, now takes into account which tests were actually run so that tests
which were excluded do not show up as possible fixes.

More complete documentation is next on my list.

Comment 2

[Rob Campbell [:rc] (:robcee)]

Comment on attachment 301661 [details] [diff] [review]
patch

nitty-bit: inconsistent use of spaces and tabs in list.sh. I think this looks good, though I haven't seen it in action with my own eyes.

Comment 3

[Bob Clary [:bc] (inactive)]

Cool. list.sh was removed, so that should be ok. There was the tail -v problem and the if !make on Mac, but those are fixed locally. I'll check in shortly and you can watch while I keep my eyes closed. ;-)

Comment 4

[Bob Clary [:bc] (inactive)]

Removing list.sh;
/cvsroot/mozilla/js/tests/list.sh,v  <--  list.sh
new revision: delete; previous revision: 1.7
done
RCS file: /cvsroot/mozilla/js/tests/spidermonkey-n-1.8.0.tests,v
done
Checking in spidermonkey-n-1.8.0.tests;
/cvsroot/mozilla/js/tests/spidermonkey-n-1.8.0.tests,v  <--  spidermonkey-n-1.8.0.tests
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/js/tests/spidermonkey-n-1.8.1.tests,v
done
Checking in spidermonkey-n-1.8.1.tests;
/cvsroot/mozilla/js/tests/spidermonkey-n-1.8.1.tests,v  <--  spidermonkey-n-1.8.1.tests
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/js/tests/spidermonkey-n-1.9.0.tests,v
done
Checking in spidermonkey-n-1.9.0.tests;
/cvsroot/mozilla/js/tests/spidermonkey-n-1.9.0.tests,v  <--  spidermonkey-n-1.9.0.tests
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/js/tests/performance-1.8.0.tests,v
done
Checking in performance-1.8.0.tests;
/cvsroot/mozilla/js/tests/performance-1.8.0.tests,v  <--  performance-1.8.0.tests
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/js/tests/performance-1.8.1.tests,v
done
Checking in performance-1.8.1.tests;
/cvsroot/mozilla/js/tests/performance-1.8.1.tests,v  <--  performance-1.8.1.tests
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/js/tests/performance-1.9.0.tests,v
done
Checking in performance-1.9.0.tests;
/cvsroot/mozilla/js/tests/performance-1.9.0.tests,v  <--  performance-1.9.0.tests
initial revision: 1.1
done
Checking in Makefile;
/cvsroot/mozilla/js/tests/Makefile,v  <--  Makefile
new revision: 1.16; previous revision: 1.15
done
Checking in browser.js;
/cvsroot/mozilla/js/tests/browser.js,v  <--  browser.js
new revision: 1.5; previous revision: 1.4
done
Checking in importList.html;
/cvsroot/mozilla/js/tests/importList.html,v  <--  importList.html
new revision: 1.4; previous revision: 1.3
done
Checking in importList.js;
/cvsroot/mozilla/js/tests/importList.js,v  <--  importList.js
new revision: 1.3; previous revision: 1.2
done
Checking in known-failures.pl;
/cvsroot/mozilla/js/tests/known-failures.pl,v  <--  known-failures.pl
new revision: 1.4; previous revision: 1.3
done
Checking in menuhead.html;
/cvsroot/mozilla/js/tests/menuhead.html,v  <--  menuhead.html
new revision: 1.7; previous revision: 1.6
done
Checking in post-process-logs.pl;
/cvsroot/mozilla/js/tests/post-process-logs.pl,v  <--  post-process-logs.pl
new revision: 1.2; previous revision: 1.1
done
Checking in process-logs.sh;
/cvsroot/mozilla/js/tests/process-logs.sh,v  <--  process-logs.sh
new revision: 1.2; previous revision: 1.1
done
Checking in public-failures.txt;
/cvsroot/mozilla/js/tests/public-failures.txt,v  <--  public-failures.txt
new revision: 1.28; previous revision: 1.27
done
Checking in runtests.sh;
/cvsroot/mozilla/js/tests/runtests.sh,v  <--  runtests.sh
new revision: 1.11; previous revision: 1.10
done
Checking in spidermonkey-extensions-n.tests;
/cvsroot/mozilla/js/tests/spidermonkey-extensions-n.tests,v  <--  spidermonkey-extensions-n.tests
new revision: 1.9; previous revision: 1.8
done
Checking in test-browser.sh;
/cvsroot/mozilla/js/tests/test-browser.sh,v  <--  test-browser.sh
new revision: 1.6; previous revision: 1.5
done
Checking in test-shell.sh;
/cvsroot/mozilla/js/tests/test-shell.sh,v  <--  test-shell.sh
new revision: 1.7; previous revision: 1.6
done
Checking in test.sh;
/cvsroot/mozilla/js/tests/test.sh,v  <--  test.sh
new revision: 1.6; previous revision: 1.5
done
Checking in userhookeach.js;
/cvsroot/mozilla/js/tests/userhookeach.js,v  <--  userhookeach.js
new revision: 1.2; previous revision: 1.1
done
Checking in e4x/browser.js;
/cvsroot/mozilla/js/tests/e4x/browser.js,v  <--  browser.js
new revision: 1.14; previous revision: 1.13
done
Checking in e4x/Regress/regress-319872.js;
/cvsroot/mozilla/js/tests/e4x/Regress/regress-319872.js,v  <--  regress-319872.js
new revision: 1.3; previous revision: 1.2
done
Checking in e4x/Regress/regress-350238.js;
/cvsroot/mozilla/js/tests/e4x/Regress/regress-350238.js,v  <--  regress-350238.js
new revision: 1.3; previous revision: 1.2
done
Checking in e4x/XML/regress-324422-2.js;
/cvsroot/mozilla/js/tests/e4x/XML/regress-324422-2.js,v  <--  regress-324422-2.js
new revision: 1.5; previous revision: 1.4
done
Checking in js1_5/Regress/regress-312588.js;
/cvsroot/mozilla/js/tests/js1_5/Regress/regress-312588.js,v  <--  regress-312588.js
new revision: 1.4; previous revision: 1.3
done
Checking in js1_5/Regress/regress-350268.js;
/cvsroot/mozilla/js/tests/js1_5/Regress/regress-350268.js,v  <--  regress-350268.js
new revision: 1.3; previous revision: 1.2
done
Checking in js1_7/iterable/regress-341815.js;
/cvsroot/mozilla/js/tests/js1_7/iterable/regress-341815.js,v  <--  regress-341815.js
new revision: 1.7; previous revision: 1.6
done

Comment 5

[(not currently active) Ted Mielczarek]

bc said to stick the rest of these JS Test bugs in Sisyphus for now.