263119 - Add ability to run js/tests from browsers

Status: RESOLVED FIXED

(Core :: JavaScript Engine, defect)

Description

[Bob Clary [:bc] (inactive)]

The current versions of the js tests do not support running multiple tests in
different browsers. The above url contains a working version which 

a) can be run in Mozilla, Internet Explorer, Opera, Konqueror (Safari?)
b) uses the same test selection as in js/tests/menu.html
c) allows selection of doctype for standards mode or quirks mode in the test pages
d) allows selection of the language attribute of the script tag to specify a
language version.

This would be helpful for other browser vendors to help improve their
compatibility with Mozilla, would help authors document the various javascript
engines/versions.

See <http://bclary.com/2004/10/03/js-tests/diff.patch> for a cvs diff against
the trunk and <http://bclary.com/2004/10/03/js-tests/status.log> for a cvs
status (showing new files).

Comment 1

[Bob Clary [:bc] (inactive)]

I just posted the results of testing the modified version of the javascript test
library on bclary.com using the js shell with  Mozilla 1.4-1.8, Firefox 1.0-1.1
and Rhino 1.6R1. Unfortunately, I have been unable to get a liveconnect enabled
js shell built (that will run) so I do not have results for lc2, lc3.

I haven't heard anything back from my post to npm.jseng about this, and would
like to begin the process of checking these changes into the tree. I will delay
the lc2 and lc3 tests until I hear from Kyle when he gets back from vacation,
but believe I can proceed with the ecma* and js1* stuff.

Any objections or comments?

Comment 2

[Bob Clary [:bc] (inactive)]

Most recent test results for the "new" test library:
<http://bclary.com/2004/10/03/results/>

Most recent test results for the "old" test library:
<http://bclary.com/2004/10/03/results-old/>

Comparison of "old" vs. "new" results
<http://bclary.com/2004/10/03/compare-old-new.txt>

Rhino results are identical between "old" and "new", there is minor variability
between the "old" and "new" for spidermonkey however when comparing subsequent
runs of "old" vs "old" <http://bclary.com/2004/10/03/compare-old-old.txt> I see
similar variability.

I can break out the patches into more manageable pieces if people which to
review them, otherwise I will start checking this in with the exception of lc2
and lc3 this evening.

Comment 3

[Brendan Eich [:brendan]]

Bob, is the variability only N vs. N+1 tests failed?  Is the extra failure to-do
with Date?

Please land this stuff, thanks for doing it -- great to have web-ification and
more automation, and tracked results over time.

/be

Comment 4

[Bob Clary [:bc] (inactive)]

(In reply to comment #3)
> Bob, is the variability only N vs. N+1 tests failed?

For the most part.

>  Is the extra failure to-do with Date?

ecma/Date/15.9.2.2-3.js (new: 0, old: -1) failed in one instance in the old
version of the test due a difference in one second where it tests the result of
calling Date() as a function due to (I believe) an unfortunate second boundary
during the test. The original test is supposed to have a tolerance factor but I
don't think it works.

> 

Other variance details:

js1_5/Regress/regress-169559.js - 
Global var access should not be more than twice as slow as local. (new: +3, old: -9)

I've found this performance test on my machine has improved since I have
disabled MS's error talkback reporting. The variance here is quite likely due to
other processes running while the tests are running.

js1_5/Regress/regress-280769-1.js
Do not crash on overflow of 64K boundary of [] offset in regexp search string
(new: +3, old: 0)

js1_5/Regress/regress-280769-5.js -
Do not overflow 64K string offset. (new: +11, old: 0)

These tests do not always crash but appear to crash more consistently with the
new tests.

js1_5/Regress/regress-278873.js -
Don't Crash (new: 0, old: -2)

This test looks for a crash in 

function SwitchTest( input) {
    switch ( input ) {
        default:   break;
        case A:    break;
    }
}

which failed to crash in the new tests for ff 1.0 and 1.0x with a 512K stack but
which did crash fairly consistently in other builds prior to the trunk. I can't
explain the difference but think it is insignificant.

Comment 5

[Bob Clary [:bc] (inactive)]

Attachment: commit.log

Everything except lc2, lc2 checked in. I will leave this open for them. Next
step is to make sure the documentation clearly describes any new requirements
for negative tests.