745237 - Remove the jit-test test suite

Status: NEW

(Core :: JavaScript Engine, defect)

Description

[Terrence Cole [:terrence]]

There are some good reasons for doing this:

1) Reducing the initial confused shock to new contributors of discovering there are two test suites and the time it takes to figure out which to run and when.

2) Reducing build times on TBPL.  The jit-tests run at build time through `make check`. Builds are run serially on the build machines, then the test suites are distributed to test machines to run in parallel.  Moving the jit-tests into jstests will make builds report completion faster and allow the jit-tests to run in parallel with other test suites.

3) Running the jit tests in a browser build, rather than in the shell.

4) Running the full test suite with various jit-flags should provide a solid improvement to our automatic test coverage.

Comment 1

[Luke Wagner [:luke]]

Nice.  You probably already have thought of this but one nice thing about jit-tests.py that I hope makes its way into jstests.py is the ability to list a comma-separate list of jit flags to --jitflags=.

Comment 2

[Christian Holler (:decoder)]

I think if this merge involves that automation does no longer run certain tests in the JS shell, then we should certainly not do it. Many crash tests that we have only reproduce in the shell seamlessly and without larger harnesses.

I do see the problem though that jit-tests are a) serial and b) part of the build.

I would rather suggest that we solve these problems independently by making jit-tests their own test on TBPL rather than being part of the build, and by making their execution parallel. We could also rename them to "JS shell tests" since the name "jit-tests" is no longer accurate.

One more reason to keep these kind of simple shell tests is that it's possible to generate many of them automatically from bugs that don't have tests checked in (which is what I'm working on now). This isn't possible for jstests.

Comment 3

[Jeff Walden [:Waldo]]

It's totally possible to generate jstests automatically from bugs that don't have tests checked in -- just put reportCompare(true, true) at the end or you'll have a bad time.

The only things you can't do in jstests, roughly, are add tests that are expected to fail in a particular way, and tests that use shell intrinsics not in the browser.  But generally there's no reason at all to say jstests can't be generated automatically.

Comment 4

[Terrence Cole [:terrence]]

jit-tests vs jstests is simply the wrong way to think about the problem. The right way is:

browser-only tests -- run via the browser on inbound
shell-only tests -- run via the shell on inbound
env agnostic tests -- run in both the shell and browser on inbound

We should start with splitting our tests into these piles, then use whatever tools we have to make that happen. Once we have the right tests running in the right places, then we can worry about what to axe.