501277 - [META] Javascript testsuite automation

Status: RESOLVED INCOMPLETE

(Core :: JavaScript Engine, defect)

Description

[Graydon Hoare :graydon]

This is a metabug to talk about and co-ordinate a variety of test-automation bugs and proposals I want to collect together. Some generally unified plan of attack would be good. Main points I mean to hit:

  - As much automation as possible:
    - 'make check' on developer machines
    - tinderbox integration on correctness tests
    - some kind of useful automated performance counters
      (memory and cycles both)
    - 'make baseline' and 'make perfcheck' on developer machines
    - talos integration on all perf tests
    - 'make fuzz' for developer machines

  - Remove all barriers to developers filing our own tests. Institute a
    one-patch/one-test policy. Currently bc mops up after all of us. This
    is a heavy burden for one person.

  - Integrate as many testsuites as we can get
    - the existing js/test suite
    - Sunspider
    - Dromaeo
    - v8
    - Microsoft ES5 suite
    - Sputnik
    - Numerous other cases I've been collecting:
      - Physics: Box2d, vapour and raytrace demos
      - Languages: cluecc, ocamljs, yhc, hs2js, llvm, cpscm, st2js, etc.
      - Emulators: jsspeccy, jsMSX, 6502asm, js8080 
      - Crypto: RSA, bignum, etc.

  - Make it easy to add performance cases we find floating around the web.

  - Valgrind/memcheck the runs, for UMR and hard-leak prevention. We should
    *never* be letting those through.

  - Unit tests via lirasm for nanojit.

Comment 1

[David Mandelin [:dmandelin]]

That's a great list. I'm wondering if doing this as a wiki might be better than a bug discussion, especially as things move forward. Brief takes on some of the individual topics:

- Automation (of running tests)
  - For minute-to-minute developer usage, what I want is a suite that runs fast (10-30 seconds), is fully automatic, and has incredibly easy-to-read results. E.g., ("PASSED ALL") as the output when it works. This way, I will want to run the suite after every little change.
  - For full tests, I think reading off the results is the worst fault of our existing stuff (q.v. Tinderbox). I think the key things are:
    - a visible progress indicator while it's running
    - if all tests pass, a clear "passed all" output
    - if any tests fail, a simple list of the failing tests with an easy way to run the failing test separately.
    - (I have some ad hoc scripts that do these to some extent for jstests and trace-test.js, which may be useful.)
  - Automating perf test results analysis is hard. For now, a "smoke test" that has a high threshold for complaining is probably the thing to do. Next is coming up with a perf regression suite with very repeatable results; e.g., some relatively long-running microbenchmarks and SS extensions.

- Developers filing tests
  - I would very much like this. Creating a new jstest by copy/pasting an existing one is not that bad, but could be made easier.
  - I have my own regression suite that tests 'js' vs. 'js -j'. Each test is simply a js program. The harness runs it both ways and compares the output. I love this because it's the easiest thing in the world to create a new test.
  - For Dehydra, we went a little further. Each test is simply a JS file, but the first line is a comment with some metadata on expected results and extra test files needed (what C file to run against). The metadata is in a tiny DSL parsed with a few lines of Python. Creating a new test is only slightly harder than creating a new JS file.
  - I like these 'lightweight' test formats because I am too lazy to copy a bunch of boilerplate around, but also because they make it incredibly easy to change how tests are run. The more test-harness stuff that goes into the test case itself, the harder it is to change how tests are run. For example, if the trace-tests.js tests were in separate files, then we could have the harness run them in various orders to increase the chances of finding interaction bugs.

Comment 2

[André Bargull [:anba]]

Closing stale bug as INCOMPLETE.