880727 - Update the test runner

Status: RESOLVED FIXED

(L20n :: JS Library, defect, P1)

Description

[Staś Małolepszy :stas]

Our test suite got very outdated.  Let's fix this so that writing tests is easy and rewarding.

Comment 1

[Staś Małolepszy :stas]

Attachment: The test runner works now (55 tests for starters)

I cleaned up a lot of things and made others easier.  A few random (well, not so random) notes:

- I removed the tests/fixtures submodule;  I haven't removed the lol-fixtures repo yet, but I want to eventually, when I moved all files from it;  this might create one-off problems when merging

- I propose the following dir layout:

  - tests/
    - lib - unit tests for lib/;  if there are many (like compiler or parser) put them in a folder
      - client - tests requiring a browser to be run
    - integration - integration tests for the whole library
    - bindings - unit tests for bindings

- unit tests should prefer L20n code inlined in the test file (thanks to this the testcases double as documentation);  integration tests can use fixtures

- run make test to run all tests

- run make watch to run tests when a file changes (+ growl notifications!)

- run make coverage and open dist/docs/coverage.html to see how well the existing test cases cover the code (not representative, but helpful)

- run make docs to extract all comments and create nice-looking documentation in dist/docs

- I moved docs to dist/docs so that we don't have to worry about keeping them up-to-date in the repo.  I still want to publish them at gh-pages, but I haven't worked on that part yet

- you might need to run npm install to get mocha, docco and jscoverage

- I removed old context tests, they were too outdated;  it would be good to reuse the fixtures from the scenarios that we had for them

- I left old compiler test cases;  the changes to the Makefile make sure they're not run;  I ported errors.js to the new test runner and I will continue porting other test cases as well

- I didn't touch tests/bindings/html which use Jasmine;  we should figure out the story for client-side testing later, in another bug.

- the patch also makes a small change to Context::freeze to fix an issue which I discovered when writing the sample tests

- the patch has 55 passing tests right now;  a modest start, but still :)

Comment 2

[Staś Małolepszy :stas]

Even if the porting to the new test runner is not complete or finalized, I'd like to land this patch asap so that we can start writing tests to 1.0 bugs next week.

Comment 3

[Zibi Braniecki [:zbraniecki][:gandalf]]

When I applied the patch and run "make test" I got an error: http://pastebin.mozilla.org/2497348

Comment 4

[Staś Małolepszy :stas]

Attachment: Patch with -M (to make renames work)

(In reply to Staś Małolepszy :stas from comment #1)
> - the patch also makes a small change to Context::freeze to fix an issue
> which I discovered when writing the sample tests

I removed this from the patch and filed bug 880948.

(In reply to Zbigniew Braniecki [:gandalf] from comment #3)
> When I applied the patch and run "make test" I got an error:
> http://pastebin.mozilla.org/2497348

Hmm, interesting.  I can reproduce this on a newly cloned repo.  I think there's a problem in how git marks renames.  Try this new patch, created with 'git diff -M'.  After you apply it, the following two directories should exist:

  tests/integration
  tests/lib/compiler

If you're still having problems, try checking out my branch at

  https://github.com/stasm/l20n.js/tree/880727-test-runner

Comment 5

[Staś Małolepszy :stas]

https://github.com/l20n/l20n.js/commit/d9856360e4816dcc450eed13a393d2b698593477

Woo hoo!