343673 - Unit Testing Framework

Status: RESOLVED INVALID

(Firefox Build System :: General, defect)

Description

[Rob Campbell [:rc] (:robcee)]

MinUnit
CxxTest
CppUnitLite...

Comment 1

[GavinS]

Boost Test? ( http://www.boost.org/libs/test/doc/index.html )

Comment 2

[Rob Campbell [:rc] (:robcee)]

thanks for the link. This might be worth looking at.

Comment 3

[Sergey Yanovich]

As a part of our project we have developed a unit testing framework for mozilla. It is meant to work in eXtreme Programming (XP) environment, allowing safe refactoring.

At the moment only a linux version is available as a xulrunner application:
http://git.aasii.org/?p=abstract/yanovich/abstract.git;a=snapshot;h=54586aac9eeb4494a4056d4dac6c686aab373ba2

Build would require automake-1.10, autoconf-2.61, libtool, xulrunner-1.9a3 (installed eighter at pkgconfig prefix or /usr/local). The framework heavily uses nsDeque, so libxpcomds_s.a needs to be copied to $prefix/lib/xulrunner-1.9a3 directory. This sucks, but works as a temporary hack.

Our testing framework is a set of XPCOM components that can test most of a XUL+XPCOM application:
* framework adds three interfaces, all scriptable:
  + nsITest - to be implemented by test xpcom components;
  + nsITestRunner, nsITestResult - implemented by the framework.

* testing is activated by '--test' command line option.

* test components can be organized into a tree hierarchy. Non-leaf components may implement nsIStringEnumerator in addition to nsITest to report subtrees.

* root test is read from preferences system;

* while tests are run, the framework collect results and reports progress. We tried to make the system as fault-tolerant as possible. The framework behavior in this respect is determined by nsITestResult implementation. The current one is designed in JUnit spirit:
  + a '.' is printed as a test begins;
  + a 'F' is printed as a failure is reported;
  + after the test tree traverse is completed, we print time elapsed since the framework initiated, traverse result (OK/Failures), number of tests run, number of failures and number of errors.
  + a list of observed test-events in the form 'location: type: message'. This allow IDE to jump to the event location.

* the following test-events are processed:
  + test failure in c++, send by NS_TEST_ASSERT preprocessor macro. The location of the failure is determined at compile time by preprocessor macros;
  + test failure in javascript, send by nsITestRunner method. The location is read from js machine.
  + a js console message. The location is read from nsISriptError if present. A console message can be an exception, a non-exception error (warning mostly) or a message.
  + a survivable c++ error.

* test-events locations in javascript are translated from installed files (file:///, chrome://) to their sources in the source tree by a RDF map. This allows IDE to open source file for chrome or js XPCOM module and jump to the line with the event.

* nsITestRunner has a doCommand method, which facilitate UI testing by issuing XUL command events, collecting all kinds of parsing/scripting errors and checking UI state after each event.

The framework is functional, but very raw. We are ready to work on documentation , porting to w32, build tree integration, and so on.

Comment 4

[Sergey Yanovich]

At this point:

http://git.aasii.org/?p=abstract/yanovich/abstract.git;a=snapshot;h=fe3687b450fb039e8cbd19e10a58a648a1b9cac7

we added support for asynchronous test. That is needed for elaborate chrome tests.

Earlier versions only collected results twice - after the main window was loaded, and after all tests completed. That was sufficient only to test one chrome window. In addition to this two point, now the framework releases ui thread to allow content processing and error collecting twice per each async test. After the test is first called, and after it tells it's done.

Comment 5

[Sergey Yanovich]

There is now an xpunit project at http://xpunit.mozdev.org/ which contains this framework.