360998 - run w3c DOM suites for automated test coverage

Status: VERIFIED FIXED

(Core :: DOM: Core & HTML, defect)

Description

[Robert Sayre]

They store their tests in an XML format, and then transform it. We'll need to adapt their XSLT to produce tests that we can run in mochikit, instead of whatever JS lib they use.

Comment 1

[Robert Sayre]

Attachment: test-to-mochikit.xslt

quick and dirty version that gets some Level 1 Core stuff working.

Comment 2

[Robert Sayre]

Attachment: new and improved

Comment 3

[Bob Clary [:bc] (inactive)]

I'm all for dropping the use of JsUnit in the DOM tests and transforming the raw XML test descriptions into something else, but if you are keeping JsUnit etc what benefit do you get from mochikit?

Comment 4

[Robert Sayre]

I need to attach some more files. I deleted half of DOMTestCase.js and wrote some shims that map assertEquals and friends to the MochiKit stuff. This means all the JSUnit-esque assertions go through the MochiKit logger (and thus out to files, stdio, etc).

Comment 5

[Robert Sayre]

Attachment: generate MochiKit from W3C dom-level1-core tests

Comment 6

[Robert Sayre]

Attachment: generate MochiKit from W3C dom-level1-core tests

this patch doesn't include all of the generated test files, but there are enough there to get the idea. 

The exclusions.js file converts failing tests to todo() calls in the named files. Most of the ones we fail are DTD-related, but there are others.

Comment 7

[Johnny Stenback (:jst)]

Comment on attachment 253214 [details] [diff] [review]
generate MochiKit from W3C dom-level1-core tests 

Seems like Bob would be a more appropriate reviewer for this. I'll happily sr if there's a need for that here.

Comment 8

[Robert Sayre]

Attachment: bz2 of patch

Comment 9

[Bob Clary [:bc] (inactive)]

following up comment 3:

The patch takes the approach of taking the output of the W3C DOM Test Suite build process, modifying the resulting html files to use a modified version of the DOMTestCase.js with mochikit replacing JsUnit. The result is a static set of pages forked from the W3C DOM Test Suite that contains code that is copyright W3C and licensed under the W3C license.

I'm not particularly in love with the W3C DOM Test Suite or JsUnit, but I don't see the major benefit in going down this path. I do have the means to run the output of the W3C DOM Test Suite build process automatically and to either generate the test results as text files or by submitting the individual test results to a web service. As soon as we can get more of the buildbot infrastructure deployed, it will be relatively easy to execute the suite and collect the results.

If mochikit is desired, I would think that we could standardize on using it for our new mozilla.org dom tests and leave the W3C DOM Test Suite as is. 

If we must replace JsUnit, then we should take the approach that the W3C DOM Test Suite build process uses which is taking the raw xml format description of the testcases and transform them into mochikit based tests. This would alleviate any issue of forking or checking in W3C copyrighted materials into CVS.

Just my opinion.

Comment 10

[Robert Sayre]

Requirements:

* no dependencies other than what is already required by our build system 
* must be runnable by developers with that setup
* must report to the console
* must run automated

I am not going to get involved with JsUnit, so anyone who wants to use that should assign this bug to themselves.

Comment 11

[Robert Sayre]

oops, forgot one requirement:

* we don't pass all of the DOM suite, and it's unlikely we ever will, so the
  harness must have a facility for "todo" or "expected fail".

Comment 12

[Bob Clary [:bc] (inactive)]

ignore me, do whatever you want.

Comment 13

[Rob Campbell [:rc] (:robcee)]

Comment on attachment 253214 [details] [diff] [review]
generate MochiKit from W3C dom-level1-core tests 

let's hook it up and try it.