407049 - Canvas regression tests

Status: RESOLVED FIXED

(Core :: Graphics: Canvas2D, defect)

Description

[Philip Taylor]

User-Agent:       Opera/9.24 (X11; Linux i686; U; en)
Build Identifier: 

Regression tests would be quite useful, and I can't see many existing ones for <canvas>.

Reproducible: Didn't try

Steps to Reproduce:
1. Break canvas.
2. Run automated tests.

Actual Results:  
Nobody notices it broke.

Expected Results:  
Somebody should notice it broke.

Comment 1

[Philip Taylor]

Attachment: tests (version 1)

Here's four hundred Mochitests. Should work when put in content/canvas/test.

With "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9b2pre) Gecko/2007120221 Minefield/3.0b2pre" I get:

  Passed: 2440
  Failed: 0
  Todo: 151

(I looked for all the failing tests and deleted them or marked them as 'todo' - I'm assuming this would be used for regression testing, and so it shouldn't include bugs that haven't been fixed yet. Some more tests might need to be deleted if they fail on other platforms.)

The tests are built from the same source data as <http://philip.html5.org/tests/canvas/suite/tests/>, and that version is useful for compliance testing and for other implementations. I expect it would be nice if there was just a single copy of the source data, rather than a separate Mozilla fork, to keep everything consistently up to date with the spec and with fixes; but that introduces a bottleneck on that external source, which might not be nice for Mozilla. I don't have any particular ideas about how this would be best handled.

Comment 2

[Justin Dolske [:Dolske]]

Fantastic!

Comment 3

[Boris Zbarsky [:bzbarsky]]

Attachment: Sort of as checked in

Gavin tested these on Windows, and I did on Mac.  There were a number of platform differences that were Mac-specific, so some of the tests currently run only on Mac, or only on non-Mac, and in one test I had to up the tolerances from 2 to 3.  With those changes, and with the changes to relative image URIs to reference the right images, things pass on all three platforms.

I had to split up the file list into multiple lists to get around the command-line length limitations of Windows.  Without that the nsinstall command line was too long.

Bugs that are filed on the remaining platform differences:
* Bug 401791 covers test_2d.line.miter.rightangle.html and
  test_2d.line.miter.lineedge.html passing on Mac and failing elsewhere.
* Bug 407104 covers test_2d.gradient.empty.html crashing on Mac.
* Bug 407105 covers test_2d.line.cap.zerobezier.round.html,
  test_2d.line.join.parallel.html, test_2d.strokeRect.zero.5.html failing on
  Mac but passing elsewhere.
* Bug 407107 covers test_2d.path.rect.selfintersect.html and
  content/canvas/test/test_2d.path.rect.zero.6.html failing on Mac but passing
  elsewhere.

Comment 4

[Boris Zbarsky [:bzbarsky]]

Fixed.  Philip, thanks again for the tests, and especially for making it so easy to get this landed!

Comment 5

[Robert O'Callahan (:roc) (email my personal email if necessary)]

test_toDataURL.primarycolours.html and test_toDataURL.complexcolours.html tend to fail randomly turning the tree orange, so I've disabled them. Bug 407986 filed to reenable them.