Reduce the amount of time it takes to run the WebGL conformance suite on the tinderboxen




Canvas: WebGL
7 years ago
4 years ago


(Reporter: Ehsan, Unassigned)


Firefox Tracking Flags

(Not tracked)


(Whiteboard: webgl-internal)


(2 attachments)



7 years ago
Benoit told me that there are a bunch of things which we can do to reduce the time it takes to run webgl tests, like not running the fuzzing tests on every check-in, and timing each test to see which ones take the longest, and just fixing them.
Created attachment 535732 [details] [diff] [review]
time the webgl mochitest

This patch measures how much each webgl test page takes.

It requests a GC run before each test, to prevent GC pauses from affecting the results (blaming a test page for stuff caused by another page).
Created attachment 535736 [details]
webgl tests timings

Here are the results, sorted in ascending order, on my linux/NVIDIA system. The median time is 335 ms, and there are 139 test. That means that if we fix the long-running tests, we can hope to get down to roughly 139*0.335 = 45 seconds, which would be a nice improvement.

The top ten offenders are:

WebGL test page conformance/read-pixels-test.html took 1222 ms
WebGL test page conformance/gl-teximage.html took 1297 ms
WebGL test page conformance/array-unit-tests.html took 1337 ms
WebGL test page conformance/buffer-preserve-test.html took 1341 ms
WebGL test page conformance/gl-vertex-attrib.html took 1654 ms
WebGL test page conformance/uninitialized-test.html took 1668 ms
WebGL test page conformance/type-conversion-test.html took 2778 ms
WebGL test page conformance/copy-tex-image-and-sub-image-2d.html took 3092 ms
WebGL test page conformance/gl-vertexattribpointer.html took 4956 ms
WebGL test page conformance/glsl-conformance.html took 5544 ms
Note --- that is in a debug build. Not sure which is most interesting to measure here.

Currently, the sum of the times taken by all the test pages here is 75 seconds. So we can hope to get from 75 to 45 seconds by fixing the long-running tests.

The main reason why WebGL mochitests can't be very fast is that just creating a OpenGL context is very long. On linux, it takes 120 ms with the NVIDIA driver, 50 ms with Mesa and open-source drivers, and 200 ms with the FGLRX driver. Here with the NVIDIA driver, the shortest-running test takes 172 ms.
Is there no way to reuse the same OpenGL/WebGL context for all the tests?
It's not impossible, but it would require resetting the OpenGL context to its default state, and there is no OpenGL function for doing that so that would have to be done manually, i.e. hundreds of OpenGL calls. Problems:
 - that would be a lot of code, potentially buggy, reducing the trust we have in our own tests
 - that could expose more driver bugs and would open the door to a bug triggered by one test impacting a subsequent test.

So, I would avoid doing that if possible.
Duplicate of this bug: 673459
Jonathan Griffin gave these interesting timings in bug 673459:
	win32-debug, 174951
	linux-opt, 21012
	macosx64-debug, 168473
	win32-opt, 24799
	linux64-debug, 150304
	linux64-opt, 20300
	linux-debug, 170302
I didn't realize that debug test times were so much higher than opt times. That's an interesting new element. It's not clear to me if there's much I can do, since many things in webgl are inherently computation-intensive, but it's interesting at least because it outlines that the incompressible part, spent waiting for the OpenGL implementation, is not that high.
Whiteboard: webgl-internal
You need to log in before you can comment on or make changes to this bug.