676138 - WebGL performance regression tests

Status: RESOLVED FIXED

(Core :: Graphics: CanvasWebGL, defect)

Description

[Benoit Jacob [:bjacob] (mostly away)]

Bug 674731 has shown that we need performance regression tests for WebGL.

Do we have a framework for performance tests that we could reuse for that? Is Talos what we want?

Any wiki page explaining how to get started with that?

Comment 1

[JP Rosevear [:jpr]]

Doesn't Marc Cote have a tool for this in progress?

Comment 2

[Jeff Muizelaar [:jrmuizel]]

Afaik this was for the microsoft demos

Comment 3

[Mark Côté [:mcote]]

It is indeed being used for the MS demos at the moment, but it could potentially be used for any JavaScript-based tests that can provide relatively simple test results (e.g. fps).  I tried to make it pretty generic.

I haven't had a chance to work on it for a little while, but I intend to get back to it and add more tests before too long.

Comment 4

[Kelsey Gilbert [:jgilbert]]

Talking to Ehsan, we should use Talos tests for this.

This should be relatively high-priority especially in light of bug 726396 and bug 724476.

Comment 5

[Benoit Jacob [:bjacob] (mostly away)]

This should be our highest priority once we're done with fixing current regressions and fixing 1.0.1 conformance (hopefully in a few days).

But, everyone seems to have a mystical fear of Talos. Maybe because of "Eros and Talos" ? anyway. Let's file a separate bug for first writing performance tests as pain web pages. This will presumably get done much faster, will be generally useful, and once it's done the Talos part will be doable by anyone without WebGL knowledge.

Comment 6

[Jeff Muizelaar [:jrmuizel]]

FWIW, the talos part is very easy. You basically just call a js function with a time value. Still, standalone performance tests are a very good idea too.

Comment 7

[Dan Mosedale (:dmosedale, :dmose)]

Facebook's jsgamebench includes a webglbench directory.  Whether it's even slightly possible that it's suitable for this I haven't checked, but it could be worth a look.

Comment 8

[Benoit Jacob [:bjacob] (mostly away)]

WebGL performance regression tests are here:
http://hg.mozilla.org/users/bjacob_mozilla.com/webgl-perf-tests

Comment 9

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Going to take a stab at this, using Talos.  Hard thing is that some tests don't really report a value, they just show a value while running (e.g. fps for the MS demos).

Comment 10

[Mark Côté [:mcote]]

Are you trying to get detailed performance information, or just what the test measures, e.g. fps? If it's the latter, my SpeedTests framework would probably be simpler for this--although at the moment it is only run twice a day. I could conceivably make this continuous. Just curious as to what your exact requirements are.

I am plotting results from various tests at http://mrcote.info/speedtests/results.html alongside those from other browsers.

Comment 11

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Well, talos already includes things like getting memory/cpu/etc. info which would be nice to have, but the core need is just a single benchmark result per test.  SpeedTests seems to do that, but how hard is that to deploy across different OS/machines?  We'd need to cover at the very least Win7, OSX, Android, B2G.

Comment 12

[Benoit Jacob [:bjacob] (mostly away)]

(In reply to Vladimir Vukicevic [:vlad] [:vladv] from comment #9)
> Going to take a stab at this, using Talos.  Hard thing is that some tests
> don't really report a value, they just show a value while running (e.g. fps
> for the MS demos).

Hey, don't hijack this bug for general performance tests! This was only about WebGL performance regression tests. The tests to be integrated into talos are http://hg.mozilla.org/users/bjacob_mozilla.com/webgl-perf-tests

Comment 13

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Not hijacking, I assumed it was for general webgl regression tests, not specifically for yours :)  Either way, we should look into whether integrating it into talos or into speedtests is better..

Comment 14

[Benoit Jacob [:bjacob] (mostly away)]

Sure, not specifically mine --- but this should remain WebGL focused; I was referring to your mention of MS tests:

(In reply to Vladimir Vukicevic [:vlad] [:vladv] from comment #9)
> Going to take a stab at this, using Talos.  Hard thing is that some tests
> don't really report a value, they just show a value while running (e.g. fps
> for the MS demos).

^ this

Comment 15

[Mark Côté [:mcote]]

Yeah, speedtests does not gather data about memory, cpu, etc. At the moment it is only running on Windows 7. OS X would not be too difficult. Android is planned but relies on some other infrastructure first. So if you are looking to get this up on all platforms asap, speedtests may not be the way, although I have no idea how much work it would be to get it going on talos.

Comment 16

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

(In reply to Benoit Jacob [:bjacob] from comment #14)
> Sure, not specifically mine --- but this should remain WebGL focused; I was
> referring to your mention of MS tests:

Pretend I said "Google WebGL samples" then :D

Comment 17

[Kelsey Gilbert [:jgilbert]]

We have some, even if they're not perfect.
More would be better, but that's a job for another bug.