Open Bug 673917 Opened 13 years ago Updated 2 years ago

Standardized API to animate spritesheets

Categories

(Core :: Layout, defect)

Product:
Core
Component:
Layout
Type:
defect

Tracking

()

People

(Reporter: gal, Unassigned)

References

Details

The currently fastest way to animate spritesheets (pngs with individual frames on a large sheet) on browsers that hardware accelerate 3D css transformed content is to put the actual spritesheet into a <div> with overflow hidden (set to the target frame size) and "shift" the spritesheet programmatically in JS via translate3d(). This is non-trivial, and my guess is that not only would a standardized API add clarity, but also might be further optimized.
Note http://blog.vlad1.com/2009/06/22/to-sprite-or-not-to-sprite/

Do we really want to _encourage_ spriting?
Hi Boris,

I see two different usecases here, which this proposal only focussed on the latter - CSS sprites and animated spritesheets.  It's true that spritesheets have downsides, but afaik, there's no better way to do 2D bitmap animations.

The only other solution would be a custom spritesheet format (like a zip file) that includes all individual frames and then have the client load up the individual frames. It's a bit like an animated GIF, but this would be a pretty huge task to solve.
Blocks: webapi
So the basic problem here is to have an element that can be flipped between different images very efficiently?

One option would be to use a canvas for your element and drawImage images into it. That would work with or without spriting. It would be simpler than the CSS transforms trick. In Gecko that would probably be faster than the transforms trick and use about the same amount of VRAM. In Webkit, it might be a little slower than the transforms trick, but it should use less VRAM.

Really though, you should be able to just do element.style.background = "url(frame.png)" and have that be very fast if frame.png has been preloaded. Or possibly element.style.background = "-moz-image-rect(url('frame.png'), 0, 0, 100, 100)" if you insist on using spriting. If that's not currently fast enough, we should improve browsers so it is, instead of adding new API.
Hi Robert,

(In reply to comment #3)
> So the basic problem here is to have an element that can be flipped between
> different images very efficiently?

Yes.

> 
> One option would be to use a canvas for your element and drawImage images
> into it. That would work with or without spriting. It would be simpler than
> the CSS transforms trick. In Gecko that would probably be faster than the
> transforms trick and use about the same amount of VRAM. In Webkit, it might
> be a little slower than the transforms trick, but it should use less VRAM.

No efficient, performance-wise. You'll need lots and lots of small canvases (in the hundreds) for a scene of ours, or one big canvas, but then we're back at the middle age, having to implement the whole scene graph and dirty rectangles ourselves. I'm still dreaming of a future where Canvas is fast enough to handle complex 2D games and JS fast enough to manage the entire redraw flow.

> 
> Really though, you should be able to just do element.style.background =
> "url(frame.png)" and have that be very fast if frame.png has been preloaded.
> Or possibly element.style.background = "-moz-image-rect(url('frame.png'), 0,
> 0, 100, 100)" if you insist on using spriting. If that's not currently fast
> enough, we should improve browsers so it is, instead of adding new API.

Yeah, the background-position thing would be the most natural way to do it today. It is, unfortunately, also the slowest way to do it today. If we can heavily speed this up, I'm all for it.
(In reply to comment #4)
> or one big canvas, but then we're
> back at the middle age, having to implement the whole scene graph and dirty
> rectangles ourselves.

Interesting. Your games benefit from not repainting the entire scene on every update?
(In reply to comment #5)
> (In reply to comment #4)
> > or one big canvas, but then we're
> > back at the middle age, having to implement the whole scene graph and dirty
> > rectangles ourselves.
> 
> Interesting. Your games benefit from not repainting the entire scene on
> every update?

We are not using canvas at all today because of perf implications, but theoretically yes, we might greatly benefit from not repainting the entire scene. Imagine an isometric map with hundreds of depth sorted elements, and a character walks by on a street. You wouldn't want to draw the whole scene over and over just to animate the character.

We would use canvas today if we could. We'd need it to redraw hundreds of animated images at > 30 fps at larger-than-1080p resolutions. There is no tech that can handle this cross-browser today, but the closest is still HTML/CSS.
(In reply to comment #6)
> We are not using canvas at all today because of perf implications,

Is that mainly the issue that canvas isn't GPU-accelerated on some important platforms? Or are there other issues?

> but
> theoretically yes, we might greatly benefit from not repainting the entire
> scene. Imagine an isometric map with hundreds of depth sorted elements, and
> a character walks by on a street. You wouldn't want to draw the whole scene
> over and over just to animate the character.

One question is whether that is worth optimizing. If your street graphics are animated themselves, it may not be.

Most game platforms redraw the entire scene on every frame, as far as I know.

> We would use canvas today if we could. We'd need it to redraw hundreds of
> animated images at > 30 fps at larger-than-1080p resolutions. There is no
> tech that can handle this cross-browser today, but the closest is still
> HTML/CSS.

Have you got benchmarks that you use to evaluate performance of various approaches? I think some public benchmarks would be really useful here. What do you think of JSGameBench?
(In reply to comment #7)
> (In reply to comment #6)
> > We are not using canvas at all today because of perf implications,
> 
> Is that mainly the issue that canvas isn't GPU-accelerated on some important
> platforms? Or are there other issues?

I'm honestly not sure. What I am pretty sure of, is that performance-wise, even in browsers were Canvas is GPU accelerated, Canvas performance is not even close to DOM performance. I'm working on a proof-of-concept for a canvas based renderer to have a solid proof.

> 
> > but
> > theoretically yes, we might greatly benefit from not repainting the entire
> > scene. Imagine an isometric map with hundreds of depth sorted elements, and
> > a character walks by on a street. You wouldn't want to draw the whole scene
> > over and over just to animate the character.
> 
> One question is whether that is worth optimizing. If your street graphics
> are animated themselves, it may not be.

Nope, streets are not animated. What kind of optimization would you propose?

> 
> Most game platforms redraw the entire scene on every frame, as far as I know.

Yeah, if you work with DirectX or OpenGL, you mostly redraw the entire scene. This is not possible with Canvas. We'd probably need a 1000-2000% performance increase to make this happen.

> 
> > We would use canvas today if we could. We'd need it to redraw hundreds of
> > animated images at > 30 fps at larger-than-1080p resolutions. There is no
> > tech that can handle this cross-browser today, but the closest is still
> > HTML/CSS.
> 
> Have you got benchmarks that you use to evaluate performance of various
> approaches? I think some public benchmarks would be really useful here. What
> do you think of JSGameBench?

We are working on a benchmark right now, but it only focusses on different aspects of DOM performance. We'll try to bake in different tech, SVG, Canvas and WebGL. JsGameBench is great, but not customizable enough for our needs.
(In reply to comment #8)
> (In reply to comment #7)
> > (In reply to comment #6)
> > > We are not using canvas at all today because of perf implications,
> > 
> > Is that mainly the issue that canvas isn't GPU-accelerated on some important
> > platforms? Or are there other issues?
> 
> I'm honestly not sure. What I am pretty sure of, is that performance-wise,
> even in browsers were Canvas is GPU accelerated, Canvas performance is not
> even close to DOM performance. I'm working on a proof-of-concept for a
> canvas based renderer to have a solid proof.

On the FishIETank canvas benchmark I get 40fps with 2000 fish on my Windows 7 (high-end) laptop in Firefox, running maximized on a 1920x1080 screen. (Note that 2000 fish is not offered directly in that page, you have to override it with JS.) Can you really beat that with a non-canvas approach? Or is this benchmark totally unrelated to what you need for games?
http://ie.microsoft.com/testdrive/Performance/FishIETank/Default.html
Using WebGL you can get considerably better sprite performance if you optimize carefully.
http://muizelaar.blogspot.com/2011/02/drawing-sprites-canvas-2d-vs-webgl.html

> > > but
> > > theoretically yes, we might greatly benefit from not repainting the entire
> > > scene. Imagine an isometric map with hundreds of depth sorted elements, and
> > > a character walks by on a street. You wouldn't want to draw the whole scene
> > > over and over just to animate the character.
> > 
> > One question is whether that is worth optimizing. If your street graphics
> > are animated themselves, it may not be.
> 
> Nope, streets are not animated. What kind of optimization would you propose?

I meant the optimization of tracking dirty rects and only repainting part of your canvas. That makes sense if you often only need to repaint a small part of the canvas. If you usually need to repaint most of the canvas, it's not worth it.

> > Most game platforms redraw the entire scene on every frame, as far as I know.
> 
> Yeah, if you work with DirectX or OpenGL, you mostly redraw the entire
> scene. This is not possible with Canvas. We'd probably need a 1000-2000%
> performance increase to make this happen.

I'm surprised, given the kind of numbers we can get on canvas benchmarks these days. I would love to see a benchmark that shows how you're trying to use canvas and how the performance isn't adequate. It may be that there are simple things we and other browsers can do to make it faster --- or it may be that we could tell you how to do things differently to go faster.
(In reply to comment #9)
> (In reply to comment #8)
> > (In reply to comment #7)
> > > (In reply to comment #6)
> > > > We are not using canvas at all today because of perf implications,
> > > 
> > > Is that mainly the issue that canvas isn't GPU-accelerated on some important
> > > platforms? Or are there other issues?
> > 
> > I'm honestly not sure. What I am pretty sure of, is that performance-wise,
> > even in browsers were Canvas is GPU accelerated, Canvas performance is not
> > even close to DOM performance. I'm working on a proof-of-concept for a
> > canvas based renderer to have a solid proof.
> 
> On the FishIETank canvas benchmark I get 40fps with 2000 fish on my Windows
> 7 (high-end) laptop in Firefox, running maximized on a 1920x1080 screen.
> (Note that 2000 fish is not offered directly in that page, you have to
> override it with JS.) Can you really beat that with a non-canvas approach?
> Or is this benchmark totally unrelated to what you need for games?
> http://ie.microsoft.com/testdrive/Performance/FishIETank/Default.html
> Using WebGL you can get considerably better sprite performance if you
> optimize carefully.
> http://muizelaar.blogspot.com/2011/02/drawing-sprites-canvas-2d-vs-webgl.html

Very interesting. It might just be that the gap between Windows and OS X is way huger than I thought. Out of curiosity, I just ran the Fishtank in Chrome 14 on my pretty capable Macbook Pro at 1680x928 and it gave me 14fps with 20(!) Fish.

Maybe my picture of the world is just skewed, and it might just be that DOM performance is more consistent across platforms.

> 
> > > > but
> > > > theoretically yes, we might greatly benefit from not repainting the entire
> > > > scene. Imagine an isometric map with hundreds of depth sorted elements, and
> > > > a character walks by on a street. You wouldn't want to draw the whole scene
> > > > over and over just to animate the character.
> > > 
> > > One question is whether that is worth optimizing. If your street graphics
> > > are animated themselves, it may not be.
> > 
> > Nope, streets are not animated. What kind of optimization would you propose?
> 
> I meant the optimization of tracking dirty rects and only repainting part of
> your canvas. That makes sense if you often only need to repaint a small part
> of the canvas. If you usually need to repaint most of the canvas, it's not
> worth it.

Yeah, true. Lots of different little regions shift a lot of the processing you need to do to the CPU (JS), which dramatically slows down the entire scene.

> 
> > > Most game platforms redraw the entire scene on every frame, as far as I know.
> > 
> > Yeah, if you work with DirectX or OpenGL, you mostly redraw the entire
> > scene. This is not possible with Canvas. We'd probably need a 1000-2000%
> > performance increase to make this happen.
> 
> I'm surprised, given the kind of numbers we can get on canvas benchmarks
> these days. I would love to see a benchmark that shows how you're trying to
> use canvas and how the performance isn't adequate. It may be that there are
> simple things we and other browsers can do to make it faster --- or it may
> be that we could tell you how to do things differently to go faster.

Yeah, only way to find out. I will work on an experimental renderer for our game engine and see how it goes, then report back.

Thanks!
(In reply to comment #10)
> (In reply to comment #9)
> > (In reply to comment #8)
> > > (In reply to comment #7)
> > > > (In reply to comment #6)
> > > > > We are not using canvas at all today because of perf implications,
> > > > 
> > > > Is that mainly the issue that canvas isn't GPU-accelerated on some important
> > > > platforms? Or are there other issues?
> > > 
> > > I'm honestly not sure. What I am pretty sure of, is that performance-wise,
> > > even in browsers were Canvas is GPU accelerated, Canvas performance is not
> > > even close to DOM performance. I'm working on a proof-of-concept for a
> > > canvas based renderer to have a solid proof.
> > 
> > On the FishIETank canvas benchmark I get 40fps with 2000 fish on my Windows
> > 7 (high-end) laptop in Firefox, running maximized on a 1920x1080 screen.
> > (Note that 2000 fish is not offered directly in that page, you have to
> > override it with JS.) Can you really beat that with a non-canvas approach?
> > Or is this benchmark totally unrelated to what you need for games?
> > http://ie.microsoft.com/testdrive/Performance/FishIETank/Default.html
> > Using WebGL you can get considerably better sprite performance if you
> > optimize carefully.
> > http://muizelaar.blogspot.com/2011/02/drawing-sprites-canvas-2d-vs-webgl.html
> 
> Very interesting. It might just be that the gap between Windows and OS X is
> way huger than I thought. Out of curiosity, I just ran the Fishtank in
> Chrome 14 on my pretty capable Macbook Pro at 1680x928 and it gave me 14fps
> with 20(!) Fish.

I wasn't content yet testing on OS X, so I tested on our Windows 7 machine, fairly recent medium-budget DELL desktop. I tested on Firefox 8 nighly, at 1080x1920, and got 21fps with 1000 fish. That being said, it was pretty clear that the reported fps didn't match the actual draws, which were rather 10-15fps.

Yes, this problem would go away with very high end graphic cards and CPUs. But other than the traditional game industry, we just can't force people to update their PCs and smartphones to play our games. The maximum we can do is to tell them to update their browsers.

> 
> Maybe my picture of the world is just skewed, and it might just be that DOM
> performance is more consistent across platforms.
> 
> > 
> > > > > but
> > > > > theoretically yes, we might greatly benefit from not repainting the entire
> > > > > scene. Imagine an isometric map with hundreds of depth sorted elements, and
> > > > > a character walks by on a street. You wouldn't want to draw the whole scene
> > > > > over and over just to animate the character.
> > > > 
> > > > One question is whether that is worth optimizing. If your street graphics
> > > > are animated themselves, it may not be.
> > > 
> > > Nope, streets are not animated. What kind of optimization would you propose?
> > 
> > I meant the optimization of tracking dirty rects and only repainting part of
> > your canvas. That makes sense if you often only need to repaint a small part
> > of the canvas. If you usually need to repaint most of the canvas, it's not
> > worth it.
> 
> Yeah, true. Lots of different little regions shift a lot of the processing
> you need to do to the CPU (JS), which dramatically slows down the entire
> scene.
> 
> > 
> > > > Most game platforms redraw the entire scene on every frame, as far as I know.
> > > 
> > > Yeah, if you work with DirectX or OpenGL, you mostly redraw the entire
> > > scene. This is not possible with Canvas. We'd probably need a 1000-2000%
> > > performance increase to make this happen.
> > 
> > I'm surprised, given the kind of numbers we can get on canvas benchmarks
> > these days. I would love to see a benchmark that shows how you're trying to
> > use canvas and how the performance isn't adequate. It may be that there are
> > simple things we and other browsers can do to make it faster --- or it may
> > be that we could tell you how to do things differently to go faster.
> 
> Yeah, only way to find out. I will work on an experimental renderer for our
> game engine and see how it goes, then report back.
> 
> Thanks!
> Very interesting. It might just be that the gap between Windows and OS X is way
> huger 

Canvas is not GPU-accelerated in a meaningful way on OS X in any browser I know of.  So far.
(In reply to comment #11)
> I wasn't content yet testing on OS X, so I tested on our Windows 7 machine,
> fairly recent medium-budget DELL desktop. I tested on Firefox 8 nighly, at
> 1080x1920, and got 21fps with 1000 fish. That being said, it was pretty
> clear that the reported fps didn't match the actual draws, which were rather
> 10-15fps.

My laptop's a year old now, so I'm surprised that it's so much faster than that sort of desktop machine. You might want to test IE9/10 as well. If they get significantly better performance than us, you could file a bug against us and provide the information in about:support.

> Yes, this problem would go away with very high end graphic cards and CPUs.
> But other than the traditional game industry, we just can't force people to
> update their PCs and smartphones to play our games. The maximum we can do is
> to tell them to update their browsers.

We're definitely working on getting canvas acceleration enabled for more users on older machines (e.g. Windows XP), and so are other browser vendors. Hopefully soon we'll have canvas acceleration on every device that has a GPU, which is most devices that aren't really old, even most existing smartphones. When we get there, I think you'll find canvas beating DOM manipulation, at least on these sprite-bashing benchmarks. I suspect even on your Dell desktop, canvas would out-perform DOM manipulation in Firefox by some distance.

Devices that don't have GPUs at all are just going to suck and there's not much anyone can do about that.

> Maybe my picture of the world is just skewed, and it might just be that DOM
> performance is more consistent across platforms.

Also, the only kind of really well-accelerated graphics shipping on phones right now AFAIK is iPhone Safari's CSS transforms, which are also accelerated on IE, Firefox and Chrome across various platforms, which is probably why people are going for that.
Hi Robert,

yes, CSS transforms make a real difference. We do sorting of objects not via zIndex but translateZ, giving us a huge performance boost. Tricks like that..

I'm definitely revisiting Canvas. Early tests on Windows with latest browsers seem promising, although it is very early to tell. Unfortunately it's not only Win XP that's missing GPU accel, but also OS X (20x slower), rendering any strategy to jump on Canvas exclusively impossible for now. The only browser on a Mac that AFAIK might give us enough Canvas perf is Safari 5.1 on Lion.

I'm eager to hear more about the progress on GPU rendering for Mac from different vendors.
Using image fragments (bug 790640) is another way to use sprites (at least in the near future). I don't know if this is any faster than transform or canvas.
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.