782054 - Canvas reads pixels from outside of the source rectangle when performing drawImage

Status: RESOLVED INVALID

(Core :: Graphics: Canvas2D, defect)

Description

[Katelyn Gadd (:kael)]

Attachment: Rendering in Firefox

When scaling or rotating an image in 2D canvas in Gecko, the pixels outside of the image are treated as opaque white. This is a problem because rotation and scaling can cause color values to be sampled from outside of the image boundaries. This creates white 'garbage pixels' (see attached screenshot) around image edges.

It's possible to compensate for this by padding all your images with a sufficiently large number of transparent pixels around the edges (though I'm not sure it's possible to trivially determine how many pixels are needed), but it's wasteful and also nontrivial if you're loading images that aren't custom-made for HTML5 canvas.

IE10 seems to do something similar, but Chrome does not - it treats pixels from outside the image as fully transparent. Arguably if a source image has an alpha channel, pixels from outside the image should always be transparent - I could see an argument for having white be the default color for images that lack an alpha channel.

URL points to a demo (private, please do not share) that suffers from this problem. You can press F12 to skip levels (the easiest place to see the artifacts are particle effects - present in the first few levels - but there are a few other rotated/scaled images that also demonstrate it.)

Seems possibly related to bug #450140.

Comment 1

[Katelyn Gadd (:kael)]

Attachment: Rendering in IE

Comment 2

[Katelyn Gadd (:kael)]

Attachment: Rendering in Chrome

Comment 3

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

Do you have a smaller test case than Escape Goat?

Comment 4

[Katelyn Gadd (:kael)]

Nope. If you need one I can try and write one, but I don't know how long it will take me to figure out the precise combination of parameters that causes it to reproduce.

Comment 5

[Katelyn Gadd (:kael)]

OK, I think this report was partially erroneous:

There used to be a bug in the FF canvas implementation where pixels outside the edges of a canvas where opaque white. I can't reproduce that anymore (thanks for fixing it :D) - I hadn't noticed the fix because I compensate by padding the edges of my canvases with empty pixels in most places.

The real bug appears to just be that when blitting a subregion of an image, the FF canvas implementation can sample from outside the subregion. I.E. if you blit the rect (2, 2)-(4, 4) from an image, pixels from outside that rectangle can contribute to the destination, *even if image smoothing is enabled*.

In Chrome, regardless of whether image smoothing is enabled, pixels from outside the source rectangle don't influence the output.
IE10 appears to display the exact same rendering characteristics as FF, probably because it's using Direct2D just like FF is.

https://dl.dropbox.com/u/1643240/canvas_artifacts.html

Note that while I think it would be okay for these artifacts to show up with image smoothing enabled - sampling a half texel around the edges isn't all that weird to me - the fact that it shows up with filtering disabled seems totally wrong to me.

I wasn't able to tell from skimming the canvas spec whether this is correct.

Comment 6

[Katelyn Gadd (:kael)]

even if image smoothing is enabled -> even if image smoothing is disabled.

Some more testing reveals that Opera has the same artifacts on Windows.

Safari nightly on windows has the same artifacts, but only if image smoothing is DISABLED - if I turn on image smoothing, the red pixel artifacts go away. Totally weird.

I figure maybe this is a Direct2D thing, because a friend tested this in Firefox on a mac and didn't see red pixels. (Same for Chrome on a mac).

Comment 7

[Katelyn Gadd (:kael)]

Disabling hardware acceleration on windows doesn't eliminate the artifacts, which is strange given that the same artifacts aren't seen on Mac.

Comment 8

[Katelyn Gadd (:kael)]

I updated the test case in dropbox to add a checkbox that causes the texture source coordinates to be offset by half a pixel (+0.5 pixels inward on all sides). While this suppresses the red pixels around the edges if smoothing is off, with smoothing enabled red pixels still show up. This suggests that this is more than just a simple texel center offset problem - it looks like pixels are always read from outside the source region, no matter what, perhaps because the quad being rendered actually covers more pixels than it should.

Clamping sampled texels to the source rectangle should fix this (you might also need to ramp down the opacity of texels from outside the source rectangle.)

Comment 9

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

This is a long-standing issue with various graphics APIs; it's not really canvas specific.  There's various solutions, but none of them perfect -- you can copy out the subimage and then render it (which is what that platform does for background images and the such).  You can handle it in your shader if you're using 3D rendering and clamp to a passed-in source rectangle.  You can pad your texture images etc.  But there's really no good generic solution for this -- any solution depends on the particulars of your source images, the rendering pipeline, etc.

Comment 10

[Katelyn Gadd (:kael)]

FYI according to whatwg they plan to update the Canvas spec to specify Chrome's behavior as the correct behavior. The Chromium guys said it was necessary for them to do this because the alternative broke too many Canvas use cases.

Comment 11

[Rik Cabanier]

This is as-designed.
If you don't want sample outside the source region, either:
- copy the source data to an intermediate canvas that has the reduced pixel size and use that to draw.
- draw with smoothing turned off.

I was unable to reproduce the issue with smoothing off. (Your example still smoothes)

Comment 12

[Katelyn Gadd (:kael)]

How does it still smooth given that I'm setting imageSmoothingEnabled to off? The artifact displayed in the screenshots still occurs on Win32 (with azure/d2d) when I run the linked test case in Aurora. The smoothing is only on the edges of the bitmap, and offsetting the coordinates hides it, so the problem appears to be that it's drawing a quad that's slightly too large or has softened edges.

Comment 13

[Rik Cabanier]

Can you create a test case that just reduces it to the minimum?
Your test does a lot of different things (scaling + animation + copying between canvas contexts).

I can see what the issue is: when you rotate, FF samples outside of the source rect on Windows. IE and Safari do this as well.

Comment 14

[Katelyn Gadd (:kael)]

The rotate is how you get the samples outside the source rect to happen; the copy to a canvas is to test the common statement that copying to a canvas fixes it.

I can reduce it to a static non-animated rendering if that helps.

Comment 15

[Rik Cabanier]

(In reply to K. Gadd (:kael) from comment #14)
> The rotate is how you get the samples outside the source rect to happen; the
> copy to a canvas is to test the common statement that copying to a canvas
> fixes it.
> 
> I can reduce it to a static non-animated rendering if that helps.

I created a test file: http://jsfiddle.net/6vh5q/9/
Browsers have wildly different behavior when you rotate. Chrome never samples outside the rect, IE always samples and FF *sometimes* samples.
Safari samples when smoothing is turned off (which makes no sense)

the test also lets you use an intermediate canvas to strip the red pixels.

Comment 16

[Katelyn Gadd (:kael)]

Thanks, I see what you mean now. I'll have to try using a temporary canvas as a workaround for rotated/scaled bitmaps in JSIL and eat the performance hit.