1083516 - "image-rendering: -moz-crisp-edges" doesn't work for <video>, on hardware accelerated backends

Status: RESOLVED WORKSFORME

(Core :: Graphics, defect, P3)

Description

[Daniel Holbert [:dholbert]]

Attachment: testcase 1

STR:
 1. Load attached testcase.

EXPECTED RESULTS:
Top video should show crisp blocks of color.

ACTUAL RESULTS:
Extremely fuzzy blocks of color. ("-moz-crisp-edges" is being ignored.)

Windows & Mac give me ACTUAL RESULTS; Linux gives me EXPECTED RESULTS.

(Also, Windows & Mac start working if I disable HW acceleration in the Advanced area of Preferences. Windows-on-BrowserStack also just works by default, presumably because it doesn't have HW acceleration available.)

Comment 1

[Daniel Holbert [:dholbert]]

(I'm using <video> with -moz-crisp-edges in my reftests for bug 624647 -- that's how I ran into this. Due to this bug, my reftests are way too fuzzy on Mac & Windows, so I'll have to mark them "fails" on those platforms for now.)

Comment 2

[Daniel Holbert [:dholbert]]

(I'm also not 100% sure that image-rendering *should* work for <video>; the spec says:
  # When specified on an element, it applies to [...] the content of
  # replaced elements when they represent an image that must be scaled.
http://dev.w3.org/csswg/css-images-3/#the-image-rendering

I'm not sure whether a video frame "represents an image", but I tend to think that it does. In any case, we're inconsistent on how we handle this currently (depending on our 2d-graphics backend), and we should be consistent.

Comment 3

[Daniel Holbert [:dholbert]]

(Minor tangent: Chrome (version >= 32) actually renders the testcase with crisp edges, REGARDLESS of the "image-rendering" property's value. This suggests that Blink is defaulting to some edge-preserving upscaling algorithm, for videos (but not for PNG images, as I've determined through separate testing).  This is a strange inconsistency. But anyway, it doesn't really impact what we should do about this bug here.)

Comment 4

[Milan Sreckovic [:milan] (needinfo for best results)]

(In reply to Daniel Holbert [:dholbert] from comment #2)
> ...
> 
> I'm not sure whether a video frame "represents an image", but I tend to
> think that it does. In any case, we're inconsistent on how we handle this
> currently (depending on our 2d-graphics backend), and we should be
> consistent.

I believe we're consistent now, in that we're fuzzy everywhere.  Wrong, assuming image-rendering applies to video, but consistent.
Incidentally, I see the same (fuzzy) results in Chrome.

Comment 5

[Daniel Holbert [:dholbert]]

(In reply to Milan Sreckovic [:milan] from comment #4)
> I believe we're consistent now, in that we're fuzzy everywhere.

Nope, Linux still gives me EXPECTED RESULTS on latest Nightly. (Top half of my testcase is crisp, bottom half is fuzzy)

> Incidentally, I see the same (fuzzy) results in Chrome.

Chrome doesn't support "-moz-crisp-edges" of course. :)  So it makes sense that they'd render the testcase all-fuzzy.  Having said that -- their rendering doesn't change (it stays fuzzy) if I add "image-rendering: pixelated" which should be equivalent & supported for them.

Comment 6

[Daniel Holbert [:dholbert]]

Attachment: testcase 2: now with "image-rendering: pixelated" for comparing against Chrome

Comment 7

[Milan Sreckovic [:milan] (needinfo for best results)]

My bad, I wasn't unaccelerated enough when I tested.  I do see the difference between basic and accelerated compositor.

Comment 8

[Milan Sreckovic [:milan] (needinfo for best results)]

Attachment: Bug 1083516: D3D11 compositor ignores the sampling filter setting for YCbCr effects. .schouten

Review commit: https://reviewboard.mozilla.org/r/129376/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/129376/

Comment 9

[Milan Sreckovic [:milan] (needinfo for best results)]

Not sure about OS X.  We seem to be asking for nearest, and the results look different between the two options, but it doesn't look quite right.

Comment 10

[Bas Schouten (:bas.schouten)]

Comment on attachment 8857386 [details]
Bug 1083516: D3D11 compositor ignores the sampling filter setting for YCbCr effects. .schouten

https://reviewboard.mozilla.org/r/129376/#review132694

The problem is this sample filter would also be used for sampling the Cb and Cr textures up to full resolution (from their native half resolution in YUV 4:2:0), that will have seriously negative quality affects beyond just sampling the full color image differently.

Comment 11

[Milan Sreckovic [:milan] (needinfo for best results)]

Another way to explain the comment above: image-rendering kicks in when the image is being scaled.  If it isn't (being scaled) than this value is irrelevant.  However, with a YUV image format, since UV are saved at a different resolution than Y, there is an implicit scaling step regardless of whether the overall image is being scaled or not.  So, setting to pixelated would change what the video looks like even when displayed in native resolution.

On the other hand, the fact that we're interpolating in the first place is wrong based on the YUV420 definition; we're supposed to reuse the UV twice/four times, rather than interpolate between them.  So, in a way, we're wrong already, though our wrongness is in the smoothing side, so not easily observed.

The attached patch is wrong as well, and actually more, in that while it does snap, it snaps to potentially wrong values.

Comment 12

[Milan Sreckovic [:milan] (needinfo for best results)]

Oh, and pixelated is what we're going for here, the crisp-edges is supposed to be edge detection/contrast preservation and we're certainly not doing anything like that here.

Comment 13

[Daniel Holbert [:dholbert]]

Maybe my EXPECTED RESULTS in comment 0 are incorrect then?  I'm OK with that; though we're still inconsistent between platforms, so it still seems like there's a bug here, one way or another.

Comment 14

[Milan Sreckovic [:milan] (needinfo for best results)]

No, you are correct, pixelated should give what you're expecting, and it doesn't.  Perhaps crisp-edges could be argued, but the spec is loose enough in saying "do your best if you can't do the right thing", that we could interpret it any way we wanted to :)

I never managed to ask this question though: beyond our tests and spec (which we all currently fail to observe :), is there a realistic use case we know of?  Is there "external" pressure to do it, or an urgency?  I'm asking because the one line above is arguably wrong, but probably good enough if we need something quickly.  And to get it 100% correct for pixelated would take a bit of extra code (which we then have to maintain), while to get it 100% for crisp-edges would take a lot more, if we're to do the actual edge detection and mess with those kinds of algorithms.

Comment 15

[Daniel Holbert [:dholbert]]

(In reply to Milan Sreckovic [:milan] from comment #14)
> I never managed to ask this question though: beyond our tests and spec
> (which we all currently fail to observe :), is there a realistic use case we
> know of?  Is there "external" pressure to do it, or an urgency?

I'm not personally aware of any external pressure for an actual better-than-nearest-neighbor "crisp edges" behavior for video (i.e. with edge detection).

Note also that the spec does explicitly say that nearest-neighbor is an acceptable algorithm for "crisp-edges" (and that no particular algorithm is required), too:
"This property does not dictate any particular scaling algorithm ...with 'image-rendering: crisp-edges', a user agent might scale images with nearest-neighbor interpolation by default, and switch to EPX interpolation in low-load situations."
 https://drafts.csswg.org/css-images-3/#the-image-rendering

Comment 16

[Gregory Pappas [:gregp]]

Are you still able to reproduce this? Both test cases seem to exhibit the expected behaviors in Firefox and Chromium on my machines (tested Windows and Linux, not sure about macOS)

Comment 17

[Daniel Holbert [:dholbert]]

Tested on macOS, and the testcases behave as-expected there too. Let's call this WORKSFORME. Thanks!