1731134 - Consider changing "image-rendering: pixelated" to use a better scaling algorithm for inexact scale factors

Status: NEW

(Core :: Graphics: WebRender, enhancement)

Description

[Daniel Holbert [:dholbert]]

As allowed/suggested in https://drafts.csswg.org/css-images-3/#valdef-image-rendering-pixelated , we should consider changing 'pixelated' to use a better scaling algorithm that allows some smoothing/blur to avoid distorting the original image's proportions too much.

Current editor's draft spec text:

pixelated
The image is scaled in a way that preserves the pixelated nature of the original as much as possible, but allows minor smoothing instead of awkward distortion when necessary.

For each axis, independently determine the integer multiple of its natural size that puts it closest to the target size and is greater than zero.

Scale it to this integer-multiple-size as for crisp-edges, then scale it the rest of the way to the target size as for smooth.

Note: At integer multiples of the natural size, this gives the same results as crisp-edges. At non-integer multiples, this usually gives better visual results, even for pixel art, but it does incur a performance penalty due to the "two-step" rendering requirement.

See also https://github.com/w3c/csswg-drafts/issues/5837 (the issue where this was discussed/resolved).

Comment 1

[Daniel Holbert [:dholbert]]

Attachment: (image file for use in testcase)

Comment 2

[Daniel Holbert [:dholbert]]

Attachment: testcase 1

Comment 3

[Daniel Holbert [:dholbert]]

Attachment: screenshot of testcase 1, with bits of distortion highlighted

I think the goal here would be for the pixelated image (the middle image on each row) to avoid the sorts of distortion shown by the red arrows here, for example. (abrupt wobbles, and abrupt size changes that don't correspond to similar abrupt size changes in the original image)

Comment 7

[Emilio Cobos Álvarez [:emilio]]

Moving this to WR since that's where the work should be. Note that as of right now crisp-edges and pixelated behave the same (see bug 1728831).

Comment 8

[Will]

Apologies if this isn't the correct bug for this comment but according to 1954589, approximately 6% of icon requests require upscaling and hence may appear blurry. Please consider using integer upscaling for this approx 6% when possible because integer upscaling doesn't "add" blur, which is also important for things like fidelity and creators intent. A fantastic webpage with more about integer upscaling here https://tanalin.com/en/articles/integer-scaling/

Comment 9

[Daniel Holbert [:dholbert]]

(In reply to Will from comment #8)

Please consider using integer upscaling [...] A fantastic webpage with more about integer upscaling here https://tanalin.com/en/articles/integer-scaling/

The approach described is in that article is already something we do when we can, essentially; and to the extent that it differs, it's more of a suggestion to web developers than to things like browsers, because it's not something that browsers could do (except perhaps with additional syntax/APIs.)

tl;dr, that article recommends that people designing applications render small images like-so:
(1) When scaling a small image to fit a larger area, find the largest whole-number scale-factor that doesn't overshoot the target area.
(2) Then: scale-up the image, using that integer scale factor (in a pixelated style, scaling each pixel by the scale factor).
(3) Usually this won't exactly fill the target area, so add some padding on all sides as-needed to center the scaled-up image in the destination area.

(The article's clearest breakdown of this approach is in the "For example, if the game resolution is 320×240 [...]" near the top of the article, and also in the "Detailed" section with a stepwise breakdown.)

Step (3) above is a non-starter for CSS image-rendering:pixellated behavior right now, since it goes beyond the scope of what the image-rendering property is able to actually do, and it violates the expectations of how image drawing works on the web (as controlled by other properties like object-fit which is meant to switch between {contain/cover/fill}-style scaling to precisely fit the image to the target area).

So we can't do what that article suggests directly. But:

• Web content itself can opt to follow the recommendations in that article, and can choose to draw images at integer-multiple sizes (which we'll then draw in a pixellated way if they specify image-rendering:pixellated)
• Perhaps there could be a web API to let authors gracefully/trivially opt in to this behavior, e.g. some sort of modifier keyword for the CSS object-fit perhaps, to say object-fit: [...] integer-scale or something along those lines.

In any case, though, this bug is about how to paint images that are draw at a non-integer size; and the solution described in that article (just snapping to the nearest integer size and centering) is a non-starter based on the spec text and how image-rendering fits into the related image-sizing/drawing CSS properties.