510394 - Add a way to get premultiplied data from a canvas

Status: NEW

(Core :: Graphics, task)

Description

[Jeff Muizelaar [:jrmuizel]]

get/putImageData on a canvas currently deal with non-premultiplied data. We store the data in a premultiplied form so there is a conversion cost any time these functions are called. It would be nice if authors could avoid this cost.

The best way to fix this is probably to add a format parameter to get/putImageData that specifies premultiplied data.

Comment 1

[Boris Zbarsky [:bzbarsky]]

Please keep in mind that there are already optional parameters get/putImageData can have per the HTML5 draft...

Comment 2

[Benoit Girard (:BenWa)]

This change sounds like a good mentored bug. If it's still relevant can you mentor this Jeff?

Comment 3

[Benoit Girard (:BenWa)]

Discussed with Jeff, we dont want to make this a mentored bug because we would like to suggest a web api for this once we see some applications bottle-necking on getImageData

Comment 4

[Claus Wahlers [:cwahlers]]

We have a need for this in Shumway. We are currently implementing (Flash) filters, and they generally operate on premultiplied data, we thus have to premultiply before applying filters (after getImageData), and un-premultiply after (before putImageData). This is unfortunate because it reduces performance and precision (rounding).

Proposed API:

Add "premultipled" property to ImageData (Boolean, read-only)

Add optional "premultipled" arguments to:
  - ImageData(data, sw, sh [,premultipled=false])
  - ImageData createImageData(sw, sh [,premultipled=false]);
  - ImageData getImageData(sx, sy, sw, sh [,premultipled=false]);

The resulting ImageData then contains non-premultiplied data (premultipled=false) or premultiplied data (premultipled=true). ImageData.premultipled is set accordingly.

Other methods respect ImageData.premultipled and operate accordingly (e.g., putImageData does not premultiply when ImageData.premultipled is true).

Comment 5

[Claus Wahlers [:cwahlers]]

Attachment: rev1

Implements the API i proposed in https://bugzilla.mozilla.org/show_bug.cgi?id=510394#c4 - (the only deviation being that i moz-prefixed ImageData.premultiplied -> ImageData.mozPremultiplied)

Comment 6

[Claus Wahlers [:cwahlers]]

Attachment: rev2

Implements the API i proposed in https://bugzilla.mozilla.org/show_bug.cgi?id=510394#c4 - (the only deviation being that i moz-prefixed ImageData.premultiplied -> ImageData.mozPremultiplied)

Comment 7

[Boris Zbarsky [:bzbarsky]]

We are generally aiming to not add new prefixed DOM APIs.  See <https://wiki.mozilla.org/WebAPI/ExposureGuidelines>.  I assume the sr you were planning to request here would have caught that.

The right way to go about doing this is probably:

1)  Raise a spec issue to add the functionality to the spec.  Unless that's already been
    done?
2)  If you can't wait for the spec, implement the functionality as a separate function
    (e.g. getImageDataPremultiplied/createImageDataPremultiplied) that's preffed off by
    default but allowed in either the Shumway scope or if some pref is set (the latter so
    that web developers can experiment with it as needed).  That's assuming you ever plan
    to expose it to the web.

This does make |new ImageData| a bit difficult, but you could use a static method on the ImageData interface, which is also preffed off by default...

Comment 8

[Jeff Muizelaar [:jrmuizel]]

Comment on attachment 8350995 [details] [diff] [review]
rev2

Review of attachment 8350995 [details] [diff] [review]:
-----------------------------------------------------------------

This seems to be a diff from the previous version. Also what bz said about standardization.

Comment 9

[Chris Peterson [:cpeterson]]

This would be useful for Shumway, but we would not need it if we got CSS/SVG Filters for Canvas (bug 927892).

Comment 10

[Jeff Muizelaar [:jrmuizel]]

One question to answer about this is what to do if the provided premultiplied data has values that are greater than the alpha values. The way we deal with this kind of data in the platform is undefined. i.e. Skia and cairo do different things.

Comment 11

[Chris Peterson [:cpeterson]]

We currently do premultiplication in JS. What is the maximum potential performance improvement? Milan recommends benchmarking with and without our JS premultiplication (even if the result looks bad during the test).

Comment 12

[Jeff Muizelaar [:jrmuizel]]

One thing to note about exposing premultiplied data to js is that allows super luminescent pixels (where the alpha is less than the color value) this produces undefined results on some of our backends (notably skia). It's probably worth asking if we're comfortable with that.

Comment 13

[Michael Bebenita [:mbx]]

Attachment: Benchmark.

Firefox 35 - unpremultiply 25ms, putImageData: 30ms ~ 16 fps.
Chrome 39 - unpremultiply 29ms, putImageData: 5ms ~ 30 fps.

This is all unnecessary overhead, and would completely disappear if we had a way to upload premultiplied data with putImageData.

Comment 14

[Till Schneidereit [:till]]

This is still highly relevant for Shumway, but doesn't block us from shipping.

Comment 15

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee didn't login in Bugzilla in the last 7 months.
:bhood, could you have a look please?
For more information, please visit auto_nag documentation.