651060 - Decode JPEGs to YCbCr and do RGB conversion on the GPU

Status: NEW

(Core :: Graphics, enhancement)

Description

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Rob notes that

For JPEG images, we could also decode to YCbCr format (i.e. skip the final
YCbCr-to-RGB conversion in the imglib decoder), and do the final YCbCr
conversion using the GPU at draw time. That would save half the memory since
JPEG subsamples Cb/Cr. We already have all the layer-side support for this.

Comment 1

[Jeff Muizelaar [:jrmuizel]]

Color correction makes this more difficult because we would need to do color correction on draw instead of decode. Mobile doesn't use color correction so it's not a problem there.

I think we would also probably only want to do this for large images, but that would need to be measured.

The memory win is probably the most appealing part of this, so it might even be interesting to investigate doing this even without using the GPU, at the same time I wouldn't rush to work on this without more evidence that it will provide big wins.

Comment 2

[Siarhei Siamashka]

As somebody mentioned on irc some time ago, it should be possible to first do "decode -> YCbCr-to-RGB -> color correction -> RGB-to-YCbCr" (which can be possibly optimized to less stages), and then "YCbCr-to-RGB" when drawing (using NEON or GPU). This way we still have color correction and memory saving.

Comment 3

[Kelsey Gilbert [:jgilbert]]

(In reply to Siarhei Siamashka from comment #2)
> As somebody mentioned on irc some time ago, it should be possible to first
> do "decode -> YCbCr-to-RGB -> color correction -> RGB-to-YCbCr" (which can
> be possibly optimized to less stages), and then "YCbCr-to-RGB" when drawing
> (using NEON or GPU). This way we still have color correction and memory
> saving.

Or just do color correction at GPU-conversion-time, as a suffix to YCbCr-to-RGB.

There's no reason we have to do this at draw time, either. Bug 750994 is an optimization to do GPU-size texture conversions at upload time.

Comment 4

[Justin Lebar (not reading bugmail)]

Not to suggest that this isn't a worthwhile cause, but I don't think this should block bug 683284.  Bug 683284 comment 1 says "I'd like this bug to only address bugs where our behaviour is a clearly wrong compared to other browsers."

Comment 5

[berrkl]

IE11 does this: https://blogs.msdn.com/b/ie/archive/2013/09/12/using-hardware-to-decode-and-load-jpg-images-up-to-45-faster-in-internet-explorer-11.aspx

High quality image downsampling also can be done in subsampled YCbCr space (just treat each channel separately, maybe doing just luma and not bothering with chroma).

Comment 6

[Chia-hung Tai [:ctai :ctai_mozilla :cht]]

I would like to do some experiments on this bug for evaluating the benefit and cost on Firefox OS.

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee didn't login in Bugzilla in the last 7 months.
:bhood, could you have a look please?
For more information, please visit auto_nag documentation.

Comment 8

[BugBot [:suhaib / :marco/ :calixte]]

The severity field for this bug is relatively low, S3. However, the bug has 10 votes.
:bhood, could you consider increasing the bug severity?

For more information, please visit auto_nag documentation.

Comment 9

[Jeff Muizelaar [:jrmuizel]]

Nope. This should be S3.