1588310 - Improving progressive rendering of incomplete JPEG scans

Status: RESOLVED FIXED

(Core :: Graphics: ImageLib, enhancement, P3)

Description

[Kornel]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:71.0) Gecko/20100101 Firefox/71.0

Steps to reproduce:

I'd like to get your opinion on a patch for libjpeg-turbo (used by Firefox), which changes how incomplete JPEG scans look like: https://github.com/libjpeg-turbo/libjpeg-turbo/issues/343

Progressive JPEGs could look nicer, and use less CPU, if Firefox adjusted its implementation.

Libjpeg applies smoothing to progressive JPEG scans, but the smoothing algorithm works as intended only when the browser instructs libjpeg-turbo to display only complete scans (it's still progressive rendering, but done in 3-5 discrete steps). The way Firefox is integrated with libjpeg-turbo makes it display all of the available image data, including incomplete scans (data within each scan appears to load from top to bottom). This breaks some assumptions in libjpeg's smoothing algorithm and makes progressive JPEGs look "blocky".

There's a related issue: progressive JPEG files may send color channels separately (MozJPEG does this by default). If the browser eagerly displays such image after receiving only one of two chroma channels, the image will be heavily tinted green, making faces look green: https://cloudinary.com/blog/progressive_jpegs_and_green_martians

The issue can be improved either on Firefox side, or worked around on libjpeg-turbo's side. Would you be interested in improving this on Firefox's side?

Expected results:

In technical terms:

1. If a JPEG is progressive, delay displaying of the image until DC data of all channels is available (DC data is in the first 10-15% of the file). Without waiting for all channels, JPEGs may exhibit the green faces problem. Libjpeg is also unable to fully apply smoothing at this stage, so incomplete DC data may look very blocky.

2. Instruct libjpeg-turbo to render only complete progressive scans. https://github.com/libjpeg-turbo/libjpeg-turbo/issues/343#issuecomment-538537190
   Currently Firefox appears to re-render JPEG whenever new data is available. That may get expensive, because the image may end up being re-rendered many many times. Firefox could libjpeg-turbo consume image data until Start of Scan or End of Image markers are seen, and re-render image only at these points.

The second change is a bit of a product decision, because rendering of incomplete progressive scans, on very large files/very slow network connections, does give more visual feedback of image loading. Limiting rendering to only complete scans will delay rendering by a scan. Would you be happy with such change?

Comment 1

[Timothy Nikkel (:tnikkel)]

Thanks for the detailed report!

I think we would be open to trying both in Firefox, but priority to assign dev resources to implement this would be fairly low.

Comment 2

[Moritz Firsching]

Attachment: Bug 1588310 - Improving progressive JPEG scans. r=tnikkel

When loading a progressive jpegs with non-interleaved DC scans,
we add a check to see if we have already seen data from all DC
channels and only start rendering when this is the case.

Comment 3

[Moritz Firsching]

By now, https://github.com/libjpeg-turbo/libjpeg-turbo/issues/343 has been fixed upstream so it will likely soon be part of the libjpeg-turbo version that is used in firefox, so the smoothing issue can be considered as fixed (thanks to Kornel!)

This leaves the "green faces" issue. I implemented the following fix described in "1." above:
We check if if at least some DC data from all color channels has arrived before displaying the first intermediate scan.

I generated two test files to exhibit the behavior: Traffic.jpg and Rubiks.jpg. They are from wikimedia commons, but recompressed using a scan script with separate DC scans for each color channel.
I attached animations showing current and fixed behavior when loading on a very slow connection, exaggerating the problem. Those are the attached files traffic_sign_old_firefox.webp, traffic_sign_new_firefox.webp, rubiks_old_firefox.webp and rubiks_new_firefox.webp.

Comment 4

[Moritz Firsching]

Attachment: Traffic.jpg

https://commons.wikimedia.org/wiki/File:Traffic_lights_sign,_Bunde_(2019)_01.jpg
Recompressed with progressive scan script

Comment 5

[Moritz Firsching]

Attachment: traffic_sign_old_firefox.webp

shows current behavior

Comment 6

[Moritz Firsching]

Attachment: traffic_sign_new_firefox.webp

shows desired behavior

Comment 7

[Moritz Firsching]

Attachment: Rubiks.jpg

Source:
https://commons.wikimedia.org/wiki/File:Rubik%27s_Cube_Collection_(4316806619).jpg
Recompressed with progressive scan script

Comment 8

[Moritz Firsching]

Attachment: rubiks_old_firefox.webp

Shows current behavior

Comment 9

[Moritz Firsching]

Attachment: rubiks_new_firefox.webp

Shows desired behavior.

Comment 10

[Moritz Firsching]

Here's the corresponding issue for chrome:
https://bugs.chromium.org/p/chromium/issues/detail?id=1145069

Comment 11

[Timothy Nikkel (:tnikkel)]

(In reply to Moritz Firsching from comment #3)

By now, https://github.com/libjpeg-turbo/libjpeg-turbo/issues/343 has been fixed upstream so it will likely soon be part of the libjpeg-turbo version that is used in firefox, so the smoothing issue can be considered as fixed (thanks to Kornel!)

This confused me because I couldn't find the change in the changelog. Looks like that change was only merged to the dev branch, which is labelled 2.1 pre beta (the 2.0.x being the current release series), so we still have some time to go before it makes it's way to a stable release of libjpeg-turbo and then to browsers.

Comment 12

[Moritz Firsching]

(In reply to Timothy Nikkel (:tnikkel) from comment #11)

(In reply to Moritz Firsching from comment #3)

By now, https://github.com/libjpeg-turbo/libjpeg-turbo/issues/343 has been fixed upstream so it will likely soon be part of the libjpeg-turbo version that is used in firefox, so the smoothing issue can be considered as fixed (thanks to Kornel!)

This confused me because I couldn't find the change in the changelog. Looks like that change was only merged to the dev branch, which is labelled 2.1 pre beta (the 2.0.x being the current release series), so we still have some time to go before it makes it's way to a stable release of libjpeg-turbo and then to browsers.

Exactly! Sorry for the confusion, I should have made that more clear. There is also an open issue for libjpeg-turbo to improve the smoothing even more. https://github.com/libjpeg-turbo/libjpeg-turbo/issues/459 . Seems like libjpeg-turbo 2.1 beta is coming pretty soon (see comment by dcommander on the issue from 22 days ago). For chrome some changes from the dev version of libjpeg-turbo are cherry picked to be present earlier (mainly security issues), so if we would really want to have these changes in faster we might consider doing that.

However the changes on this bug here are independent of the improvements in smoothing and since they are fixable without any changes to libjpeg-turbo, they could improve the rendering of images immediately.

Comment 13

[Timothy Nikkel (:tnikkel)]

Attachment: Bug 1588310. Add reftest.

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5ead3aca145c
Improving progressive JPEG scans. r=tnikkel

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1d2c31d3e489
Add reftest.

Comment 16

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/5ead3aca145c
https://hg.mozilla.org/mozilla-central/rev/1d2c31d3e489