Firefox does not show arithmetic coded jpegs
Categories
(Core :: Graphics: ImageLib, defect)
Tracking
()
People
(Reporter: mail2jmsmith, Unassigned)
References
()
Details
|
||
Updated•13 years ago
|
|
||
Comment 1•13 years ago
|
||
|
||
Updated•13 years ago
|
|
|
||
Updated•13 years ago
|
|
|
||
Updated•13 years ago
|
|
|
||
Comment 3•13 years ago
|
||
|
|
||
Updated•13 years ago
|
|
|
||
Comment 4•13 years ago
|
||
|
|
||
Comment 5•13 years ago
|
||
|
|
||
Comment 6•13 years ago
|
||
|
||
Comment 7•13 years ago
|
||
|
|
||
Updated•13 years ago
|
|
||
Comment 10•13 years ago
|
||
|
|
||
Updated•13 years ago
|
|
||
Comment 11•13 years ago
|
||
|
Reporter | |
Comment 12•13 years ago
|
||
|
||
Comment 13•10 years ago
|
||
|
|
||
Comment 14•10 years ago
|
||
|
|
||
Comment 15•10 years ago
|
||
|
||
Comment 16•10 years ago
|
||
|
||
Comment 17•10 years ago
|
||
|
||
Comment 18•8 years ago
|
||
|
|
||
Comment 19•8 years ago
|
||
|
|
||
Comment 20•8 years ago
|
||
|
||
Comment 21•8 years ago
|
||
|
|
||
Comment 22•8 years ago
|
||
|
||
Comment 23•8 years ago
|
||
|
|
||
Comment 24•8 years ago
|
||
|
|
||
Comment 25•8 years ago
|
||
|
|
||
Comment 26•8 years ago
|
||
|
|
||
Comment 27•8 years ago
|
||
|
||
Comment 28•5 years ago
|
||
(In reply to Seth Fowler [:seth] [:s2h] from comment #17)
Before reopening, we'd need at least two browser vendors to
commit to supporting arithmetic-coded JPEGs. Mozilla could be one of them,
but we won't implement this without evidence that it's going to be
interoperable in the future.
By this logic, no web browser would support anything, and there wouldn't be any web browsers in the first place.
It's not other browsers that determine the requirements, it's the needs of your users. Being one of them, I can tell you that support of arithmetic coding is badly needed. Whenever I put Gimp-created images on the Web, people complain that they don't see them. Then I have to tell them to download the image and open it with a graphics viewer, because Firefox left this bug unresolved for more than 9 years. That's ridiculous.
|
|
||
Comment 29•5 years ago
|
||
I don't ultimately have any skin in this game, but as the sole maintainer and principal developer of libjpeg-turbo, I am in a good position to play devil's advocate:
-- Supporting arithmetic decoding in Firefox would require enhancing the arithmetic decoder in libjpeg-turbo to support suspension. I am unwilling to do that work unless someone fully pays for my labor, and the project is not likely to be cheap.
-- The current performance of the arithmetic encoder and decoder is quite poor. Much better performance can be achieved with progressive Huffman encoding/decoding, and the progressive Huffman codec will receive additional performance and fault tolerance enhancements in libjpeg-turbo 2.1.
-- Arithmetic coding provides only a small size advantage (in my testing, ~7-8% on average) over optimized progressive Huffman coding, and that advantage goes away if you are willing to use an asymmetric encoder such as mozjpeg, which exchanges encoding performance (to the tune of about 50-100x) for an incremental decrease in size.
The official builds of libjpeg-turbo have supported arithmetic coding for many years, so I don't really care one way or another. If the community wants arithmetic decoding in browsers badly enough for someone to pay for my labor to implement suspension in the libjpeg-turbo arithmetic codec, then great. Bring it. But also, GIMP doesn't generate arithmetic-coded images unless you tell it to, so I don't buy the argument that arithmetic decoding is necessary to support GIMP. As a web developer, why would you purposefully put an arithmetic-coded image on the web knowing that most browsers can't decode it? That's just creating your own pain in order to try to prove a point.
|
|
||
Comment 30•5 years ago
|
||
DRC, thanks for your verbose reply. Could you please share comparison results and methods you've used to compare arithmetic v.s. progressive Huffman encoding? I haven't made such a comparison myself, but if the advantage is 10% then probably it does not worth the effort to pull that into browser. Although one may argue that 10% is anyway good saving of your hard drive space...
|
|
||
Comment 31•5 years ago
•
|
||
Using the five images described in this article,
I compared
TJ_OPTIMIZE=1 tjbench {image} 95 -rgb -quiet -progressive
with
TJ_ARITHMETIC=1 tjbench {image} 95 -rgb -quiet -progressive
(NOTE: you can add -benchtime 0.001 -warmup 0 to speed things up, if you just want to measure the compression ratio rather than the performance. Replace -quiet with -qq to generate spreadsheet-friendly results.)
Compression ratio improvement with progressive arithmetic coding vs. optimized progressive Huffman coding was 2.4-10.2% (average 7.7%.)
For the set of Kodak test images, compression ratio improvement with progressive arithmetic coding vs. optimized progressive Huffman coding was 1.7-8.8% (average 5.2%.)
For the complete set of 8-bit RGB test images (the aforementioned set of five test images I typically use contains two of these), compression ratio improvement with progressive arithmetic coding vs. optimized progressive Huffman coding was 3.0-12.1% (average 7.9%.) The image with 12.1% improvement (spider_web.ppm) was an outlier. Without that image, the improvement was 3.0-10.4% (average 7.7%), which was very similar to the results from the set of five test images I use. (NOTE: that's why I use those images. Even though the origin of the first three images is somewhat odd, those five images have historically proven to represent a good range of "typical" JPEG codec behavior.)
As far as disk space, that's the raison d'etre for mozjpeg. It's intended as an offline tool for recompressing and optimizing existing JPEG images to save space. By combining optimized progressive Huffman coding with other techniques (trellis quantization, etc.), mozjpeg generally produces smaller JPEG files than can be produced using progressive arithmetic coding, and the files it produces are compatible with any browser. However, the tradeoff is that generating those files is extremely slow compared to generating "plain" optimized progressive Huffman images using libjpeg-turbo.
|
||
Comment 32•5 years ago
|
||
No horse in the race, but I'm curious whether or not arithmetic jpeg decoding time is actually a significant factor relative to the time needed for the rest of the page (e.g. network requests, disk/memory/cache i/o, rendering, flowing, vsync, etc.), using whatever current min-spec PC/mobile hardware and connection a web author would expect a user to have.
For instance, given a page horribly-encrusted with jpeg images, would the real-world time difference between loading it as huffman jpegs and arithmetic jpegs be palpably different to the user?
|
|
||
Comment 33•5 years ago
|
||
Let's put some rough (back-of-the-envelope) numbers on that, from my nearly 10-year-old machine (quad-core 2.8 GHz Intel Intel Xeon W3530.) This is old data and was obtained with libjpeg-turbo 1.3.x, but the relative differences should at least be similar with the current version.
For a set of test images, I compared the average decompression performance and compression ratio of baseline Huffman coding, progressive Huffman coding, and progressive arithmetic coding ("bits per pixel" is just 24 bits, the bit depth of the source images, divided by the compression ratio.)
| Entropy coding type | Average compression ratio / bits per pixel | Average decompression performance (Megapixels/sec) |
|---|---|---|
| Baseline Huffman | 13.0:1 / 1.85 | 127 |
| Progressive Huffman | 14.2:1 / 1.69 | 46.4 |
| Progressive Arithmetic | 15.3:1 / 1.57 | 19.3 |
For a hypothetical 1-megapixel image, this would amount to the following decompression and transmission times (all in milliseconds) for various networks:
| Entropy coding type | Decomp. time | Trans. time (1 Mbps) | Trans. time (10 Mbps) | Trans. time (100 Mbps) | Trans. time (1 Gbps) |
|---|---|---|---|---|---|
| Baseline Huffman | 7.87 | 1850 | 185 | 18.5 | 1.85 |
| Progressive Huffman | 21.6 | 1690 | 169 | 16.9 | 1.69 |
| Progressive Arithmetic | 51.8 | 1570 | 157 | 15.7 | 1.57 |
Combining decompression and transmission time (NOTE: this assumes no pipelining of transmission and decompression, which may be a spherical chicken assumption), we get:
| Entropy coding type | Total time (1 Mbps) | Total time (10 Mbps) | Total time (100 Mbps) | Total time (1 Gbps) |
|---|---|---|---|---|
| Baseline Huffman | 1858 | 193 | 26.4 | 9.72 |
| Progressive Huffman | 1712 | 191 | 38.5 | 23.3 |
| Progressive Arithmetic | 1622 | 209 | 67.5 | 53.4 |
Actual mileage may vary, but for this specific test case, arithmetic coding is so slow that the decompression time outweighs any performance advantage from the compression ratio except on the slowest of networks. On a 1 Mbps network, arithmetic is only 5% faster than progressive Huffman, and on all other networks, it's much slower. Progressive Huffman definitely gives you more "bang for the buck" in terms of compression ratio vs. performance. That is likely to improve, given that progressive Huffman coding is receiving a lot of attention right now in the libjpeg-turbo community, whereas arithmetic coding is receiving exactly none.
|
|
||
Comment 34•5 years ago
|
||
NOTE: As a purveyor of remote desktop software that uses libjpeg-turbo (TurboVNC, specifically), I have evaluated whether progressive Huffman coding is even worth it. I ultimately concluded that, no, it wasn't. In real-world performance scenarios, I see pretty much what you see in the last table above-- you can get a barely perceptible performance improvement with progressive Huffman vs. baseline Huffman on very slow networks, but otherwise, baseline is as fast or faster.
|
|
||
Comment 35•5 years ago
|
||
Thanks, @DRC, that's exactly the sort of info I was curious to see.
I have to say, if it's that cpu-intensive to unpack the arithmetic-encoded images, then I think it's probably a favor to humanity as a whole not to use it. The added power consumption across the world doesn't seem worth the ~10% of storage space and bandwidth you get back. That not only hits the power grid but also shortens battery life on mobile devices.
Arithmetic coding is good for something like an embedded system (e.g. game console) where you can't just throw a few more dollars at the problem to create more space, so you have to make it work it the hard way, but it doesn't seem pragmatic on the web.
That's my 2¢ anyway.
|
|
||
Comment 36•5 years ago
|
||
Not even 10% if you're comparing arithmetic to progressive Huffman. In that case, the incremental storage/bandwidth saving is only about 8%, in exchange for more than doubling the compute time. I absolutely agree that arithmetic coding isn't worth it-- at least not until/unless it can be significantly sped up.
It's also worth noting that, if bandwidth is a concern for JPEG images on a web site, mozjpeg can be used to perform offline near-lossless recompression of existing JPEG images (the lossiness is due only to the use of trellis quantization, to the best of my understanding.) mozjpeg is an "asymmetric" codec, so it trades off extremely slow compression performance (like 30x slower than libjpeg-turbo) for a 15:1 compression ratio, which is almost as good as that of arithmetic coding. Since mozjpeg is generating standard progressive Huffman JPEG images, the decompression performance will be the same as that of any other progressive Huffman JPEG image. It's not a solution for real-time compression, but then again, neither is arithmetic coding at the moment.
|
|
||
Comment 37•5 years ago
|
||
DRC, thanks for your informative replies, we all really appreciate that. Am I right saying that arithmetic JPEG provides currently the best compression ration in comparison to other formats/algorithms (like WebP) for the comparable output image quality? The comparisons I found (like this one https://developers.google.com/speed/webp/docs/webp_study) seems to use traditional Huffman JPEG encoder.
I agree that support for arithmetic encoding should not become de-facto standard for JPEG also because of compatibility issues. Although I must admit having that support in Mozilla engine would be a plus for applications that use this engine, like Thunderbird. Maybe that is also not good example, as once I insert/embed arithmetic-encoded image into email body, the recipient would need an email client that supports it as well, but I can think of other engine applications where one would like to render HTML with arithmetic-encoded image when rendering time does not matter much, but disk saving matters.
Maybe support of arithmetic encoding could be just a build option that is disabled by default?
|
|
||
Comment 38•5 years ago
|
||
I haven't tested the lossy modes of webp, but I've done similar research with other codecs in the past (including lossless webp), and my methodology was similar to the methodology used in Google's webp study. (I used the JPEG codec with specific settings as a baseline, measured the perceptual image quality using the structural dissimilarity [DSSIM] metric, dialed in the webp codec so that it barely produced the same DSSIM as the JPEG image, and compared the compression ratios.) If we can assume that the same relative difference in compression ratios (1.17) from my tests above (comparing progressive arithmetic coding to baseline Huffman coding) bears out for the images used in the Google study, then I expect that webp would still compress better than arithmetic-coded JPEG.
You highlighted the compatibility problem exactly. If one piece of software supports arithmetic-coded JPEG images, then that's just going to create cross-compatibility issues unless all popular software supports them. No web developer would ever use such images right now, because no browsers support those images. That doesn't just include browsers that use libjpeg-turbo. It also includes proprietary browsers, such as Safari. It wouldn't surprise me if the underlying proprietary Windows and macOS JPEG codecs supplied by Microsoft and Apple lack support for arithmetic coding, and that could be because arithmetic coding was encumbered by patents until the late 2000s. Even if web developers wanted to use arithmetic-coded JPEG images, PhotoShop (which doesn't use libjpeg-turbo) will not even open an arithmetic-coded JPEG image, much less create one. I think you'd be hard pressed to find any software, other than libjpeg-turbo itself, that enables the creation of arithmetic-coded JPEG images. Firefox and Chrome aren't the only libjpeg-turbo-based programs that have chosen to disable arithmetic coding, even though libjpeg-turbo enables it by default.
The only way to break that chicken-and-egg scenario would be for one popular piece of software to lead by example, to begin supporting arithmetic-coded JPEG images so that other pieces of software could follow suit. However, in terms of web development, it wouldn't do much good for Firefox and Chrome to support arithmetic-coded JPEG images unless proprietary browsers like Safari also supported them, and it wouldn't do much good for proprietary browsers to support them unless PhotoShop also supported them. If there were a solid argument to be made for the use of arithmetic-coded JPEG images, then it might make sense to try to steer that train from the caboose, but right now, that argument doesn't seem to exist. webp probably compresses just as well if not better, and mozjpeg can be used to produce progressive JPEG images with almost the same compression ratio as arithmetic-coded JPEG images, without the compatibility or decompression performance concerns.
As far as enabling arithmetic coding in Firefox, my understanding (from https://github.com/libjpeg-turbo/libjpeg-turbo/issues/120) is that you couldn't do that right now, because libjpeg-turbo's arithmetic decoder doesn't support suspension. That seems like a gating issue for this whole discussion. I've been an independent OSS contractor for 11 years, so I'll take anyone's money to develop any enhancement that makes sense for libjpeg-turbo, but adding suspension to the arithmetic decoder is probably not something that will happen without that money or a very-well-crafted code contribution.
Description
•