2033250 - Use libjpeg-turbo IDCT scaling to reduce memory and CPU cost of JPEG downscaling

Status: ASSIGNED

(Core :: Graphics: ImageLib, enhancement)

Description

[Tim Elliott]

Steps to reproduce:

The JPEG decoder currently decodes at full resolution and hands the result to the DownscalingFilter (Lanczos3 via SkConvolve). For images painted much smaller than their encoded size, this wastes both CPU and memory.

libjpeg-turbo can perform IDCT-time scaling at 1/2, 1/4, and 1/8 via a SIMD-accelerated path. This patch wires that into nsJPEGDecoder: when a smaller output size has been requested, we pick the largest supported IDCT factor that still leaves enough work for Lanczos3 to land on the target size, decode into that smaller intermediate, and let the downscaling filter finish the job.

Memory savings at the intermediate surface are significant — a 1/8 IDCT step cuts the buffer to 1/64 the full-decode size before Lanczos3 runs.

Notes / caveats:

• Skipped for images whose dimensions are not an integer number of MCUs. Some encoders (and a meaningful share of JPEGs in the wild) leave zeros or garbage in the unused rows/columns of the trailing MCU. At full-size decode those samples are cropped away, but IDCT scaling mixes them into neighboring output pixels and the artifacts become visible. See https://crbug.com/890745 and https://github.com/libjpeg-turbo/libjpeg-turbo/issues/297.

• A new pref image.jpeg.dct-scaling.min-factor controls how aggressive we're willing to be: it's the minimum remaining scale factor that must be left for Lanczos3 after the IDCT step. The default is 2.5, chosen conservatively after visually comparing a range of values — at 2.5 I saw essentially no scaling artifacts vs. pure Lanczos3.

• IDCT scaling is not gamma-aware. This matches the current behavior of the SkConvolve-based downscaler, so it's not a regression. Worth revisiting if/when we add gamma-aware scaling.

• Only 1/2, 1/4, and 1/8 are used here. libjpeg-turbo exposes additional intermediate factors (e.g. 3/8, 5/8) that could be considered later; I stuck to the SIMD-accelerated ones for this initial landing.

Side-by-side visual comparison of IDCT + Lanczos3 vs. pure Lanczos3:
https://idct-scaling-viz.netlify.app/

A new pref image.jpeg.dct-scaling.enabled (default false) gates the whole thing.

Benchmark Results: IDCT prescale + Lanczos3 vs. pure Lanczos3

Benchmark repo is here:
https://github.com/ender672/ff-streaming-scaler-scaffold

Input: giant-map.jpg (4992×4272, DCT-aligned — both axes divisible by 16)
Flags: --idct-min-factor 2.5
Methodology: 5 iterations, minimum wall time reported; peak RSS measured via getrusage(RUSAGE_SELF) in a forked child per run.

CPU (min wall time over 5 iterations)

config	target	IDCT + Lanczos3	pure Lanczos3	speedup (pure / IDCT)
1/2 barely	998×854	94.09 ms	167.19 ms	1.78×
1/2 heavy	500×428	90.81 ms	159.78 ms	1.76×
1/4 barely	499×427	78.06 ms	159.55 ms	2.04×
1/4 heavy	250×214	77.49 ms	156.39 ms	2.02×
1/8 barely	249×213	65.63 ms	156.29 ms	2.38×
1/8 heavy	125×107	65.62 ms	154.67 ms	2.36×

Memory (peak RSS via getrusage in forked child)

config	target	IDCT intermediate	IDCT + Lanczos3	pure Lanczos3	ratio (pure / IDCT)
1/2 barely	998×854	2496×2136	26.33 MB	87.66 MB	3.33×
1/2 heavy	500×428	2496×2136	24.04 MB	85.21 MB	3.54×
1/4 barely	499×427	1248×1068	8.59 MB	85.19 MB	9.92×
1/4 heavy	250×214	1248×1068	8.03 MB	84.41 MB	10.51×
1/8 barely	249×213	624×534	4.12 MB	84.39 MB	20.50×
1/8 heavy	125×107	624×534	4.17 MB	84.33 MB	20.23×

Comment 1

[Tim Elliott]

Attachment: Bug 2033250 - Use libjpeg-turbo IDCT scaling to reduce JPEG decode cost when downscaling.

When a JPEG is being decoded to a significantly smaller output size, ask
libjpeg-turbo to produce a 1/2, 1/4, or 1/8 scaled intermediate via its IDCT
scaling rather than decoding at full resolution and discarding most of the
data in the downscaling filter. The SurfacePipe is fed the scaled dimensions
and Lanczos3 finishes the job.

Restricted to images whose dimensions are a whole number of MCUs; otherwise
padding in the trailing MCU row/column can smear into visible pixels at
reduced scales (crbug.com/890745, libjpeg-turbo issue 297).

Gated by image.jpeg.dct-scaling.enabled (default off) with
image.jpeg.dct-scaling.min-factor controlling how much downscaling to leave
for Lanczos (default 2.5x).

Adds DownscaleDctOn/DownscaleDctOff gtest benchmarks.

Comment 2

[BugBot [:suhaib / :marco]]

The Bugbug bot thinks this bug should belong to the 'Core::Graphics: ImageLib' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 3

[BugBot [:suhaib / :marco]]

There is an r+ patch which didn't land and no activity in this bug for 1 week.
:tle, could you have a look please?
If you still have some work to do, you can add an action "Plan Changes" in Phabricator.
For more information, please visit BugBot documentation.