2020030 - (CVE-2026-4695) Invalid memory read access in aom_sse_avx2

Status: VERIFIED FIXED

(Core :: Audio/Video: Web Codecs, defect, P2)

Description

[Atte Kettunen]

Attachment: repro_cdef_min.html

Tested on:

OS: Ubuntu 25.10
Firefox: fuzzfetch fetched AddressSanitizer build

Version=150.0a1
BuildID=20260227090653
SourceStamp=69326b7edf8166a75eba45b4f156f7a9727be83a
ID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}

ASAN-trace(with repro-file as it is):

=================================================================
==85175==ERROR: AddressSanitizer: SEGV on unknown address 0x7bff2cb94880 (pc 0x7bfff31a9b5d bp 0x7bffa91e1730 sp 0x7bffa91e1610 T30)
==85175==The signal is caused by a READ memory access.
SCARINESS: 20 (wild-addr-read)
    #0 0x7bfff31a9b5d in aom_sse_avx2 /builds/worker/checkouts/gecko/third_party/aom/aom_dsp/x86/sse_avx2.c:0:0
    #1 0x7bfff38591dc in get_filt_error /builds/worker/checkouts/gecko/third_party/aom/av1/encoder/pickcdef.c:428:13
    #2 0x7bfff38591dc in av1_cdef_mse_calc_block /builds/worker/checkouts/gecko/third_party/aom/av1/encoder/pickcdef.c:601:33
    #3 0x7bfff37442c2 in cdef_filter_block_worker_hook /builds/worker/checkouts/gecko/third_party/aom/av1/encoder/ethread.c:3332:5
    #4 0x7bfff3204bf5 in execute /builds/worker/checkouts/gecko/third_party/aom/aom_util/aom_thread.c:197:27
    #5 0x7bfff3743657 in launch_workers /builds/worker/checkouts/gecko/third_party/aom/av1/encoder/ethread.c:1481:7
    #6 0x7bfff3743657 in av1_cdef_mse_calc_frame_mt /builds/worker/checkouts/gecko/third_party/aom/av1/encoder/ethread.c:3362:3
    #7 0x7bfff385cd20 in av1_cdef_search /builds/worker/checkouts/gecko/third_party/aom/av1/encoder/pickcdef.c:880:5
    #8 0x7bfff3703233 in cdef_restoration_frame /builds/worker/checkouts/gecko/third_party/aom/av1/encoder/encoder.c:2770:5
    #9 0x7bfff3703233 in loopfilter_frame /builds/worker/checkouts/gecko/third_party/aom/av1/encoder/encoder.c:2898:3
    #10 0x7bfff3703233 in encode_with_recode_loop_and_filter /builds/worker/checkouts/gecko/third_party/aom/av1/encoder/encoder.c:3724:5
    #11 0x7bfff36dbce0 in encode_frame_to_data_rate /builds/worker/checkouts/gecko/third_party/aom/av1/encoder/encoder.c:4370:9
.
.
.

If you modify the repro-file dimensions array from:

        const dimensions = [
            { width: 16384, height: 17 },
            { width: 64, height: 55572 }
        ];

to

        const dimensions = [
            //{ width: 16384, height: 17 },
            { width: 64, height: 55572 }
        ];

You get a different ASAN-trace:

=================================================================
==3761615==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7bff241c5c80 at pc 0x7bfff35aa89a bp 0x7bff975e15e0 sp 0x7bff975e15d8
READ of size 32 at 0x7bff241c5c80 thread T36
    #0 0x7bfff35aa899 in yy_loadu_256 /builds/worker/checkouts/gecko/third_party/aom/aom_dsp/x86/synonyms_avx2.h:35:10
    #1 0x7bfff35aa899 in sse_w32_avx2 /builds/worker/checkouts/gecko/third_party/aom/aom_dsp/x86/sse_avx2.c:25:24
    #2 0x7bfff35aa899 in aom_sse_avx2 /builds/worker/checkouts/gecko/third_party/aom/aom_dsp/x86/sse_avx2.c:165:9
    #3 0x7bfff3c591dc in get_filt_error /builds/worker/checkouts/gecko/third_party/aom/av1/encoder/pickcdef.c:428:13
    #4 0x7bfff3c591dc in av1_cdef_mse_calc_block /builds/worker/checkouts/gecko/third_party/aom/av1/encoder/pickcdef.c:601:33
    #5 0x7bfff3b442c2 in cdef_filter_block_worker_hook /builds/worker/checkouts/gecko/third_party/aom/av1/encoder/ethread.c:3332:5
...
0x7bff241c5c80 is located 89 bytes after 21414951-byte region [0x7bff22d59800,0x7bff241c5c27)
allocated by thread T36 here:
    #0 0x5555556c3484 in malloc /builds/worker/fetches/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:67:3
    #1 0x7bfff35fdd66 in aom_memalign /builds/worker/checkouts/gecko/third_party/aom/aom_mem/aom_mem.c:59:22
    #2 0x7bfff35fe4b1 in realloc_frame_buffer_aligned /builds/worker/checkouts/gecko/third_party/aom/aom_scale/generic/yv12config.c:132:38
    #3 0x7bfff35fe4b1 in aom_realloc_frame_buffer /builds/worker/checkouts/gecko/third_party/aom/aom_scale/generic/yv12config.c:259:12
    #4 0x7bfff3c60dcf in av1_pick_filter_level /builds/worker/checkouts/gecko/third_party/aom/av1/encoder/picklpf.c:348:9
    #5 0x7bfff3b0304c in loopfilter_frame /builds/worker/checkouts/gecko/third_party/aom/av1/encoder/encoder.c:2874:5
 ...

I have also seen this stack-trace on my fuzzing cluster, I suspect it to be from the same root cause, but haven't been able to get that reproduced, by modifying this repro-file:

==796764==ERROR: AddressSanitizer: stack-use-after-return on address 0x7bff5eed9c80 at pc 0x7bfff31aa767 bp 0x7bffad3e1600 sp 0x7bffad3e15f8
READ of size 16 at 0x7bff5eed9c80 thread T32
SCARINESS: 64 (multi-byte-read-stack-use-after-return)
    #0 0x7bfff31aa766 in xx_loadu_128 /builds/worker/checkouts/gecko/third_party/aom/aom_dsp/x86/synonyms.h:46:10
    #1 0x7bfff31aa766 in aom_sse_avx2 /builds/worker/checkouts/gecko/third_party/aom/aom_dsp/x86/sse_avx2.c:132:30
    #2 0x7bfff385944a in get_filt_error /builds/worker/checkouts/gecko/third_party/aom/av1/encoder/pickcdef.c:461:23
    #3 0x7bfff385944a in av1_cdef_mse_calc_block /builds/worker/checkouts/gecko/third_party/aom/av1/encoder/pickcdef.c:601:33
    #4 0x7bfff37442c2 in cdef_filter_block_worker_hook /builds/worker/checkouts/gecko/third_party/aom/av1/encoder/ethread.c:3332:5
    #5 0x7bfff3204bf5 in execute /builds/worker/checkouts/gecko/third_party/aom/aom_util/aom_thread.c:197:27
...

Address 0x7bff5eed9c80 is located in stack of thread T60 at offset 128 in frame
    #0 0x7bffd181750f in mozilla::CycleCollectedJSRuntime::DescribeGCThing(bool, JS::GCCellPtr, nsCycleCollectionTraversalCallback&) const /builds/worker/checkouts/gecko/xpcom/base/CycleCollectedJSRuntime.cpp:961:0

  This frame has 4 object(s):
    [32, 40) 'aThing'
    [64, 136) 'name' (line 967) <== Memory access at offset 128 partially overflows this variable
    [176, 328) 'chars' (line 982)
    [400, 488) 'fname' (line 984)

Comment 1

[az [:az]]

Tagging Paul on this since I think he'd be the one to ask here.

Comment 2

[Jim Mathies [:jimm]]

Chatted with Ashley, bouncing this over to ChunMin since it's in WebCodecs.

Comment 3

[C.M.Chang[:chunmin]]

Updating libaom doesn't fix the issue. We need to patch somewhere.

Comment 4

[C.M.Chang[:chunmin]]

Attachment: 0001-Add-test-for-CDEF-int16_t-overflow-with-large-frame-.patch

Here is a standalone libaom test reproducing the issue demonstrated in the test page.

Comment 5

[C.M.Chang[:chunmin]]

Attachment: 0002-Fix-int16_t-overflow-in-CDEF-search-for-frames-32768.patch

This is a proposed fix by Claude AI.

My knowledge AV1 encoding is limited and insufficient to judge if this is the right fix right now. Need more investigation to verify this. In the in the meanwhile, maybe it's better to check this issue with someone familiar with libaom's internal mechanism?

TODO

• The fix bypass the get_offset_from_fullmv usage to avoid using FULLPEL_MV, but FULLPEL_MV is fairly common structure used widely in the libaom codebase. Doesn't that imply all call sites s to be revamped? unless get_offset_from_fullmv is misued?
  • Perhaps the calculation here is not for motion vectors so it's okay to skip using get_offset_from_fullmv
  • or replacing it with something with the int types?
• get_offset_from_fullmv was introduced to pickcdef.c in commit 332b730d1e (Oct 2022) by Ranjit Kumar Tulabandu — "Speed-up error calculation in av1_cdef_search()". Need to check the behavior before the changes.
  • key question: Were row and col in int before the commit 332b730d1e ? (updated: YES)

Comment 6

[C.M.Chang[:chunmin]]

Attachment: 0002-Add-tracing-logs-for-CDEF-int16_t-overflow-through-e.patch

This is a patch adding tracing log on top of attachment #9549579 [details] [diff] [review].

Tracing logs added to the libaom encode pipeline to demonstrate the
int16_t overflow in get_filt_error() for frames with dimensions
exceeding 32768 pixels.

Traced code path

#	File	Function	Trace tag
1	aom/src/aom_encoder.c	aom_codec_encode()	TRACE-1
2	av1/av1_cx_iface.c	encoder_encode()	TRACE-2
3	av1/encoder/encoder.c	av1_get_compressed_data()	TRACE-3
4	av1/encoder/encode_strategy.c	av1_encode_strategy()	TRACE-4
5	av1/encoder/encoder.c	av1_encode()	TRACE-5
6	av1/encoder/encoder.c	encode_frame_to_data_rate()	TRACE-6
7	av1/encoder/encoder.c	encode_with_recode_loop_and_filter()	TRACE-7
8	av1/encoder/encoder.c	loopfilter_frame()	TRACE-8
9	av1/encoder/encoder.c	cdef_restoration_frame()	TRACE-9
10	av1/encoder/pickcdef.c	av1_cdef_search()	TRACE-10
11	av1/encoder/pickcdef.c	cdef_mse_calc_frame()	TRACE-11
12	av1/encoder/pickcdef.c	av1_cdef_mse_calc_block()	TRACE-12
13	av1/encoder/pickcdef.c	get_filt_error()	TRACE-13

At location 13, the BUG-SITE-1 through BUG-SITE-4 logs detect the
actual FULLPEL_MV int16_t truncation and print both the correct and
truncated values.

Prerequisites

• CMake >= 3.16
• Clang with -fsanitize=implicit-signed-integer-truncation (Clang 7+)
• nasm or yasm
• Perl

Build

mkdir aom-build && cd aom-build

cmake /path/to/aom \
  -DCMAKE_BUILD_TYPE=Release \
  -DENABLE_TESTS=1 \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_C_FLAGS="-fsanitize=implicit-signed-integer-truncation -fsanitize-recover=implicit-signed-integer-truncation" \
  -DCMAKE_CXX_FLAGS="-fsanitize=implicit-signed-integer-truncation -fsanitize-recover=implicit-signed-integer-truncation" \
  -DCMAKE_EXE_LINKER_FLAGS="-fsanitize=implicit-signed-integer-truncation"

make -j$(nproc) test_libaom

Run tests and capture logs

cd aom-build

# TallFrame (height > 32768): overflow in row
UBSAN_OPTIONS="halt_on_error=0:print_stacktrace=0" \
  ./test_libaom --gtest_filter="EncodeLargeHeight.TallFrame" \
  > cdef-overflow-trace-tall.log 2>&1

# WideFrame (width > 32768): overflow in col
UBSAN_OPTIONS="halt_on_error=0:print_stacktrace=0" \
  ./test_libaom --gtest_filter="EncodeLargeHeight.WideFrame" \
  > cdef-overflow-trace-wide.log 2>&1

halt_on_error=0 allows execution to continue past the UBSan diagnostic
so all trace points and truncation detections are captured in a single
run.

Reading the logs

The trace lines show the 64x32832 (or 32832x64) frame dimensions
propagating through the entire encode pipeline, ending at the overflow:

[TRACE-1]  aom_codec_encode         img=64x32832
[TRACE-2]  encoder_encode           width=64 height=32832
  ...
[TRACE-11] cdef_mse_calc_frame      nvfb=513 nhfb=1
[TRACE-12] av1_cdef_mse_calc_block  fbr=512 fbc=0 -> row=32768 col=0
[TRACE-13] get_filt_error           row=32768 col=0 ref_stride=256
[BUG-SITE-1] FULLPEL_MV truncation: row=32768 -> int16_t=-32768
  correct buf_offset=8388608, truncated buf_offset=-8388608

The key values:

• nvfb=513: 513 superblock rows (32832 / 64)
• fbr=512: last superblock row index
• row=32768: pixel position = 512 * 64 = 32768 (exceeds int16_t max 32767)
• int16_t=-32768: truncated value wraps negative
• buf_offset=-8388608: OOB read 8 MB before the source buffer

Comment 7

[C.M.Chang[:chunmin]]

Attachment: cdef-overflow-trace-tall.log

Comment 8

[C.M.Chang[:chunmin]]

Attachment: cdef-overflow-trace-wide.log

Comment 9

[Jim Mathies [:jimm]]

I've reported this to the libaom maintainers. Should have some updates here shortly.

https://aomedia.issues.chromium.org/issues/489473886

Comment 10

[C.M.Chang[:chunmin]]

I don't have access to the link 🙁

Comment 11

[Jim Mathies [:jimm]]

(In reply to C.M.Chang[:chunmin] from comment #10)

I don't have access to the link 🙁

Sorry, I don't have access controls to open that up. Suffice to say Google is looking at this currently. I sent you some feedback on Matrix.

Comment 12

[C.M.Chang[:chunmin]]

Chrome avoid the overflow by checking the framesize in advance: https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/renderer/modules/webcodecs/video_encoder.cc;l=443-454;drc=3e781fb193605c0dae407c09766c4436d268ef2c

While the max frame size in AV1 spec is 65536 x 65536, their internal check rejects any frame size larger than 32767 x 32767, which is the reason triggering the overflow issue here.

Before we have a fix in libaom, we could probably do the same check in Firefox.

Another workaround is to set enable-cdef=0 on libaom via FFmpeg's av_opt_set in FFmpegVideoEncoder::InitEncoderInternal(). This disables the CDEF filter entirely, so av1_cdef_search() is never called and get_filt_error() is never reached. However, this leads to quality degradation.

Comment 13

[C.M.Chang[:chunmin]]

Attachment: Bug 2020030 - Cap AV1 encoder dimensions

Comment 14

[C.M.Chang[:chunmin]]

Attachment: Bug 2020030 - Add crashtest

Comment 15

[C.M.Chang[:chunmin]]

Attachment: bmo-2020030-analysis.zip

This is my WIP analysis docs. Will travel back to Taiwan next week so I won't be able to have access on my daily working machine. The docs are temporarily uploaded here so I can pick it up from my laptop when needed.

Comment 16

[Jim Mathies [:jimm]]

There is a sec fix coming out in the library as well.

get_filt_error() used FULLPEL_MV (int16_t row/col) to store pixel 
positions within the frame and compute buffer offsets via 
get_offset_from_fullmv(). For frames with height or width exceeding 
32768 pixels, the pixel coordinate overflows int16_t, wrapping to a 
negative value. This causes get_offset_from_fullmv() to compute a 
negative buffer offset, resulting in an out-of-bounds read. 
 
FULLPEL_MV was designed for motion vectors (bounded by search range), 
not for absolute pixel coordinates in large frames. The AV1 spec 
allows frames up to 65536x65536, so these dimensions are within valid 
encoder input range. 
 
Replace FULLPEL_MV + get_offset_from_fullmv() / get_buf_from_fullmv() 
with direct int arithmetic at all four usage sites in get_filt_error(). 
Remove the now-unused mcomp.h include. 
 
Based on original set of patches from Chun-Min Chang. 
 
Bug: 489473886 
Change-Id: I01fb3c981a5a7750fc029b6c98fc73f60614849c

Comment 17

[C.M.Chang[:chunmin]]

The upstream fix is in commit e8b13a0: https://aomedia.googlesource.com/aom/+/e8b13a0936be81657c466f6a99f9a86e0365ef99, which fixes the issue in av1/encoder/pickcdef.c, similar to what attachment 9549585 [details] [diff] [review] suggested (Thanks Claude!).

I'll vendor the fix in bug 1994052 (secretly).

As for uplift, we can try if cherry pick e8b13a0 works. If it doesn't, according to https://aomedia.googlesource.com/aom/+log/e8b13a0936be81657c466f6a99f9a86e0365ef99/av1/encoder/pickcdef.c, we may need to cherry pick all or part of the following commits without updating the whole libaom library:

• e8b13a0 Fix int16_t overflow in CDEF search for frames > 32768 pixels by James Zern · 3 days ago
• 40f676a Improve sf 'fast_intra_tx_type_search' by Anusuya · 7 weeks ago
• 8b67dd8 Adaptive CDEF: do not apply to chroma in CDEF_PICK_FROM_Q mode by Julio Barba · 3 months ago
• 027a236 Adaptive CDEF: code review feedback follow-up by Julio Barba · 3 months ago
• 9a1181f Adaptive CDEF: make decoder-friendly at middle quality levels by Julio Barba · 4 months ago

Comment 18

[C.M.Chang[:chunmin]]

Attachment: Bug 2020030

Comment 19

[C.M.Chang[:chunmin]]

Attachment: (secure)

Original Revision: https://phabricator.services.mozilla.com/D287089

Comment 20

[C.M.Chang[:chunmin]]

Attachment: (secure)

Original Revision: https://phabricator.services.mozilla.com/D287089

Comment 21

[C.M.Chang[:chunmin]]

Attachment: (secure)

Original Revision: https://phabricator.services.mozilla.com/D287089

Comment 22

[C.M.Chang[:chunmin]]

Attachment: (secure)

Original Revision: https://phabricator.services.mozilla.com/D287089

Comment 23

[C.M.Chang[:chunmin]]

Comment on attachment 9551787 [details]
Bug 2020030

Beta/Release Uplift Approval Request

• User impact if declined/Reason for urgency: malicious webpage can a read out-of-bound memory, accessing the data they shouldn't touch
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: 1. Ensure the test page here https://bugzilla.mozilla.org/attachment.cgi?id=9548741 cause a crash, or has ASAN or other memory error logs in the console

2. Apply the fix, and make sure no crash ans no unexpected memory error

• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This patch is cherry-picked from libaom update, which correct the calculation of the offset for the memory address reading, and should not any behavior other than this.
• String changes made/needed: None
• Is Android affected?: No

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration:
• User impact if declined: malicious webpage can a read out-of-bound memory, accessing the data they shouldn't touch
• Fix Landed on Version: 150
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This patch is cherry-picked from libaom update, which correct the calculation of the offset for the memory address reading, and should not any behavior other than this.

Comment 24

[Donal Meehan [:dmeehan]]

Comment on attachment 9551787 [details]
Bug 2020030

Rejecting release uplift request. This is not suitable for inclusion in a dot release since ESR is affected.

Comment 25

[Ryan VanderMeulen [:RyanVM]]

This was fixed on Nightly by bug 1994052.

Comment 26

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9551787 [details]
Bug 2020030

Approved for 149.0b9 and 140.9esr.

Comment 27

[Pulsebot]

https://github.com/mozilla-firefox/firefox/commit/734e1c885a67
https://hg.mozilla.org/releases/mozilla-beta/rev/c2e7c844dfe0

Comment 28

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr140/rev/d58d7a4fb983

Comment 29

[Catalin Sasca, Desktop Test Engineering [:csasca]]

Reproduced the issue on Firefox 150.0a1 (2026-02-27) on Ubuntu 24.0.4 by following the informations provided in Comment 0.

The issue is fixed on Firefox ESR 140.9.0, Firefox 149.0b9 and Firefox 150.0a1 (2026-03-15). Tests were performed on Ubuntu 24.04, Windows 11 and macOS 26.3.1.

Comment 30

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/1b8d8d6f61fe
https://hg.mozilla.org/integration/autoland/rev/08e9ab8a1780
Add crashtest r=media-playback-reviewers,padenot

Comment 31

[Goloman Adrian[:agoloman]]

https://hg.mozilla.org/mozilla-central/rev/08e9ab8a1780