2029422 - Wild free of uninitialized pointer in av1_cdef_dealloc_data after partial allocation failure

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, defect)

Description

[Bugmon [:jkratzer for issues]]

When the libaom AV1 encoder runs CDEF search for the first time, it allocates a CdefSearchCtx struct with aom_malloc() (raw malloc, not zero-initialized) at pickcdef.c:875-877. The struct's mse[0], mse[1], and sb_index pointer fields therefore contain uninitialized heap garbage. cdef_alloc_data() then performs three sequential CHECK_MEM_ERROR allocations; CHECK_MEM_ERROR assigns the result to the lvalue and, on NULL, calls aom_internal_error() which longjmps out of the function. If the first allocation (sb_index) succeeds but the second (mse[0]) fails, mse[0] is set to NULL but mse[1] is never assigned and retains its uninitialized value. The encode call returns AOM_CODEC_MEM_ERROR via the longjmp target in av1_get_compressed_data().

When the encoder is later destroyed (e.g., VideoEncoder.close() from JavaScript), av1_remove_compressor() → dealloc_compressor_data() → av1_cdef_dealloc_data() unconditionally calls aom_free(cdef_search_ctx->mse[1]). aom_free() reads a stored header pointer from ((size_t)memblk - 1) and passes that to free(). Under ASAN (malloc_fill_byte=0xe4) the uninitialized value is the non-canonical address 0xe4e4e4e4e4e4e4e4 and the header read SEGVs immediately. In a production build the value is whatever previously occupied that heap chunk; an attacker who grooms the heap (allocate-then-free a same-size chunk with a controlled value at the mse[1] offset) gains an arbitrary-free primitive that can be escalated to use-after-free.

The vulnerable code path is reachable from web content via the WebCodecs VideoEncoder API with default Firefox settings: latencyMode:'quality' selects FFmpeg's default usage=0 (AOM_USAGE_GOOD_QUALITY), and in GOOD mode set_good_speed_feature_framesize_dependent() never sets CDEF_PICK_FROM_Q (it tops out at CDEF_FAST_SEARCH_LVL4 for speed≥6), so av1_cdef_search() proceeds past the early return into the vulnerable allocation path even at the default cpu-used=7. The trigger condition — aom_malloc returning NULL — occurs naturally on OOM (Firefox sets allocator_may_return_null=1) or when the requested size exceeds AOM_MAX_ALLOCABLE_MEMORY.

Build Info

• Branch: main
• Revision: 6164ea4bacaeaed1f617c11911df7fc32f2e6ec2
• Timestamp: 2026-04-02T19:36:24+00:00

Affected Code

File: third_party/aom/av1/encoder/pickcdef.c, line 875-877

  if (!cpi->cdef_search_ctx)
    CHECK_MEM_ERROR(cm, cpi->cdef_search_ctx,
                    aom_malloc(sizeof(*cpi->cdef_search_ctx)));

File: third_party/aom/aom_mem/aom_mem.c, line 55-67

void *aom_memalign(size_t align, size_t size) {
  void *x = NULL;
  if (!check_size_argument_overflow(1, size, align)) return NULL;
  const size_t aligned_size = size + GetAllocationPaddingSize(align);
  void *const addr = malloc(aligned_size);
  if (addr) {
    x = aom_align_addr((unsigned char *)addr + ADDRESS_STORAGE_SIZE, align);
    SetActualMallocAddress(x, addr);
  }
  return x;
}

void *aom_malloc(size_t size) { return aom_memalign(DEFAULT_ALIGNMENT, size); }

File: third_party/aom/av1/encoder/pickcdef.c, line 640-651

static void cdef_alloc_data(AV1_COMMON *cm, CdefSearchCtx *cdef_search_ctx) {
  const int nvfb = cdef_search_ctx->nvfb;
  const int nhfb = cdef_search_ctx->nhfb;
  CHECK_MEM_ERROR(
      cm, cdef_search_ctx->sb_index,
      aom_malloc(nvfb * nhfb * sizeof(cdef_search_ctx->sb_index[0])));
  cdef_search_ctx->sb_count = 0;
  CHECK_MEM_ERROR(cm, cdef_search_ctx->mse[0],
                  aom_malloc(sizeof(**cdef_search_ctx->mse) * nvfb * nhfb));
  CHECK_MEM_ERROR(cm, cdef_search_ctx->mse[1],
                  aom_malloc(sizeof(**cdef_search_ctx->mse) * nvfb * nhfb));
}

File: third_party/aom/aom_mem/aom_mem.h, line 70-77

#define AOM_CHECK_MEM_ERROR(error_info, lval, expr)       \
  do {                                                    \
    assert((error_info)->setjmp);                         \
    lval = (expr);                                        \
    if (!lval)                                            \
      aom_internal_error(error_info, AOM_CODEC_MEM_ERROR, \
                         "Failed to allocate " #lval);    \
  } while (0)

File: third_party/aom/aom/src/aom_codec.c, line 159-168

void aom_internal_error(struct aom_internal_error_info *info,
                        aom_codec_err_t error, const char *fmt, ...) {
  va_list ap;

  va_start(ap, fmt);
  set_error(info, error, fmt, ap);
  va_end(ap);

  if (info->setjmp) longjmp(info->jmp, info->error_code);
}

File: third_party/aom/av1/encoder/encoder_alloc.h, line 291-295

  // This call ensures that CDEF search context buffers are deallocated in case
  // of an error during cdef search.
  av1_cdef_dealloc_data(cpi->cdef_search_ctx);
  aom_free(cpi->cdef_search_ctx);
  cpi->cdef_search_ctx = NULL;

File: third_party/aom/av1/encoder/pickcdef.c, line 659-668

void av1_cdef_dealloc_data(CdefSearchCtx *cdef_search_ctx) {
  if (cdef_search_ctx) {
    aom_free(cdef_search_ctx->mse[0]);
    cdef_search_ctx->mse[0] = NULL;
    aom_free(cdef_search_ctx->mse[1]);
    cdef_search_ctx->mse[1] = NULL;
    aom_free(cdef_search_ctx->sb_index);
    cdef_search_ctx->sb_index = NULL;
  }
}

File: third_party/aom/aom_mem/aom_mem.c, line 50-82

static void *GetActualMallocAddress(void *const mem) {
  const size_t *const malloc_addr_location = GetMallocAddressLocation(mem);
  return (void *)(*malloc_addr_location);
}
...
void aom_free(void *memblk) {
  if (memblk) {
    void *addr = GetActualMallocAddress(memblk);
    free(addr);
  }
}

The CdefSearchCtx struct (pickcdef.h:158,163) contains raw pointer fields mse[0], mse[1], and sb_index. When the struct is allocated with aom_malloc() (which calls plain malloc() with no zero-initialization), these pointers contain whatever was previously in that heap region. cdef_alloc_data() initializes them sequentially via CHECK_MEM_ERROR, but CHECK_MEM_ERROR longjmps out immediately on NULL — so a failure at the second allocation leaves mse[1] never written. The cleanup path in dealloc_compressor_data() then passes that uninitialized pointer to aom_free(), which dereferences it (reads the stored header at memblk-8) and calls free() on the result. The same hazard applies if the very first allocation (sb_index) fails: in that case both mse[0] and mse[1] remain uninitialized and two wild frees occur. This bug was introduced by upstream libaom commit 4d28dff7255a which moved cdef_search_ctx from stack to heap and changed the alloc error handling from check-all-then-free to longjmp-on-first-failure.

Exploit Chain

1. Attacker-controlled web page creates a WebCodecs VideoEncoder configured for AV1 (codec='av01.0.04M.08') with latencyMode='quality', mapping to FFmpeg's default usage=0 (AOM_USAGE_GOOD_QUALITY).
2. Optionally: groom the heap by allocating and freeing chunks of size sizeof(CdefSearchCtx) so that the next malloc of that size returns a chunk containing attacker-controlled bytes at the offset of mse[1].
3. Attacker drives the process toward memory exhaustion (e.g., by allocating large ArrayBuffers, ImageData, or many encoder instances) so that the second aom_malloc() in cdef_alloc_data() returns NULL while the first succeeds.
4. Page calls encoder.encode(frame). On the first frame, av1_cdef_search() allocates cpi->cdef_search_ctx via aom_malloc() — uninitialized memory containing the groomed value at mse[1].
5. cdef_alloc_data() runs: sb_index allocation succeeds, mse[0] allocation returns NULL → CHECK_MEM_ERROR assigns mse[0]=NULL and longjmps to the setjmp handler in av1_get_compressed_data() (encoder.c:5311). mse[1] is never written.
6. The encode call returns AOM_CODEC_MEM_ERROR. The encoder error callback fires in JavaScript. cpi->cdef_search_ctx remains live with mse[1] still containing the uninitialized/groomed value.
7. Page calls encoder.close() (or the encoder is GC'd). FFmpegDataEncoder::ProcessShutdown() → avcodec_free_context() → aom_codec_destroy() → av1_remove_compressor() → dealloc_compressor_data() → av1_cdef_dealloc_data(cpi->cdef_search_ctx).
8. av1_cdef_dealloc_data() calls aom_free(mse[0]=NULL) which is a no-op, then aom_free(mse[1]=<groomed value>). aom_free reads ((size_t)mse[1] - 1) and passes that to free() — an arbitrary-free primitive.
9. The arbitrary free can be aimed at a live object to create a use-after-free, leading to memory corruption and potential code execution in the content process.

Steps to Reproduce

1. Apply the fault-injection patch to third_party/aom/av1/encoder/pickcdef.c, which replaces the second aom_malloc() in cdef_alloc_data() with NULL to deterministically simulate the OOM condition that aom_malloc returns naturally.
2. Build Firefox: MOZCONFIG=<asan-mozconfig> ./mach build
3. Load test.html which creates a VideoEncoder with codec='av01.0.04M.08', latencyMode='quality', encodes a single 256x256 I420 keyframe, awaits flush(), then calls close().
4. Observe SEGV in aom_free called from av1_cdef_dealloc_data with rdi=0xe4e4e4e4e4e4e4e4 (ASAN's malloc_fill_byte poison value, proving the pointer was never initialized).
5. Note: no Firefox prefs are required — the vulnerable path is reached with the default media.ffmpeg.encoder.cpu-used=7 because GOOD-quality mode never sets CDEF_PICK_FROM_Q.

Security Impact

• Severity: Moderate
• Attacker capability: Wild free of an uninitialized pointer in the content process. With heap grooming (allocate-then-free a same-size chunk so the CdefSearchCtx allocation reuses it with a controlled value at the mse[1] offset), the attacker controls the argument to aom_free(), which then reads a header pointer from attacker_value-8 and passes it to free(). This yields an arbitrary-free primitive that can be escalated to use-after-free and code execution within the content process. Also exploitable in any other application using libaom encoding under memory pressure.
• Preconditions: Web page must be able to drive aom_malloc() to return NULL for the second allocation in cdef_alloc_data() while the first succeeds. aom_malloc returns NULL when the system allocator fails (Firefox sets allocator_may_return_null=1; mozjemalloc returns NULL on exhaustion) or when the requested size exceeds AOM_MAX_ALLOCABLE_MEMORY. The attacker controls frame dimensions (larger frames → larger mse[] allocations) and can apply memory pressure via JavaScript allocations, but precise triggering of the OOM at exactly this allocation is not demonstrated, so per Mozilla guidelines this is Moderate (memory safety bug constrained by OOM). The vulnerable code path is reachable from web content with the default Firefox configuration — no prefs required.

ASAN Report

==5235==ERROR: AddressSanitizer: SEGV on unknown address (pc 0x74a3df30b8e2 bp 0x74a391a84b70 sp 0x74a391a84b70 T20)
==5235==The signal is caused by a READ memory access.
==5235==Hint: this fault was caused by a dereference of a high value address (see register values below).  Disassemble the provided pc to learn which register was used.
    #0 0x74a3df30b8e2 in GetActualMallocAddress /firefox/third_party/aom/aom_mem/aom_mem.c:52:19
    #1 0x74a3df30b8e2 in aom_free /firefox/third_party/aom/aom_mem/aom_mem.c:79:18
    #2 0x74a3df94f398 in av1_cdef_dealloc_data /firefox/third_party/aom/av1/encoder/pickcdef.c:668:5
    #3 0x74a3df7bc61c in dealloc_compressor_data /firefox/third_party/aom/av1/encoder/encoder_alloc.h:293:3
    #4 0x74a3df7bc61c in av1_remove_compressor /firefox/third_party/aom/av1/encoder/encoder.c:1864:3
    #5 0x74a3df31aa2f in av1_destroy_context_and_bufferpool /firefox/third_party/aom/av1/av1_cx_iface.c:3125:3
    #6 0x74a3df31aa2f in encoder_destroy /firefox/third_party/aom/av1/av1_cx_iface.c:3189:7
    #7 0x74a3defb7e81 in aom_codec_destroy /firefox/third_party/aom/aom/src/aom_codec.c:76:3
    #8 0x74a3920d6524 in aom_free /firefox/media/ffvpx/libavcodec/libaomenc.c:449:5
    #9 0x74a391f3512a in ff_codec_close /firefox/media/ffvpx/libavcodec/avcodec.c:445:13
    #10 0x74a39212c4ea in avcodec_free_context /firefox/media/ffvpx/libavcodec/options.c:171:5
    #11 0x74a3c099d95b in mozilla::FFmpegDataEncoder<46465650>::ReleaseCodecContext() /firefox/dom/media/platforms/ffmpeg/FFmpegDataEncoder.cpp:431:3
    #12 0x74a3c099d653 in mozilla::FFmpegDataEncoder<46465650>::ShutdownInternal() /firefox/dom/media/platforms/ffmpeg/FFmpegDataEncoder.cpp:378:5
    #13 0x74a3c099c5b8 in mozilla::FFmpegDataEncoder<46465650>::ProcessShutdown() /firefox/dom/media/platforms/ffmpeg/FFmpegDataEncoder.cpp:346:3

==5235==Register values:
rax = 0x1c9c9c9c9c9c9c9b  rbx = 0x000076f3e2cf3110  rcx = 0x000075f3e2d11070  rdx = 0xe4e4e4e4e4e4e4dc
rdi = 0xe4e4e4e4e4e4e4e4  rsi = 0xe4e4e4e4e4e4e4e4  rbp = 0x000074a391a84b70  rsp = 0x000074a391a84b70
 r8 = 0x00000000000001e7   r9 = 0x000076e3e2d80000  r10 = 0x00007fffffffff01  r11 = 0x0000000000000001
r12 = 0x000076f3e2cf5020  r13 = 0x00000ede7c59ea04  r14 = 0x000076f3e2cf5018  r15 = 0x000076f3e2cf5020
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/firefox/obj-firefox-asan/dist/bin/libgkcodecs.so+0xe8f8e2) (BuildId: 25a7f2f89be744179cd005391e108954)
Thread T20 created by T0 (Isolated Web Co)
==5235==ABORTING

Comment 1

[Bugmon [:jkratzer for issues]]

Attachment: test.html

Comment 2

[Bugmon [:jkratzer for issues]]

Attachment: source_patch.diff

Comment 3

[Bugmon [:jkratzer for issues]]

Attachment: fix.patch

Comment 4

[Bugmon [:jkratzer for issues]]

Attachment: crash_stack.txt

Comment 5

[Bugmon [:jkratzer for issues]]

Attachment: README.txt

Comment 7

[John Lin [:jhlin][:jolin]]

This is a third-party library (libaom) bug,

To summarize, cdef_search_ctx is heap-allocated via aom_malloc() and pointers in it are not initialized. When encountering allocation error, the clean-up code could try to free garbage memory locations and cause problems. It should be allocated/initialized with aom_calloc() instead.

The issue still exists in current upstream code and I've filed a security issue there (https://aomedia.issues.chromium.org/issues/501471698). Since this library is vendored, we could fixed it in tree for now and pick the upstream fix later.

Comment 8

[John Lin [:jhlin][:jolin]]

Attachment: (secure)

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/40fded5b5a05
https://hg.mozilla.org/integration/autoland/rev/aafb7ef5b683
local patch to fix libaom issue. r=media-playback-reviewers,alwu

Comment 10

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/aafb7ef5b683

Comment 11

[BugBot [:suhaib / :marco]]

The patch landed in nightly and beta is affected.
:jhlin, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
  • See https://wiki.mozilla.org/Release_Management/Requesting_an_Uplift for documentation on how to request an uplift.
• If no, please set status-firefox150 to wontfix.

For more information, please visit BugBot documentation.

Comment 12

[Ryan VanderMeulen [:RyanVM]]

Doesn't need an uplift for 150 at this point, but an ESR140 uplift request would still be good.

Comment 13

[John Lin [:jhlin][:jolin]]

Attachment: (secure)

Original Revision: https://phabricator.services.mozilla.com/D293707

Comment 14

[Phabricator Automation]

firefox-esr140 Uplift Approval Request

• User impact if declined/Reason for urgency: Potential crash when encoding AV1 video.
• Code covered by automated testing?: no
• Fix verified in Nightly?: yes
• Needs manual QE testing?: no
• Steps to reproduce for manual QE testing:
• Risk associated with taking this patch: low
• Explanation of risk level: The change is simple.
• String changes made/needed?: N/A
• Is Android affected?: yes

Comment 15

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr140/rev/50ab5d78096f