2061529 - GMPVideoFrameType loaded from IPC without range check causes invalid-enum-load (UB) and MOZ_CRASH in GMPVideoEncodedFrameImpl::FrameType()

Status: UNCONFIRMED

(Core :: Audio/Video, defect, P3)

Description

[Tom Ritter [:tjr]]

The encoded-frame metadata struct sent from the GMP (plugin) process to the process driving a video encoder carries the frame type as a raw integer, and the receiving side loads it as an enum with no range check. This is undefined behavior (UBSan invalid-enum-load) and, in non-sanitizer builds, a MOZ_CRASH.

GMPVideoEncodedFrameData declares uint32_t mFrameType on the wire (dom/media/gmp/GMPTypes.ipdlh:49), even though a range-checked serializer for the enum exists (dom/media/gmp/GMPMessageUtils.h:28, ContiguousEnumSerializer<GMPVideoFrameType, kGMPKeyFrame, kGMPVideoFrameInvalid>). The receiving constructors cast the wire value straight to the enum with no check (GMPVideoEncodedFrameImpl.cpp:38 and line 55), and the validation gate CheckFrameData (GMPVideoEncodedFrameImpl.cpp:81) checks only mTemporalLayerId and mSize, never mFrameType. The valid range for GMPVideoFrameType is 0-5 per dom/media/gmp/gmp-api/gmp-video-frame-encoded.h:42-47.

When an out-of-range value arrives, GMPVideoEncodedFrameImpl::FrameType() (line 213) performs the invalid enum load flagged by UBSan. The in-tree consumers are GmpFrameTypeToWebrtcFrameType (WebrtcGmpVideoCodec.cpp:146), whose default: branch is MOZ_CRASH("Unexpected GMPVideoFrameType"), and the WebCodecs keyframe classification at GMPVideoEncoder.cpp:400, which misclassifies keyframes. PGMPVideoEncoder is managed by PGMPContent (PGMPContent.ipdl:18); GMPVideoEncoderParent::RecvEncodedData (GMPVideoEncoderParent.cpp:248) is where the value enters.

The fix is to give mFrameType a ParamTraits<GMPVideoFrameType> on the wire (the range-checked serializer already exists) or to validate the value in CheckFrameData.

Build Info

• Branch: main
• Revision: ca6e9493686b3e5ed1cddb8d3a3974068463df71
• Timestamp: 2026-07-29T00:55:15+00:00

Affected Code

File: dom/media/gmp/GMPTypes.ipdlh, line 44-56

struct GMPVideoEncodedFrameData {
  ...
  uint32_t mFrameType;   // raw integer on the wire; no ParamTraits<GMPVideoFrameType>
  uint32_t mSize;
  int32_t mTemporalLayerId;
  ...
};

File: dom/media/gmp/GMPVideoEncodedFrameImpl.cpp, line 55

mFrameType(static_cast<GMPVideoFrameType>(aFrameData.mFrameType())),  // unchecked cast from IPC value

File: dom/media/gmp/GMPVideoEncodedFrameImpl.cpp, line 81-96

bool GMPVideoEncodedFrameImpl::CheckFrameData(
    const GMPVideoEncodedFrameData& aFrameData, size_t aBufferSize) {
  if (aFrameData.mTemporalLayerId() < -1 ||
      aFrameData.mTemporalLayerId() >= 4) {
    return false;
  }
  static constexpr uint32_t kMaxFrameSize = 100u * 1024u * 1024u;
  return aFrameData.mSize() <= aBufferSize &&
         aFrameData.mSize() <= kMaxFrameSize;
  // NOTE: mFrameType is never validated
}

File: dom/media/gmp/GMPVideoEncodedFrameImpl.cpp, line 213

GMPVideoFrameType GMPVideoEncodedFrameImpl::FrameType() { return mFrameType; }  // UBSan: invalid-enum-load

File: dom/media/webrtc/libwebrtcglue/WebrtcGmpVideoCodec.cpp, line 146-162

static int GmpFrameTypeToWebrtcFrameType(GMPVideoFrameType aIn,
                                         webrtc::VideoFrameType* aOut) {
  switch (aIn) {
    case kGMPKeyFrame: ...
    case kGMPDeltaFrame: ...
    case kGMPSkipFrame: ...
    default:
      MOZ_CRASH("Unexpected GMPVideoFrameType");
  }

Steps to Reproduce

1. Start from a clean checkout at ca6e9493686b3e5ed1cddb8d3a3974068463df71 and apply source_patch.diff: (a) a change to dom/media/gmp/GMPVideoEncoderChild.cpp, guarded by XRE_IsGMPluginProcess(), overwriting the outgoing frameData.mFrameType() with 0x41414141 to inject an out-of-range value; (b) an environment shim in dom/media/gmp/GMPServiceParent.cpp registering the in-tree test codec obj-firefox-asan/dist/bin/gmp-fakeopenh264/1.0 when MOZ_GMP_PATH is unset (replicates the mochitest MOZ_GMP_PATH mechanism because the evaluator cannot set env vars).
2. Build with /app/mozconfigs/mozconfig.linux.asan.fuzzing (ASan+UBSan fuzzing build).
3. Run testcase.html via browser_evaluator with prefs {"media.navigator.streams.fake": true, "media.navigator.permission.disabled": true, "media.peerconnection.video.h264_enabled": true, "browser.dom.window.dump.enabled": true, "fuzzing.enabled": true}. The page sets up a WebRTC loopback, munges the offer SDP to H264-only so the GMP encoder is selected, and streams a fake camera track.
4. Observe the marker 'CLAUDITOR: compromised GMP sending mFrameType=0x41414141' in stderr and a UBSan invalid-enum-load report (load of value 1094795585) at GMPVideoEncodedFrameImpl.cpp:213 in the content process, with stack RecvEncodedData -> WebrtcGmpVideoEncoder::Encoded -> FrameType. Verified reproduced in this run (crashed: true, crashed_parent: false — the content process, not the browser parent).

UBSan/ASAN Report

../../../../dom/media/gmp/GMPVideoEncodedFrameImpl.cpp:213:66: runtime error: load of value 1094795585, which is not a valid value for type 'GMPVideoFrameType'
    #0 0x7b60a84783a3 in mozilla::gmp::GMPVideoEncodedFrameImpl::FrameType() /firefox/obj-firefox-asan/dom/media/gmp/./../../../../dom/media/gmp/GMPVideoEncodedFrameImpl.cpp:213:66
    #1 0x7b60a98534d7 in mozilla::WebrtcGmpVideoEncoder::Encoded(GMPVideoEncodedFrame*, nsTArray<unsigned char> const&) /firefox/obj-firefox-asan/dom/media/webrtc/libwebrtcglue/./../../../../../dom/media/webrtc/libwebrtcglue/WebrtcGmpVideoCodec.cpp:657:48
    #2 0x7b60a847d480 in mozilla::gmp::GMPVideoEncoderParent::RecvEncodedData(mozilla::gmp::GMPVideoEncodedFrameData const&, nsTArray<unsigned char>&&, nsTArray<unsigned char>&&) /firefox/obj-firefox-asan/dom/media/gmp/./../../../../dom/media/gmp/GMPVideoEncoderParent.cpp:256:16
    #3 0x7b60a847d480 in non-virtual thunk to mozilla::gmp::GMPVideoEncoderParent::RecvEncodedData(...) /firefox/obj-firefox-asan/dom/media/gmp/./../../../../dom/media/gmp/GMPVideoEncoderParent.cpp:0:0
    #4 0x7b60a85380c2 in mozilla::gmp::PGMPVideoEncoderParent::OnMessageReceived(IPC::Message const&) /firefox/obj-firefox-asan/dom/media/gmp/./../../../ipc/ipdl/PGMPVideoEncoderParent.cpp:506:52
    #5 0x7b60a84e8157 in mozilla::gmp::PGMPContentParent::OnMessageReceived(IPC::Message const&) /firefox/obj-firefox-asan/dom/media/gmp/./../../../ipc/ipdl/PGMPContentParent.cpp:427:32
    #6 0x7b60a0e2e775 in mozilla::ipc::MessageChannel::DispatchAsyncMessage(...) /firefox/obj-firefox-asan/ipc/glue/./../../../ipc/glue/MessageChannel.cpp:1797:25
    #7 0x7b60a0e2ab4e in mozilla::ipc::MessageChannel::DispatchMessage(...) /firefox/obj-firefox-asan/ipc/glue/./../../../ipc/glue/MessageChannel.cpp:1723:9
    #8 0x7b60a0e2bb07 in mozilla::ipc::MessageChannel::RunMessage(...) /firefox/obj-firefox-asan/ipc/glue/./../../../ipc/glue/MessageChannel.cpp:1512:3
    #9 0x7b60a0e2cfae in mozilla::ipc::MessageChannel::MessageTask::Run() /firefox/obj-firefox-asan/ipc/glue/./../../../ipc/glue/MessageChannel.cpp:1614:14
    #10 0x7b60a0ba17da in nsThread::ProcessNextEvent(bool, bool*) /firefox/obj-firefox-asan/xpcom/threads/./../../../xpcom/threads/nsThread.cpp:1174:16
    ...
SUMMARY: UndefinedBehaviorSanitizer: invalid-enum-load ../../../../dom/media/gmp/GMPVideoEncodedFrameImpl.cpp:213:66

Comment 1

[Tom Ritter [:tjr]]

Attachment: crash_stack.txt

Comment 2

[Tom Ritter [:tjr]]

Attachment: fix.patch

Comment 3

[Tom Ritter [:tjr]]

Attachment: prefs.json

Comment 4

[Tom Ritter [:tjr]]

Attachment: source_patch.diff

Comment 5

[Tom Ritter [:tjr]]

Attachment: testcase.html

Comment 6

[Tom Ritter [:tjr]]

Attachment: prefs.js

Comment 7

[Tom Ritter [:tjr]]

This bug was filed automatically by an AI-powered security testing tool. The analysis, testcase, and any suggested patches were generated by an AI model and have NOT been verified by a human.

Please review all details carefully before triaging. The issue was confirmed by an independent verifier, but for non-ASAN issues, the verifier has been made less strict. Additionally, the root cause analysis and suggested fix may contain errors.

Comment 8

[media-triaging-bot]

Confirmed against current trunk — every cited point in comment 0 checks out.

Analysis

• GMPVideoEncodedFrameData.mFrameType is declared uint32_t in GMPTypes.ipdlh:49, so IPDL never applies the range-checked ParamTraits<GMPVideoFrameType> that already exists at GMPMessageUtils.h:28. The sibling PGMPVideoEncoder::Encode message, which passes GMPVideoFrameType[] directly, is validated — only the struct field is raw.
• Both receiving constructors cast straight to the enum (GMPVideoEncodedFrameImpl.cpp:37 and :54), and CheckFrameData validates only mTemporalLayerId and mSize.
• GMPVideoFrameType is an unscoped enum with no fixed underlying type and enumerators 0–5 (gmp-video-frame-encoded.h:41), so a value outside the 3-bit representable range 0–7 is genuine undefined behaviour at FrameType(), not only a sanitizer artifact.

Impact — deliberately bounded

Every consumer of the deserialized value was enumerated. They are: an equality compare against kGMPKeyFrame in the WebCodecs keyframe classification at GMPVideoEncoder.cpp:400 (worst case: a misclassified bool), and the switch in GmpFrameTypeToWebrtcFrameType whose default: is a MOZ_CRASH. There is no array indexing, no pointer arithmetic and no type discrimination anywhere on the path, so the worst reachable outcome is a controlled abort in the receiving process. The field only originates in the plugin process, so there is no path to it from web content: this is process-boundary input hardening, not a user-facing crash. Crash-stats has no reports matching this signature over the last three months.

Two points that go beyond comment 0 and should shape the patch:

1. Retyping the wire field alone does not remove the abort. ContiguousEnumSerializer's HighBound is exclusive, so the existing serializer admits 0–4 — meaning kGMPGoldenFrame (2) and kGMPAltRefFrame (3) are valid on the wire and still land in GmpFrameTypeToWebrtcFrameType's default: MOZ_CRASH. In other words that abort is already reachable with entirely legal values, so the missing range check is not what gates it. The default: should become a graceful failure rather than an abort.
2. A naive retype risks a compatibility regression. Because the bound is exclusive, kGMPVideoFrameInvalid (5) would become illegal on the wire. No in-tree code sets it, but third-party plugins are opaque — one that legitimately sends it would then hit the MOZ_RELEASE_ASSERT in EnumSerializer::Write on its side and fail deserialization on ours. Validating in CheckFrameData, or using ContiguousEnumSerializerInclusive, avoids that. Worth noting too that the in-tree comment above that assertion calls it "somewhat meaningless" for enums without a fixed underlying type, and it only constrains cooperative senders in any case — receiver-side validation is what matters here.

On classification

No csectype-* keyword is being set here. The accurate class if the security team wants one is csectype-undefined (the UBSan-reported invalid-enum-load); that call is theirs to make rather than something to infer from sibling bugs. Same defect family as bug 2016329 and bug 2016342, though both of those had real out-of-bounds primitives that this one does not.

Nothing further is needed from the reporter.