Closed Bug 2044535 Opened 5 months ago Closed 4 months ago

Perma macOS 15 AArch64 Shippable MediaDataEncoderTest.H264EncodeBatch4KAnnexBRealtime | Expected: (GetKeyFrameCount(output)) >= (r.mInputKeyframes), actual: 3 vs 4

Categories

(Core :: Audio/Video, defect, P4)

defect

Tracking

()

RESOLVED FIXED
154 Branch
Tracking Status
firefox-esr153 --- fixed
firefox154 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: chunmin)

References

Details

(Keywords: intermittent-failure)

Attachments

(3 external files)

Filed by: sstanca [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=570310382&repo=mozilla-central&task=JO5onxcMQjOxkfusphBxZg.0
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/JO5onxcMQjOxkfusphBxZg/runs/0/artifacts/public/logs/live_backing.log


[task 2026-06-03T01:06:59.562+00:00] 01:06:59     INFO - TEST-PASS | MediaDataEncoderTest.H264EncodeBatch4KAnnexBRecord | took 841ms
[task 2026-06-03T01:06:59.562+00:00] 01:06:59     INFO - TEST-START | MediaDataEncoderTest.H264EncodeBatch4KAnnexBRealtime
[task 2026-06-03T01:06:59.993+00:00] 01:06:59     INFO - TEST-UNEXPECTED-FAIL | MediaDataEncoderTest.H264EncodeBatch4KAnnexBRealtime | Expected: (GetKeyFrameCount(output)) >= (r.mInputKeyframes), actual: 3 vs 4
[task 2026-06-03T01:06:59.993+00:00] 01:06:59     INFO - 
[task 2026-06-03T01:06:59.993+00:00] 01:06:59     INFO - ./../../../../../checkouts/gecko/dom/media/gtest/TestMediaDataEncoder.cpp:535
[task 2026-06-03T01:07:00.004+00:00] 01:07:00     INFO - TEST-UNEXPECTED-FAIL | MediaDataEncoderTest.H264EncodeBatch4KAnnexBRealtime | expected PASS
[task 2026-06-03T01:07:00.004+00:00] 01:07:00     INFO - TEST-INFO took 442ms
[task 2026-06-03T01:07:00.004+00:00] 01:07:00     INFO - TEST-START | MediaDataEncoderTest.H264EncodeBatch4KAVCCRecord
[task 2026-06-03T01:07:00.845+00:00] 01:07:00     INFO - TEST-PASS | MediaDataEncoderTest.H264EncodeBatch4KAVCCRecord | took 841ms
[task 2026-06-03T01:07:00.845+00:00] 01:07:00     INFO - TEST-START | MediaDataEncoderTest.H264EncodeBatch4KAVCCRealtime
[task 2026-06-03T01:07:01.276+00:00] 01:07:01     INFO - TEST-UNEXPECTED-FAIL | MediaDataEncoderTest.H264EncodeBatch4KAVCCRealtime | Expected: (GetKeyFrameCount(output)) >= (r.mInputKeyframes), actual: 3 vs 4
[task 2026-06-03T01:07:01.276+00:00] 01:07:01     INFO - 
[task 2026-06-03T01:07:01.276+00:00] 01:07:01     INFO - ./../../../../../checkouts/gecko/dom/media/gtest/TestMediaDataEncoder.cpp:535
[task 2026-06-03T01:07:01.287+00:00] 01:07:01     INFO - TEST-UNEXPECTED-FAIL | MediaDataEncoderTest.H264EncodeBatch4KAVCCRealtime | expected PASS
[task 2026-06-03T01:07:01.287+00:00] 01:07:01     INFO - TEST-INFO took 441ms
[task 2026-06-03T01:07:01.287+00:00] 01:07:01     INFO - TEST-START | MediaDataEncoderTest.H264EncodeAfterDrainAnnexBRecord
[task 2026-06-03T01:07:01.556+00:00] 01:07:01     INFO - TEST-PASS | MediaDataEncoderTest.H264EncodeAfterDrainAnnexBRecord | took 269ms
[task 2026-06-03T01:07:01.556+00:00] 01:07:01     INFO - TEST-START | MediaDataEncoderTest.H264EncodeAfterDrainAnnexBRealtime

As https://bugzilla.mozilla.org/show_bug.cgi?id=2044018#c1, here it looks like a simmilar case. Jim, do you know someone who could take care of this one? Thank you!

Severity: S4 → --
Flags: needinfo?(jmathies)
Priority: P5 → --

This is a perma-failing gtest on the macOS 15 AArch64 Shippable config. The test MediaDataEncoderTest.H264EncodeBatch4KAnnexBRealtime (and its AVCC sibling) feeds the VideoToolbox H.264 encoder a 4K batch containing 4 forced input keyframes and asserts the encoder emits at least that many keyframes (EXPECT_GE(GetKeyFrameCount(output), r.mInputKeyframes) at TestMediaDataEncoder.cpp:535), but in Realtime mode at 4K it emits only 3.

The test already relaxes its output-length check for the Realtime+4K case (lines 529-531: "Realtime encoding may drop frames for large frame sizes" -> EXPECT_LE), but the keyframe-count assertion at line 535 was not relaxed the same way. So this is likely one of: (a) a dropped frame happened to carry a forced keyframe, making the strict >= invariant invalid for the lossy Realtime path -- a test fix; or (b) the VideoToolbox encoder on this OS/arch coalesces or ignores a forced keyframe request at 4K -- an AppleVTEncoder fix. Per the see-also bug 2044018 (a sibling perma-fail on the same new macOS 15 AArch64 config, resolved in Networking), a batch of tests appears to have been unhidden together on this worker config rather than a media-specific regression.

Setting S4/P4: this is a CI/test-infra failure with no user-facing playback impact. Taking this for investigation.

Needinfo'ing the test author, Chun-Min Chang (:chunmin), who added the keyframe-count assertion at TestMediaDataEncoder.cpp:535 and the Realtime+4K output-length relaxation just above it (bug 1986544 / bug 1987888): since the Realtime path can legitimately drop a frame that happens to carry a forced keyframe, should the >= keyframe-count invariant at line 535 be relaxed for the Realtime+4K case the same way the length check was (a test fix), or do you expect VideoToolbox to always honor every forced-keyframe request at 4K (which would point at AppleVTEncoder)?

Severity: -- → S4
Flags: needinfo?(jmathies) → needinfo?(cchang)
Priority: -- → P4

This is the remaining issue for the macOS AArch64 GTest suite. Chun-Min, could you take a look?

When the encoder drops a frame in real-time usage, a frame that was requested as
a keyframe is silently lost from the output, so fewer keyframes than requested
are produced -- a more consequential outcome than an ordinary dropped frame, yet
indistinguishable in the log. The drop callback did not identify which frame was
dropped.

Tag each submitted frame that requested a keyframe so the drop callback can
recognise that case, and note it in the existing drop log.

Assignee: nobody → cchang
Status: NEW → ASSIGNED

In real-time usage the encoder may drop frames for large frame sizes. Submitting
a whole batch of frames at once can make it drop the frame that requested a
keyframe, so the output legitimately contains fewer keyframes than were
requested. Dropping frames in real-time mode is permitted by the WebCodecs
specification and matches other browsers, so requiring every requested keyframe
to survive is not a valid expectation on this path.

The output-length check already tolerated dropped frames for real-time 4K, but
the neighbouring keyframe-count check did not. Relax the keyframe-count check the
same way in the batch test, requiring only that some output was produced and that
the stream still starts with a keyframe. The single-frame tests submit with
enough slack that the requested-keyframe frame is not dropped, so they keep the
strict check.

This fixes the permafailing MediaDataEncoderTest.H264EncodeBatch4KAnnexBRealtime
and H264EncodeBatch4KAVCCRealtime gtests on macosx1500-aarch64-shippable/opt.

The single-frame and batch H.264 encode tests duplicated the entire post-encode
validation. Keeping the two copies in sync is what let this bug happen: the
real-time 4K carve-out was applied to the frame-count check but not to the
neighbouring keyframe-count check, in either copy.

Move the shared validation into one helper so the expectations live in a single
place and a carve-out cannot be applied to one copy only. The batch caller
tolerates a dropped keyframe on the real-time 4K path; the single-frame caller
keeps the strict check. No behaviour change, and the helper also guards its
output accesses against an empty result.

Pushed by cchang@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/cc7aa311f91e https://hg.mozilla.org/integration/autoland/rev/0b01156d16e4 Note forced-keyframe frame drops in the macOS video encoder log. r=media-playback-reviewers,padenot https://github.com/mozilla-firefox/firefox/commit/b81be3cc4cf7 https://hg.mozilla.org/integration/autoland/rev/4343be2833c3 Relax the keyframe-count check for real-time 4K batch encoding. r=media-playback-reviewers,padenot https://github.com/mozilla-firefox/firefox/commit/7c6006daaec1 https://hg.mozilla.org/integration/autoland/rev/62077c37e2bd Share the encode-output validation across the H.264 encode tests. r=media-playback-reviewers,padenot
Status: ASSIGNED → RESOLVED
Closed: 4 months ago
Resolution: --- → FIXED
Target Milestone: --- → 154 Branch
See Also: → 2053103
Blocks: 2045883
Flags: needinfo?(cchang)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: