Open Bug 2087196 Opened 19 days ago Updated 5 days ago

WebCodecs decoders and encoders can deliver outputs queued before reset()

Categories

(Core :: Audio/Video: Web Codecs, defect, P3)

defect

Tracking

()

ASSIGNED

People

(Reporter: nicfio, Assigned: nicfio, NeedInfo)

References

(Blocks 1 open bug)

Details

Attachments

(2 external files, 2 obsolete files)

After reset() (or a configure() that implies one), DecoderTemplate and EncoderTemplate can still invoke the output callback with frames or chunks that were scheduled before the reset. This happens for example when the page reconfigures the codec from inside the output callback. The specification requires that outputs pending at the time of the reset are discarded. The code is shared by all platforms; it was found while running the WebCodecs WPT on Android.

Steps to reproduce:

  1. Configure an AudioEncoder (or a VideoDecoder) and queue many inputs.
  2. In the first output callback, call reset() and configure() again, then encode new input.
  3. Outputs produced from the old input are still delivered after the reset.

Proposed fix (patches will follow on Phabricator):

  • Part 1: on every reset, discard the pending outputs and reset the marker tracker, in both templates. The WPT subtest "Verify reset() suppresses outputs" now passes with openh264, so its expected failure is removed.
  • Part 2: a new WPT, audio-encoder-reset-during-output, which checks that no output produced before reset() is delivered after the encoder was reconfigured in the output callback. Without Part 1 it fails (19 stale packets are delivered after the reset); with Part 1 it passes.

See also bug 2026835 (close() and pending output callbacks), which touches the same code.

AI assistance: these patches were developed with AI coding assistants: the code, its review and the test runs were done by AI agents under my direction, on my own devices (an Android emulator and a Samsung Galaxy S23+). I take responsibility for submitting them and will answer review comments, with the same tools.

I'm attaching the two patches as files because I can't use Phabricator at the moment. They are git format-patch output and apply in order on top of the patches of bug 2087195 (Part 1 touches the same files). If someone could import them into Phabricator, or tell me the preferred way to proceed, I'd be grateful. Suggested reviewers: #media-playback-reviewers.

Thank you for the report and the patches.

Analysis

The current tree shows the problem in both templates. In DecoderTemplate.cpp, OutputDecodedData has only a MOZ_ASSERT on the state and no check at runtime. Its loop calls the output callback for each frame of the batch, also after the callback itself calls reset(). ResetInternal keeps no reset counter. Also, configure() sets the state back to Configured at once, so a state check cannot detect reset() followed by configure().

In EncoderTemplate.cpp, the output loop only checks mActiveConfig. reset() clears mActiveConfig, but configure() sets it again at once. Thus the remaining outputs of the batch pass the check. This probably explains the 19 stale packets in the Part 2 test, if the encoder returns all 20 packets in one batch.

The WebCodecs algorithm text is not clear on this point. The open spec issue w3c/webcodecs#755 discusses it, and the code comment in EncoderTemplate.cpp says that the editors agreed that no output comes after a reset. The WPT subtest "Verify reset() suppresses outputs" also expects this. It fails on Linux with openh264, as videoDecoder-codec-specific.https.any.js.ini shows.

Part 1 (attachment 9671633 [details] [diff] [review]) adds a reset counter. Each output task captures the counter, and the task checks it before the output loop and after each callback. This addresses the cause in both templates. One path is still open: the "Destroy + recreate encoder" task in DrainAndReconfigure checks only mState (EncoderTemplate.cpp). After reset() and configure() in an output callback, this task can destroy the new encoder and configure it again with the old configuration. The same counter check can protect this task.

Part 1 applies on top of the patches of bug 2087195. It also changes the same function as the patch of bug 2026835, so one of the two patches needs a rebase. This bug blocks the WebCodecs meta bug 1746557.

Next steps

Phabricator is the usual path for review. The page How to submit a patch shows how to submit with moz-phab.


If you'd like to provide feedback on this comment, please use the 👍 or 👎 reaction.

Blocks: webcodecs
Severity: -- → S3
Priority: -- → P3
See Also: → 2026835, 2087195

After reset(), DecoderTemplate and EncoderTemplate could still deliver
outputs scheduled before the reset, for example when the application
reconfigures from inside the output callback. Discard the pending
outputs on every reset, as the specification requires, and clear the
profiler marker tracker so that it does not keep entries from before the
reset.

The task that recreates the encoder after a drain for reconfiguration
checks the reset counter too, so that a reset() and configure() from an
output callback are not undone by the old configuration.

Part 2 adds a WPT test for the AudioEncoder case.

Assignee: nobody → nicfio
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true

Check that no output produced before reset() is delivered after the
encoder was reconfigured in the output callback.

Opus encoding fails on Android (isConfigSupported() reports it as
supported but configure() errors, as in the existing audio-encoder
tests), so the test is expected to FAIL there.

Comment on attachment 9671633 [details] [diff] [review]
Part 1: Drop outputs already queued when WebCodecs reset() is called

Thanks for the review. You're right about the "Destroy + recreate encoder" task in DrainAndReconfigure. I've updated Part 1 so that this task also captures the reset counter, and returns if a reset happened after the drain. A reset before the drain completes already disconnects mDrainAfterReconfigureRequest, and a reset alone was already caught by the mState check. The remaining case was a reset() followed by configure() after the drain completed but before this task ran, for example from the output callback of the drained data.

I didn't add a test for this path. It needs an encoder that cannot reconfigure in place, so that configure() drains it, and a reset() and configure() at exactly that moment. I couldn't make that deterministic across encoders.

I also reworked Part 2 after a second look. It now tells old outputs from new ones by timestamp: the first input covers 0–400 ms and the input after the reset starts at 1 s, so any chunk under 500 ms delivered after the reset is stale. On Android, Opus encoding fails at configure() even though isConfigSupported() says it is supported, as in the existing audio-encoder tests, so the new test is marked as expected FAIL there in its meta file. On Linux desktop, without Part 1 the new test fails with 19 stale chunks delivered after the reset; with Part 1 it passes.

The patches are now on Phabricator: Part 1 is D338352 and Part 2 is D338353. They apply on top of the patches of bug 2087195. Attachments 9671633 and 9671634 are obsolete.

About bug 2026835: D335529 adds a state check in DecoderTemplate::OutputDecodedData for the close() case. Part 1 here covers that case too, because close() goes through ResetInternal() and increments the reset counter, so the output is dropped before the assertion. It also covers reset() followed by configure(). The two patches conflict in that function. Whichever lands first, I'm happy to rebase Part 1, and the crashtest of D335529 is worth keeping in any case.

A correction to comment 0: I wrote that the "Verify reset() suppresses outputs" subtest now passes with openh264. That was wrong. I've now run it with the real openh264 plugin (2.6.0) on Linux, and it still fails, with and without these patches ("expected 5 but got 4"), so the cause is a different one. Part 1 no longer touches its expectations, and the commit message no longer makes that claim.

Attachment #9671633 - Attachment is obsolete: true
Attachment #9671634 - Attachment is obsolete: true

Hi Paul, could you take a look at D338352 and D338353, or tell me who should review them? They have been in the media-playback-reviewers queue for about two weeks, with no reviewer assigned. This fix does not depend on bug 2087195 and applies cleanly on current mozilla-central. Bug 2026835 (D335529) touches the same code in OutputDecodedData; if it lands first, I'll rebase.

Flags: needinfo?(padenot)
See Also: → 2014892
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: