Closed Bug 2051945 Opened 4 months ago Closed 4 months ago

Unbounded loop in ReenqueueUnplayedAudioDataIfNeeded intermittently leads to OOM

Categories

(Core :: Audio/Video: Playback, defect)

defect

Tracking

()

RESOLVED FIXED
154 Branch
Tracking Status
firefox154 --- fixed

People

(Reporter: yannis, Assigned: yannis)

References

(Blocks 1 open bug)

Details

Crash Data

Attachments

(1 external file)

Seen in CI failures that get incorrectly reported to bug 2049921.

Trees: autoland (18), mozilla-central (2)
Platform: windows11-32-25h2 (20)
Typical failure log:

GECKO(2664) | out of memory: 0x0000000033300000 bytes requested
GECKO(2664) | [7116] Hit MOZ_CRASH(out of memory: 0x0000000033300000 bytes requested) at checkouts\gecko\memory\mozalloc\mozalloc_abort.cpp:32
 Initializing stack-fixing for the first stack frame, this may take a while...
GECKO(2664) | #01: mozalloc_abort(char const* const) [git:github.com/mozilla-firefox/firefox:memory/mozalloc/mozalloc_abort.cpp:629864ddf550a09ce5eb9bffc50f4452353b7f4f:32]
GECKO(2664) | #02: mozalloc_handle_oom [D:\task_178286284827486\build\application\firefox\mozglue.dll + 0x291f9]
GECKO(2664) | #03: moz_xrealloc [D:\task_178286284827486\build\application\firefox\mozglue.dll + 0x292ea]
GECKO(2664) | #04: nsTArray_base<nsTArray_RelocateUsingMemutils>::EnsureCapacityImpl<nsTArrayInfallibleAllocator>(unsigned int, unsigned int) [git:github.com/mozilla-firefox/firefox:xpcom/ds/nsTArray.h:629864ddf550a09ce5eb9bffc50f4452353b7f4f:3183]
GECKO(2664) | #05: mozilla::AudioSink::ReenqueueUnplayedAudioDataIfNeeded() [git:github.com/mozilla-firefox/firefox:dom/media/mediasink/AudioSink.cpp:629864ddf550a09ce5eb9bffc50f4452353b7f4f:253]
GECKO(2664) | #06: mozilla::AudioSink::ShutDown() [git:github.com/mozilla-firefox/firefox:dom/media/mediasink/AudioSink.cpp:629864ddf550a09ce5eb9bffc50f4452353b7f4f:295]
GECKO(2664) | #07: mozilla::AudioSinkWrapper::ShutDownAudioSink() [git:github.com/mozilla-firefox/firefox:dom/media/mediasink/AudioSinkWrapper.cpp:629864ddf550a09ce5eb9bffc50f4452353b7f4f:356]
GECKO(2664) | #08: mozilla::AudioSinkWrapper::Stop() [git:github.com/mozilla-firefox/firefox:dom/media/mediasink/AudioSinkWrapper.cpp:629864ddf550a09ce5eb9bffc50f4452353b7f4f:532]
GECKO(2664) | #09: mozilla::VideoSink::Stop() [git:github.com/mozilla-firefox/firefox:dom/media/mediasink/VideoSink.cpp:629864ddf550a09ce5eb9bffc50f4452353b7f4f:284]
GECKO(2664) | #10: mozilla::MediaDecoderStateMachine::StopMediaSink() [git:github.com/mozilla-firefox/firefox:dom/media/MediaDecoderStateMachine.cpp:629864ddf550a09ce5eb9bffc50f4452353b7f4f:3912]
GECKO(2664) | #11: mozilla::MediaDecoderStateMachine::ShutdownState::Enter() [git:github.com/mozilla-firefox/firefox:dom/media/MediaDecoderStateMachine.cpp:629864ddf550a09ce5eb9bffc50f4452353b7f4f:3358]
GECKO(2664) | #12: mozilla::MediaDecoderStateMachine::StateObject::SetState<mozilla::MediaDecoderStateMachine::ShutdownState>() [git:github.com/mozilla-firefox/firefox:dom/media/MediaDecoderStateMachine.cpp:629864ddf550a09ce5eb9bffc50f4452353b7f4f:269]
GECKO(2664) | #13: mozilla::MediaDecoderStateMachine::StateObject::HandleShutdown() [git:github.com/mozilla-firefox/firefox:dom/media/MediaDecoderStateMachine.cpp:629864ddf550a09ce5eb9bffc50f4452353b7f4f:2857]
GECKO(2664) | #14: mozilla::MediaDecoderStateMachine::Shutdown() [git:github.com/mozilla-firefox/firefox:dom/media/MediaDecoderStateMachine.cpp:629864ddf550a09ce5eb9bffc50f4452353b7f4f:3765]
GECKO(2664) | #15: mozilla::detail::ProxyFunctionRunnable<`lambda at .\..\..\..\..\checkouts\gecko\dom\media\MediaDecoderStateMachineBase.cpp:142:7',mozilla::MozPromise<bool,bool,0> >::Run() [git:github.com/mozilla-firefox/firefox:xpcom/threads/MozPromise.h:629864ddf550a09ce5eb9bffc50f4452353b7f4f:1836]
GECKO(2664) | #16: mozilla::AutoTaskDispatcher::TaskGroupRunnable::Run() [git:github.com/mozilla-firefox/firefox:xpcom/threads/TaskDispatcher.h:629864ddf550a09ce5eb9bffc50f4452353b7f4f:235]
...

Failing jobs:

Likely root cause: if typicalPacketFrameCount is zero, the unbounded loop below from ReenqueueUnplayedAudioDataIfNeeded loops forever, allocating more and more memory each time the data structure needs to grow, and eventually we OOM.

  // Extract all audio data from the ring buffer, we can only read the data from
  // the most recent, so we reenqueue the data, packetized, in a temporary
  // array.
  while (framesRemaining) {
    uint32_t packetFrameCount =
        std::min(framesRemaining, typicalPacketFrameCount);
    framesRemaining -= packetFrameCount;

    int packetSampleCount = packetFrameCount * channelCount;
    AlignedAudioBuffer packetData(packetSampleCount);
    DebugOnly<int> samplesRead =
        mProcessedSPSCQueue->Dequeue(packetData.Data(), packetSampleCount);
    MOZ_ASSERT(samplesRead == packetSampleCount);

    packetsToReenqueue.AppendElement(packetData);
  }

Note that the same pattern exists in PushFakeAudioDataIfNeeded.

The signatures I found on crash-stats that would match with this issue are low volume, so this is mostly a CI concern as far as I can tell. It seems that our tests can make mFrames reach zero but that this situation is unlikely in the wild. In Claude's words:

Verdict on "test-only": No. The reenqueue-on-teardown path itself is ordinary (it fires on pause / mute / volume change / seek / tab close whenever the ring buffer still holds processed audio — e.g. the OnMuted -> SetVolume -> VolumeChanged stack seen in the dumps), and the 0-frame trigger is a genuine SetTrimWindow outcome. What makes it predominantly a test finding is that hitting all conditions at once (trimmed 0-frame front packet and a non-empty ring buffer and sink teardown) is rare; the ipc/glue/test/browser suite hammers exactly this by looping a tiny clip and repeatedly stopping/muting/crashing decoders. A real user with looping/gapless/MSE content and an ill-timed pause could still hit it.

Assignee: nobody → yjuglaret

Both AudioSink::ReenqueueUnplayedAudioDataIfNeeded and
MediaDecoderStateMachine::PushFakeAudioDataIfNeeded derive their packet
size from the front/back audio packet's frame count. A packet can
legitimately report zero frames (e.g. fully trimmed via SetTrimWindow),
which makes packetFrameCount zero, so the loop never decrements its
remaining-frames counter and spins forever, allocating until OOM.

Fall back to the default 1024-frame packet size when the reference
packet has no frames.

Severity: -- → S3
Pushed by yjuglaret@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/a436b9bcc050 https://hg.mozilla.org/integration/autoland/rev/e328deca0ac2 Avoid unbounded packetizing loop on zero-frame audio packets. r=media-playback-reviewers,padenot
Status: NEW → RESOLVED
Closed: 4 months ago
Resolution: --- → FIXED
Target Milestone: --- → 154 Branch
QA Whiteboard: [qa-triage-done-c155/b154]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: