1495251 - Resource leak in audiosrv triggered by WASAPI backend

Status: NEW

(Core :: Audio/Video: cubeb, defect, P3)

Description

[János Tóth F.]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0
Build ID: 20180920131237

Steps to reproduce:

Whenever some application uses WASAPI exclusive mode for audio output while Firefox is running the total commited virtual memory size (as indicated by the Windows Task Manager) continuously increases until the it's system wide limit is reached (= amount of physical memory + pagefile size). This leaked commit size is not freed after closing these applications (both Firefox and the other WASAPI user software), however the commit size doesn't grow further if Firefox is kept open but the other WASAPI user is closed (or the playback is stopped to release the sound device from exclusive mode).

Observed over the last few months (several auto-updates on many components, so version are examples) with:
Foobar2000 v1.4 + WASAPI plugin v3.3 or MPC-HC v1.7.x media-players
Firefox v62.0.x
Windows 10 x64 (builds 17134.x and 17763.1)
nVidia sound driver (GTX1070 card, v1.3.37.5 driver)


Actual results:

If allowed to continue for long enough (depending on the system it might take a few minutes or an hour) the entire system becomes unstable as the virtual memory limit runs out and no memory can be allocated for anything useful.


Expected results:

There should be no memory leak or at least this "something" (assuming it might be some kind of buffer or queue which is otherwise necessary) should be capped to a size which is not probable to cause system wide issues (say, a few hundred megabytes instead of tens of gigabytes or infinity).

Comment 1

[Matthew Gregan [:kinetik]]

To clarify, do you see commit size growing only while Firefox is playing media or all the time that Firefox is running?

I'm seeing something similar to what you describe, but want to make sure we're seeing the same issue.

If I have foobar2000 playing a continuous MP3 stream via WASAPI exclusive mode and Firefox playing a YouTube video (audio inaudible because it can't open the same audio output device), I can see the Firefox content process associated with the YouTube tab growing during playback.  about:memory shows media/decoded/audio is 535.8MB after playing a video on YouTube for 23 minutes.  535.8 MB would be around 26 minutes of 44.1 kHz stereo float32 audio, so that implies we're filling the audio queue upstream of libcubeb even when we've failed to open an output device.

If I seek in the video, media/decoded/audio drops to a few MB and starts growing again.  I'll move this to Audio/Video: Playback, since it looks like the issue is upstream of libcubeb.

Comment 2

[János Tóth F.]

(In reply to Matthew Gregan [:kinetik] from comment #1)
> To clarify, do you see commit size growing only while Firefox is playing
> media or all the time that Firefox is running?

Technically all the time but I always have a few tabs open with some sites often per-loading multimedia content or allowed to send audio alerts, so may be I always have "digital silence" playing from some sources even if the un/mute speaker icon is not displayed on any tabs (I am not sure how reliable that speaker icon is).

Here is a screenshot of Windows Task Manager after playing a hi-res multi-channel audio track with MPC-HC for ~10 minutes (while no Firefox tab displays the un/mute speaker icon but there are multimedia-rich sites open):
https://prohardver.hu/dl/upc/2018-10/35956_commitsize.png
The commit size looks normal (~150% of used memory) before the WASAPI bug triggers but the dead weight does not disappear after closing both Firefox and MPC-HC (the system has to be rebooted).

So, I am not entirely sure we are seeing the same thing. But I think these are probably related.

I don't see this pile in about:memory (although I am not experienced with it):

Main Process
          0.00 MB ── gfx-d2d-vram-draw-target
          0.00 MB ── gfx-d2d-vram-source-surface
          0.02 MB ── gfx-surface-win32
          0.00 MB ── gfx-textures
          0.00 MB ── gfx-textures-peak
          0.00 MB ── gfx-tiles-waste
                0 ── ghost-windows
          0.00 MB ── gpu-committed
          0.00 MB ── gpu-dedicated
          0.00 MB ── gpu-shared
        349.96 MB ── heap-allocated
          1.00 MB ── heap-chunksize
        372.00 MB ── heap-mapped
          2.70 MB ── imagelib-surface-cache-estimated-locked
          2.70 MB ── imagelib-surface-cache-estimated-total
                0 ── imagelib-surface-cache-overflow-count
          1.93 MB ── js-main-runtime-temporary-peak
        489.42 MB ── private
        729.05 MB ── resident
        479.86 MB ── resident-unique
        169.46 MB ── shmem-allocated
        169.52 MB ── shmem-mapped
          3.73 MB ── system-heap-allocated
                0 ── unresolved-ipc-responses
  2,103,621.14 MB ── vsize
129,415,501.75 MB ── vsize-max-contiguous
          0.00 MB ── wasm-runtime


Web Content (pid 1344)
        685.71 MB ── d3d11-shared-textures
          0.00 MB ── gfx-d2d-vram-draw-target
          0.00 MB ── gfx-d2d-vram-source-surface
          0.00 MB ── gfx-surface-win32
          0.00 MB ── gfx-textures
          0.00 MB ── gfx-textures-peak
          0.00 MB ── gfx-tiles-waste
                0 ── ghost-windows
        794.24 MB ── gpu-committed
        964.95 MB ── gpu-dedicated
         24.07 MB ── gpu-shared
        819.78 MB ── heap-allocated
          1.00 MB ── heap-chunksize
      1,144.00 MB ── heap-mapped
          1.74 MB ── imagelib-surface-cache-estimated-locked
          6.69 MB ── imagelib-surface-cache-estimated-total
                0 ── imagelib-surface-cache-overflow-count
         16.18 MB ── js-main-runtime-temporary-peak
      2,166.77 MB ── private
      1,425.21 MB ── resident
      1,357.17 MB ── resident-unique
         27.52 MB ── system-heap-allocated
  2,106,899.90 MB ── vsize
129,316,863.09 MB ── vsize-max-contiguous
          0.00 MB ── wasm-runtime


WebExtensions
          0.00 MB ── gfx-d2d-vram-draw-target
          0.00 MB ── gfx-d2d-vram-source-surface
          0.00 MB ── gfx-surface-win32
          0.00 MB ── gfx-textures
          0.00 MB ── gfx-textures-peak
          0.00 MB ── gfx-tiles-waste
                0 ── ghost-windows
          3.85 MB ── gpu-committed
          2.95 MB ── gpu-dedicated
          0.17 MB ── gpu-shared
         91.07 MB ── heap-allocated
          1.00 MB ── heap-chunksize
        272.00 MB ── heap-mapped
          0.00 MB ── imagelib-surface-cache-estimated-locked
          0.00 MB ── imagelib-surface-cache-estimated-total
                0 ── imagelib-surface-cache-overflow-count
          1.87 MB ── js-main-runtime-temporary-peak
        283.50 MB ── private
        304.89 MB ── resident
        258.13 MB ── resident-unique
          3.00 MB ── system-heap-allocated
  2,109,526.66 MB ── vsize
128,887,570.56 MB ── vsize-max-contiguous
      6,143.94 MB ── wasm-guard-pages
          0.00 MB ── wasm-runtime


GPU
          0.00 MB ── gfx-surface-win32
          2.44 MB ── heap-allocated
          1.00 MB ── heap-chunksize
          5.00 MB ── heap-mapped
        268.32 MB ── private
        122.25 MB ── resident
         28.16 MB ── resident-unique
          2.35 MB ── system-heap-allocated
  2,102,042.27 MB ── vsize
130,165,332.94 MB ── vsize-max-contiguous

Comment 3

[János Tóth F.]

I found another case where this bug gets triggered (without using the Exclusive mode): when any kind of audio starts playing in Firefox (notification sounds, alarms, etc) after Windows turned off all the HDMI/DP display devices where the sound could go (and there are no other non-HDMI/DP audio devices active/enabled).

This is why I found my PC with inflated commit size a few times and not remembering if I used any WASAPI multimedia applications at all.

Since Windows power management turns off displays by default, I think this issue should be taken a little more seriously.
The bug can trigger on any Windows systems without of any "special" application interference.

All you need is FireFox in the background and the integrated non-HDMI/DP sound devices disabled and some webpage capable of starting audio playback without further user interaction and you have inflated commit size. I think some people will assume this to be some kind of power save related issue (like a sleep-resume bug, especially if the system entered sleep state later).

Comment 4

[Matthew Gregan [:kinetik]]

Thanks for the additional information.  I was only looking at Firefox's memory use, but it sounds like you're seeing the Committed value in Task Manager->Performance->Memory rise when this bug occurs, and it does not drop when Firefox is quit?  That may be a separate issue to what I found - I'll see if I can find a system to test on with HDMI/DP audio devices and try again.

Rather than reboot, can you please try restarting the "Windows Audio" and "Windows Audio Endpoint Builder" services in the Services control panel and see if that causes the Committed value to drop as you expect?  If that doesn't work, please try opening about:preferences in Firefox and creating a new string pref called "media.cubeb.backend" with the value "winmm", restarting Firefox, then reproducing the issue - that's not a fix, just a way to narrow down the code causing the issue you're seeing.  Once you've tested that pref, please revisit about:preferences and delete the "media.cubeb.backend" pref.

Comment 5

[János Tóth F.]

Restarting the "Windows Audio" service seems to release the memory. I did a single quick test on a live system, so the baseline wasn't accurately established but the commit size shrank by several gigabytes upon restarting the service (if something still remains that's smaller by at least an order of magnitude). I will check the winmm option tomorrow.

Comment 6

[János Tóth F.]

media.cubeb.backend=winmm eliminates the commit size problem (tested with HDMI AVR as sound output and MPC-HC in Exclusive mode while FireFox played a youtube clip).
Note to other readers: it's about:config not about:preferences

Comment 7

[Matthew Gregan [:kinetik]]

Thanks for the update, that's very useful.  I'll try to reproduce locally and see what we can do to fix this.

Comment 8

[VEG]

I have a similar issue in my program which uses WASAPI. It seems that it is a bug in newer versions of Windows. I can reproduce it on latest Windows 10, but I can't reproduce it on Windows 7.

In my code, the memory leak in the audio service appears on this line:
hr = m_audio_client->Initialize(AUDCLNT_SHAREMODE_SHARED, AUDCLNT_STREAMFLAGS_NOPERSIST, static_cast<UINT64>(m_buffer_size_in_ms) * 10000, 0, m_mix_format, NULL);

The function call returns AUDCLNT_E_DEVICE_IN_USE, but the audio service allocates the requested buffer anyway, and doesn't free it for some reason. So, if you retry to initialize an audio client every second, the audio service will allocate more and more memory for leaked buffers. Apparently, it is a bug in Windows Audio Service itself.

Didn't figure out how to workaround it.

Comment 9

[VEG]

The Windows Audio Service leaks not only memory, but handles also. When you run one program which plays audio in exclusive mode, and another program, which tries to initialize an audio client in shared mode once a second (and of course you release everything properly when it has failed), you will clearly see how amount of handles is increasing using Process Explorer. Process Hacker allows you to see anonymous handles of type Section which are leaked, and their size is exactly the size of buffer which was requested.

I hope that you have some contacts in Microsoft to report them about this bug. It would be really nice if they fix it =)

Comment 10

[VEG]

A possible workaround: as soon as the audio renderer thread is interrupted, we try to reinitialize immediately, and if we receive AUDCLNT_E_DEVICE_IN_USE, it means that there is an exclusive mode session on the device, so we can enumerate all active sessions on this device (using IAudioSessionEnumerator), there should be just one active session, so we subscribe to their events and wait until OnSessionDisconnected is fired, it will mean that the exclusive session is over, and we can try to initialize our own session again.

Didn't implement it yet, but it seems that it is feasible. It will still leak a buffer (because we try to initialize one time anyway), but won't hurt the system performance as much as now it can.

Comment 11

[János Tóth F.]

The media.cubeb.backend=winmm workaround stopped working with v76.0 (audio and video playback stops working completely with that value, no matter if anything else is using the sound device or not).
Could anyone report this issue in detail to Microsoft in the mean time?

Comment 12

[VEG]

I reported this bug to Microsoft. It must be fixed in the upcoming Windows 10 version 20H2. Didn't test it yet.

Comment 13

[VEG]

Seems like the bug is fixed in Windows 11 from its first release, and Microsoft decided to not fix it in Windows 10.