Open Bug 1847453 Opened 1 years ago Updated 22 days ago

firefox uses too much vram and doesn't release it if i close tabs (decode device memory leak)

Categories

(Core :: Graphics, defect, P1)

Product:
Core
Component:
Graphics
Version:
Firefox 116
Platform:
Unspecified
Windows
Type:
defect

Tracking

()

People

(Reporter: gagauzanatoly, Assigned: ahale)

References

(Depends on 1 open bug, Blocks 2 open bugs)

Details

(Keywords: stalled)

Attachments

(10 files)

123.rar
189.07 KB, application/vnd.rar
Details
memory-report.json
2.82 MB, application/json
Details
about support.txt
52.17 KB, text/plain
Details
Memory consumption while watching 4k60fps video
11.23 KB, image/png
Details
patch - Add log of nv12 RenderTextureHost count log
2.09 KB, patch
Details | Diff | Splinter Review
about:support from jgilbert's machine
88.65 KB, application/json
Details
about:support
54.72 KB, text/plain
Details
about:support when repeatedly opening and closing a bird video
33.44 KB, text/plain
Details
memory-report-bird-video-4-before.json.gz
188.58 KB, application/x-gzip
Details
memory-report-bird-video-4-after.json.gz
212.36 KB, application/x-gzip
Details
Attached file 123.rar —

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/116.0

Steps to reproduce:

open many tabs

Actual results:

the browser starts to consume a lot of video memory, when closing video tabs, the memory is not freed

Expected results:

Video memory should have been freed

Attached file memory-report.json —
Attached file about support.txt —

We have had some bugs in firefox recently related to leaking memory when there is something animated in an extension / theme. It looks from your about:support like you have quite a lot of extensions. Would it be possible to test locally in a new profile without any extensions, and see if can still reproduce the memory usage problem with that fresh profile?

Flags: needinfo?(gagauzanatoly)

(In reply to Glenn Watson [:gw] from comment #3)

We have had some bugs in firefox recently related to leaking memory when there is something animated in an extension / theme. It looks from your about:support like you have quite a lot of extensions. Would it be possible to test locally in a new profile without any extensions, and see if can still reproduce the memory usage problem with that fresh profile?

i did, nothing changed

Flags: needinfo?(gagauzanatoly)

The Bugbug bot thinks this bug should belong to the 'Firefox::Tabbed Browser' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Component: Untriaged → Tabbed Browser
Component: Tabbed Browser → Graphics
Product: Firefox → Core
Blocks: gfx-triage
Severity: -- → S2
Priority: -- → P3

Ok! Well, it's strange that it uses more than Chrome!
I wonder if it's some less-than-ideal WR allocations.
Looking at the memory report, we see:

201.70 MB (100.0%) -- explicit
├──184.95 MB (91.69%) -- gfx
│ ├──184.95 MB (91.69%) -- webrender
│ │ ├──161.95 MB (80.29%) ── swgl
│ │ ├────9.00 MB (04.46%) ── upload-stagin-memory
│ │ ├────5.61 MB (02.78%) -- resource-cache
│ │ │ ├──5.61 MB (02.78%) ── fonts
│ │ │ └──0.00 MB (00.00%) ++ (3 tiny)
│ │ ├────3.40 MB (01.68%) ── texture-cache/structures
│ │ ├────2.93 MB (01.45%) ++ (6 tiny)
│ │ └────2.06 MB (01.02%) ++ interning
│ └────0.00 MB (00.00%) ── native-font-resource-data

"Other measurements:":

275.18 MB (100.0%) -- gfx
└──275.18 MB (100.0%) -- webrender
├──268.90 MB (97.72%) -- textures
│ ├──152.00 MB (55.24%) ── render-targets
│ ├───47.74 MB (17.35%) -- texture-cache
│ │ ├──45.25 MB (16.44%) ── atlas
│ │ └───2.49 MB (00.90%) ── standalone
│ ├───32.00 MB (11.63%) ── depth-targets
│ ├───17.58 MB (06.39%) ── swap-chains
│ ├────9.25 MB (03.36%) ── upload-staging-textures
│ ├────6.79 MB (02.47%) ── render-texture-hosts
│ └────3.55 MB (01.29%) ++ (4 tiny)
└────6.27 MB (02.28%) ++ images/mapped_from_owner

so, there is no solution i guess?

Not yet, still poking at this but trying to figure out how to debug it.

Ohh, vram, adding those columns to the "details" task manager tab, I can see that for me it says that (the main?) one of my firefox processes is using 15.9GB of vram.
Sorry for misunderstanding you! I'm looking into this.

Assignee: nobody → jgilbert
Status: UNCONFIRMED → NEW
Ever confirmed: true

Specifically:

  • Open (Win11) Task Manager
  • Go to Details tab/page
  • Right-click a column heading (e.g. Name)
  • Click "Select columns"
  • (scroll down a lot, to near the bottom, and) enable "Dedicated GPU memory" and "Shared GPU memory" and click Ok
  • Sort by either of these columns
  • firefox.exe and dwm.exe show up with large amounts.
    On this igpu machine, I see:
    • firefox.exe: dedicated: 348,xxx K, Shared: 813,xxx K
    • dwm.exe: dedicated: 304,xxx K, Shared: 426,xxx K
See Also: → 1854407
Assignee: jgilbert → nobody

Sotaro, can you reproduce this?

Flags: needinfo?(sotaro.ikeda.g)
Priority: P3 → --

It might be related to Bug 1834615. Since Bug 1717119 fix, a lot of video frames are decoded with 4k+ video. A lot of video frame happens with software video decoding.

But the video frames should be freed with tab close.

Flags: needinfo?(sotaro.ikeda.g)
See Also: → 1834615
Flags: needinfo?(sotaro.ikeda.g)

D3D11YCbCrImage or D3D11ShareHandleImage allocation in RDD process could increase GPU memory usage.

This GPU memory usage could be reduced when Bug 1855277 and Bug 1856516 are addressed.

:gagauzanatoly, any chance you could re-test with the current Nightly (Fx120) since those two issues Sotaro referenced (in comment #14) have landed?

Flags: needinfo?(gagauzanatoly)

(In reply to Bob Hood [:bhood] from comment #15)

:gagauzanatoly, any chance you could re-test with the current Nightly (Fx120) since those two issues Sotaro referenced (in comment #14) have landed?

https://youtu.be/Zp8GQQxvdds

i opened a video on youtube 4k and closed it, then look at memory load on the video when browser is open (no tabs), and when browser is closed

Flags: needinfo?(gagauzanatoly)

I'll attempt repro and see if I can identify where the memory is being held.

Flags: needinfo?(ahale)

I will try building a leakdetector version of firefox locally and check gpuview as well for gpu resources held when following instructions in comment #11 as soon as I have some cycles to poke at this.

Blocks: 4k-video

ETW (or gpuview which consumes ETW logs) can list objects, so we could detect leaked objects that way.

Other leak detector methods could only detect the leaked object references on our side, not the magnitude of leak.

ETW can list objects.
D3D has an API in debug layers that should let us ask for a list or info about live d3d objects. Does it work for DXGI objects?
Normal Firefox leak detector (not DMD) builds might tell us if we're holding on to some type of gecko object also.
DMD builds might show us if we are leaking DXGI objects also.

No longer blocks: gfx-triage

I've noticed this as well. Devtools seems to be extremely heavy on VRAM now, just opening a few tabs and devtools will spike the VRAM to 3gb in a matter of seconds.

It's completely freezing my PC up after vram fills up and overflows into page or system memory - I was seeing about 100gb of VRAM usage according to task manager today with about 40 tabs open, even closing all the tabs doesn't release the VRAM and requires a Firefox restart.

Hi tbob13, can you attach about:support to this bug?

Flags: needinfo?(tbob13)

(In reply to Kelsey Gilbert [:jgilbert] from comment #12)

Sotaro, can you reproduce this?

I could not reproduce the problem :( With Attachment 9378196 [details] [diff], a number of NV12 RenderTextureHost always became zero. From it, the problem might exist around graphics driver or WMF.

Flags: needinfo?(sotaro.ikeda.g)
Depends on: 1859432
Depends on: 1861605

If the problem always happens without video playback. The problem might exist around graphics driver.

OS: Unspecified → Windows

From matrix,

it seems to grow over time if I am on youtube playing a video (e.g. https://www.youtube.com/watch?v=i1f_bpRPDio) and scroll down so the video is off the viewport, and then scroll back so that the video is on-display again
퇴근길, 골목에서 들려오는 시티팝 감성, 한국 가요 🌆📀||LP||Citypop||1990s||감성 - YouTube
it grows and shrinks, but averages out to growing
kinda like, -40MB, +50MB or something (not exact numbers)
(I'm not saying it's just youtube, that's just what I happened to test)
(it might not even be videos at-fault, but it's kinda suspicious)
I'm now at 27.3 GB up from 27.0GB

It repros better for me if I choose like "theater" mode or whatever it's called, rather than the default view. Don't need to fullscreen, but bigger=quicker climb

If we set gfx.webrender.software to true to force SWGL, does this still happen?

(In reply to Kelsey Gilbert [:jgilbert] from comment #11)

Specifically:

  • Open (Win11) Task Manager
  • Go to Details tab/page
  • Right-click a column heading (e.g. Name)
  • Click "Select columns"
  • (scroll down a lot, to near the bottom, and) enable "Dedicated GPU memory" and "Shared GPU memory" and click Ok

From the following, "Dedicated GPU memory" in Details of TaskManager might not show correct value.

https://learn.microsoft.com/en-us/troubleshoot/windows-client/performance/gpu-process-memory-counters-report-wrong-value

See Also: → 1879521

The counters that I believe TaskManager uses are these ones:
https://stackoverflow.com/questions/73476947/how-to-get-the-dedicated-gpu-memory-number-for-every-running-process-in-window

@jrmuizel says we should include these numbers in about:support.

This bug has been around since last summer, implying it probably doesn't meet the criteria for S2.

Attached file about:support —
(In reply to Sotaro Ikeda [:sotaro] from comment #22) > Hi tbob13, can you attach about:support to this bug?

(In reply to tbob13 from comment #32)

(In reply to Sotaro Ikeda [:sotaro] from comment #22)

Hi tbob13, can you attach about:support to this bug?

Today it got really bad, entire PC was taking seconds to respond in some cases until I closed Firefox, this is not exactly a slow PC using all NVME storage, i7 9960x, 3600mhz DDR4. The GPU is an older GTX 1080.

Task manager reported that Firefox was using over 950gb(!) of GPU memory - probably getting pushed into the page file or similar. I hadn't restarted Firefox in several days and had maybe 200 tabs open and 20-30 windows. I ordered an RX 6800 with 16gb of VRAM to help alleviate the issue as I rely on Firefox but I don't know if it will really help in the end. It doesn't seem to be an issue on OSX only Windows - I think it's the way it is interacting with DWM as it was reporting very high VRAM usage (about 30gb) as well until Firefox was closed.

Flags: needinfo?(tbob13)

It is better with the 6800. I've seen a max of about 12gb of VRAM usage according to GPU-Z with several hundred windows open. Task manager still shows very high numbers but slowdowns are less common at least. None of this happens with other browsers.

(In reply to tbob13 from comment #34)

It is better with the 6800. I've seen a max of about 12gb of VRAM usage according to GPU-Z with several hundred windows open. Task manager still shows very high numbers but slowdowns are less common at least. None of this happens with other browsers.

can't believe you bough a new gpu only because of firefox 💀💀💀, i just close and reopen browser when running out of memory lmao

(In reply to gagauzanatoly from comment #35)

(In reply to tbob13 from comment #34)

It is better with the 6800. I've seen a max of about 12gb of VRAM usage according to GPU-Z with several hundred windows open. Task manager still shows very high numbers but slowdowns are less common at least. None of this happens with other browsers.

can't believe you bough a new gpu only because of firefox 💀💀💀, i just close and reopen browser when running out of memory lmao

Well I can't just restart the browser when I'm in the middle of a project with several hundred windows/tabs open... It is a bit frustrating though.

Ashley will focus attention on furthering this investigation this week.

Assignee: nobody → ahale

An experiment for reporters to try - can you go to about:config and changing the setting gfx.direct3d11.reuse-decoder-device to false and see if the issue gets better? We're investigating that as part of a different bug (Bug 1890622) and it might be a culprit on more GPUs than we realize.

Flags: needinfo?(tbob13)
Flags: needinfo?(gagauzanatoly)
Flags: needinfo?(ahale)

I'm going to lay out my assumptions to investigate:

  • Not GPU driver specific - NVIDIA and AMD dGPUs reported here, I am not sure about iGPUs though
  • May be related to video decoder reuse ( bug 1890622 ) - can try toggling gfx.direct3d11.reuse-decoder-device off in about:config
  • May depend on whether HDR is enabled in Windows (as this alters how the dwm.exe process operates).
  • Ads could contribute significantly to number of total videos started in a session, if that is a trigger.
  • Issue takes some time to repro (~days) to become severe, but is quite severe.
  • Could depend on whether dcomp is used (gfx.webrender.dcomp-win.enabled) as that may significantly alter the refcounting of DXGI objects and nearly eliminates sharing swapchains with dwm.exe as there is just one per window.
  • Could depend on whether DXGIVirtualSurface is used (gfx.webrender.dcomp-use-virtual-surfaces), as that feature is only used by Firefox and likely some XAML and UWP apps.
  • May be measurable in Task Manager -> Performance -> GPU -> Dedicated GPU memory usage and Shared GPU memory usage graphs
    • On this work laptop (Intel iGPU + NVIDIA) after closing tabs on a long-running Firefox Nightly browser, system memory usage is at 20.8GiB/63.7GiB, Shared GPU memory usage is at 1.3/31.9GiB. Closing that browser and reopening it (same tab after reopening the browser) puts system memory usage at 20.4/63.7Gib and GPU 0 Shared GPU memory usage at 1.2/31.9GiB, so no appreciable leak in these metrics which we consider to be reliable. Also no change in size of pagefile.sys or swapfile.sys, which indicates that if any leaks occurred, they were not sufficient to cause swapping in this case.
  • May be measurable in Task Manager -> Details -> Dedicated GPU memory usage, Shared GPU memory usage
    • To enable these columns, right click on any column title and choose select columns, scroll down and check relevant boxes.
    • Dedicated GPU memory usage column can be unreliable according to https://learn.microsoft.com/en-us/troubleshoot/windows-client/performance/gpu-process-memory-counters-report-wrong-value but the Performance pane is reliable.
    • Shared GPU memory usage on processes can count memory multiple times for compositor surfaces shared across processes and such things according to https://devblogs.microsoft.com/directx/gpus-in-the-task-manager/ but there is no advisory that it is fundamentally unreliable, unlike the Dedicated GPU memory usage column.
    • Note: On this work laptop after leaving Firefox running for a week in usual office usage patterns (gmail, calendar, zoom calls, etc) with a considerable number of open tabs (bugzilla), Shared GPU memory usage is at 44,397,168KiB (which is physically possible using swapfile but seems unlikely as the swapfile is sitting at 16MiB) and does not go down much if I close tabs. But Shared GPU memory usage in the Performance tab shows only 1.4GiB/32GiB shared GPU memory in use, down from 4.1GiB before closing all but one tab (this one). 1.4GiB still seems like more than I'd expect for a single Firefox Nightly tab and the Windows Task Manager window though. Shared GPU memory reported on dwm.exe (Desktop Window Manager) is 4,177,956KiB, which is also more than the Performance tab shows for that metric. So I have significant doubts about the accuracy of Shared GPU memory usage in this Details tab.

(In reply to Ashley Hale [:ahale] from comment #38)

An experiment for reporters to try - can you go to about:config and changing the setting gfx.direct3d11.reuse-decoder-device to false and see if the issue gets better? We're investigating that as part of a different bug (Bug 1890622) and it might be a culprit on more GPUs than we realize.

I'll give it a shot. I understand that the VRAM usage numbers in task manager may not be accurate but GPU-Z should be. When I would check VRAM in GPU-Z it would show 7.8gb on the GTX 1080 and closing Firefox would immediately cause it to drop to about 1-2gb.

At the moment I have about 50 tabs open and see about 5gb of VRAM usage according to GPU-Z, task manager seems to agree at the moment but I did reboot this morning.

Flags: needinfo?(tbob13)

After setting setting gfx.direct3d11.reuse-decoder-device to false the inspector in dev tools has been very buggy at times. It will completely freeze up and won't respond for minutes then all of a sudden it will start responding again if I close the tab and open another. The whole dev tools window will go white randomly as well.

Sotaro, do you have any insights on why gfx.direct3d11.reuse-decoder-device = false would cause instability (comment #41 above)?

Flags: needinfo?(sotaro.ikeda.g)

With pref gfx.direct3d11.reuse-decoder-device = false, each hardware video decoder uses its own D3D11Device. It should not directly cause the problem like Comment 41. If the problem happened by the pref change, it might be caused by too much GPU memory usage.

The driver might cause memory leak when hardware video decoder(DXVA2.0 with WMF) is used.

Flags: needinfo?(sotaro.ikeda.g)
See Also: → 1890622

I've been seeing this in the log too and the GPU driver seems to crash. Windows page file is set to 500gb.

Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: RadeonSoftware.exe (9200) consumed 10145042432 bytes, firefox.exe (28184) consumed 7409647616 bytes, and firefox.exe (23420) consumed 4530593792 bytes.

Task manager with a few hundred tabs open
https://i.imgur.com/AAUz7Nt.png

GPU-Z taken immediately after the above
https://i.imgur.com/QzweXb2.png

GPU-Z after closing firefox immediately after the above
https://i.imgur.com/0hmtgrw.png

After I updated to version 126 a few days ago I cannot even use Firefox anymore as I have 12-15 tabs open and within 2-3 hours I am hitting almost 20 GB of Virtual commit which shows the GPU taking the most of that, this is something that has only started with version 126 for me I'm not sure about dropping links but the crash ID for my crash is <b7d82378-5744-4fb9-ba37-0bcfd0240518> .

Thanks for the crash id, the link for that one is https://crash-stats.mozilla.org/report/index/b7d82378-5744-4fb9-ba37-0bcfd0240518

I'm fairly certain the issue you are seeing in comment #45 is https://bugzilla.mozilla.org/show_bug.cgi?id=1897006 but I can see that in the current version of Firefox 126 the pref gfx.direct3d11.reuse-decoder-device is true, you might want to check for an optional Firefox update (open the menu and go to Help -> About Firefox - and click the refresh button near the top which may check for an updated version), or check if the gfx.direct3d11.reuse-decoder-device is true in about:config, keeping it true seems to reduce the memory leak in some usage scenarios.

Flags: needinfo?(iceman50)

Furthering my investigation into this issue, I seem to be able to reproduce a Commit Size memory leak of 330MiB on AMD Software: Adrenalin Edition version 24.5.1 drivers by opening and closing a video tab repeatedly in Firefox 126 stable release (where gfx.direct3d11.reuse-decoder-device is true). I can't repro the same issue on an Intel iGPU.

Steps to reproduce:

  • Open Firefox
  • Make sure there are only a couple of tabs and none have video.
  • Enable autoplay of videos with audio in Settings -> Privacy & Security if you are testing with a video that has audio.
  • Open a video tab, in this case I tested with a bird video on YouTube Shorts https://www.youtube.com/shorts/5GlgIRLQ96g , then use Ctrl-W to close the video tab.
  • Repeat the following process 10 times:
    • Press ctrl-shift-T to reopen the video tab.
    • Press ctrl-W to close the tab.
  • Observe the Windows Task Manager showing that the GPU process has increased its Commit Size by about 3.3GiB from the 10 repetitions of this compared to the first time.
Flags: needinfo?(paul.blinzer)

iceman50, can you check if the problem happens with latest nightly?

I've attached before and after memory reports for viewing that video 10 times in Firefox Nightly, I see reserved/private went up by 32MiB and gpu-dedicated went up 236.75MiB, I am not entirely sure how to interpret these numbers, but I have more memory reports if debugging is necessary. Notably there is very little of this memory increase tracked in explicit allocations, so I suspect it is media decoding related.

Explicit Allocations

3.21 MB (100.0%) -- explicit
├──3.20 MB (99.57%) -- gfx/webrender
│ ├──3.16 MB (98.22%) ── texture-cache/structures
│ ├──-0.17 MB (-05.23%) -- interning
│ │ ├──-0.12 MB (-03.60%) ── text_run/interners
│ │ ├──-0.06 MB (-01.82%) ── line_decoration/interners
│ │ └───0.01 MB (00.20%) ++ (2 tiny)
│ ├──0.13 MB (04.00%) ── display-list
│ ├──0.04 MB (01.37%) ++ (2 tiny)
│ └──0.04 MB (01.22%) ── swgl
├──0.09 MB (02.84%) -- threads
│ ├──0.07 MB (02.07%) ++ stacks
│ └──0.02 MB (00.77%) ++ overhead
└──-0.08 MB (-02.41%) ── heap-unclassified

Other Measurements

0.00 MB (100.0%) -- address-space
├──-39.51 MB (100.0%) ── free(segments=NNNN)
├──32.38 MB (100.0%) ── reserved/private(segments=NNNN)
└──7.13 MB (100.0%) -- commit
├──7.12 MB (100.0%) -- private
│ ├──6.98 MB (100.0%) ── readwrite(segments=NNNN)
│ ├──0.09 MB (100.0%) ── readwrite+stack(segments=NNNN)
│ ├──0.04 MB (100.0%) ── readwrite+guard(segments=NNNN)
│ ├──0.01 MB (100.0%) ── readonly(segments=NNNN)
│ └──0.01 MB (100.0%) ── readonly+guard(segments=NNNN)
└──0.01 MB (100.0%) -- mapped
├──0.07 MB (100.0%) ── readonly(segments=NNNN)
├──-0.05 MB (100.0%) ── noaccess(segments=NNNN)
└──-0.00 MB (100.0%) ── readwrite(segments=NNNN)
...

236.75 MB ── gpu-dedicated
0.03 MB ── gpu-shared
3.12 MB ── heap-allocated
7.11 MB ── private
5.94 MB ── resident
5.62 MB ── resident-unique
-0.03 MB ── shmem-mapped
1.18 MB ── system-heap-allocated
39.51 MB ── vsize

It's likely that the best solution to the leak on AMD drivers will be Bug 1893427 in the long run, but hopefully there are other solutions and workarounds we can look into sooner.

Depends on: hw-ffvpx
See Also: → hw-ffvpx
See Also: → 1897875
Depends on: 1897006

(In reply to Ashley Hale [:ahale] from comment #53)

It's likely that the best solution to the leak on AMD drivers will be Bug 1893427 in the long run, but hopefully there are other solutions and workarounds we can look into sooner.

We are actively working to analyze the root for the memory use so that DX device reuse wouldn't be needed. I will kep this ticket updated, it may be useful for testing to have the workaround controllable via an about:config option as the mechanism may be useful in other scenarios for regression or functional testing.

Flags: needinfo?(paul.blinzer)

Still seeing high VRAM usage. I am currently on the beta.

5,341.09 MB (100.0%) -- explicit
├──2,161.93 MB (40.48%) -- gfx
│ ├──2,161.10 MB (40.46%) -- webrender
│ │ ├──1,700.49 MB (31.84%) ── swgl
│ │ ├────152.30 MB (02.85%) ── display-list
│ │ ├────114.00 MB (02.13%) ── upload-stagin-memory
│ │ ├─────92.28 MB (01.73%) ++ interning
│ │ ├─────54.90 MB (01.03%) ── texture-cache/structures
│ │ └─────47.13 MB (00.88%) ++ (6 tiny)
│ └──────0.83 MB (00.02%) ++ (8 tiny)
├──1,664.17 MB (31.16%) -- js-non-window
│ ├──1,627.59 MB (30.47%) -- zones
│ │ ├──1,619.13 MB (30.31%) -- zone(0x2a3ccb3b000)
│ │ │ ├──1,263.74 MB (23.66%) -- strings
│ │ │ │ ├────948.01 MB (17.75%) -- string(length=530210, copies=546, (truncated))
│ │ │ │ │ ├──948.00 MB (17.75%) ── malloc-heap/two-byte
│ │ │ │ │ └────0.01 MB (00.00%) ── gc-heap/two-byte
│ │ │ │ └────315.73 MB (05.91%) ++ (1206 tiny)
│ │ │ ├────210.70 MB (03.94%) -- realm([System Principal], DevTools (Module loader))
│ │ │ │ ├──209.55 MB (03.92%) -- classes
│ │ │ │ │ ├──146.22 MB (02.74%) ++ (12 tiny)
│ │ │ │ │ └───63.33 MB (01.19%) ++ class(Object)/objects
│ │ │ │ └────1.15 MB (00.02%) ++ (4 tiny)
│ │ │ ├─────73.83 MB (01.38%) -- realm([System Principal], shared JSM global)
│ │ │ │ ├──69.03 MB (01.29%) ++ classes
│ │ │ │ └───4.80 MB (00.09%) ++ (4 tiny)
│ │ │ └─────70.85 MB (01.33%) ++ (22 tiny)
│ │ └──────8.46 MB (00.16%) ++ (2 tiny)
│ └─────36.58 MB (00.68%) ++ (4 tiny)
├────627.87 MB (11.76%) ── heap-unclassified
├────509.73 MB (09.54%) -- window-objects
│ ├──364.65 MB (06.83%) ++ (47 tiny)
│ ├───90.89 MB (01.70%) -- top(about:devtools-toolbox, id=6391)/active
│ │ ├──84.55 MB (01.58%) -- window(chrome://devtools/content/inspector/index.xhtml)
│ │ │ ├──71.65 MB (01.34%) -- dom
│ │ │ │ ├──69.57 MB (01.30%) ── orphan-nodes
│ │ │ │ └───2.08 MB (00.04%) ++ (3 tiny)
│ │ │ └──12.90 MB (00.24%) ++ (3 tiny)
│ │ └───6.33 MB (00.12%) ++ (3 tiny)
│ └───54.18 MB (01.01%) ++ top(about:devtools-toolbox, id=5471)/active
├────287.60 MB (05.38%) ++ (29 tiny)
└─────89.79 MB (01.68%) ++ workers/workers(chrome)

Other Measurements
134,217,727.94 MB (100.0%) -- address-space
├──132,084,663.82 MB (98.41%) ── free(segments=4683)
├────2,116,817.18 MB (01.58%) -- reserved
│ ├──2,097,110.25 MB (01.56%) ── mapped(segments=21)
│ └─────19,706.92 MB (00.01%) ── private(segments=16275)
└───────16,246.95 MB (00.01%) ++ commit

 8,250.54 MB ── gpu-committed
 4,729.42 MB ── gpu-dedicated
   348.73 MB ── gpu-shared
 4,909.68 MB ── heap-allocated

(In reply to Paul Blinzer from comment #54)

(In reply to Ashley Hale [:ahale] from comment #53)

It's likely that the best solution to the leak on AMD drivers will be Bug 1893427 in the long run, but hopefully there are other solutions and workarounds we can look into sooner.

We are actively working to analyze the root for the memory use so that DX device reuse wouldn't be needed. I will kep this ticket updated, it may be useful for testing to have the workaround controllable via an about:config option as the mechanism may be useful in other scenarios for regression or functional testing.

It is controllable using the prefs gfx.direct3d11.reuse-decoder-device and gfx.direct3d11.reuse-decoder-device-force-enabled, I assume the latter forces it to reuse the decoder device even if the former pref is false, so you probably want that latter one set to false so the former fully controls the behavior.

That said, the leak was worse at 330MiB per video decoder in Firefox 126 (release version) than in current Nightly where it is about 1-3MiB in my experience, you may want to pick a version using mozregression that demonstrates the issue most succinctly and then keep using that version for testing.

Flags: needinfo?(paul.blinzer)

(In reply to tbob13 from comment #55)

Still seeing high VRAM usage. I am currently on the beta.

5,341.09 MB (100.0%) -- explicit
├──2,161.93 MB (40.48%) -- gfx
│ ├──2,161.10 MB (40.46%) -- webrender
│ │ ├──1,700.49 MB (31.84%) ── swgl

That seems like a lot of memory usage for swgl, in about:support I am guessing it says WebRender (Software) ? I'm not sure if the decoder device leak behaves the same when using software webrender, though that's an interesting question as well.

(In reply to Ashley Hale [:ahale] from comment #57)

(In reply to tbob13 from comment #55)

Still seeing high VRAM usage. I am currently on the beta.

5,341.09 MB (100.0%) -- explicit
├──2,161.93 MB (40.48%) -- gfx
│ ├──2,161.10 MB (40.46%) -- webrender
│ │ ├──1,700.49 MB (31.84%) ── swgl

That seems like a lot of memory usage for swgl, in about:support I am guessing it says WebRender (Software) ? I'm not sure if the decoder device leak behaves the same when using software webrender, though that's an interesting question as well.

Yeah, I'm not sure. It's also worth mentioning there are no videos open. About 200 tabs spread across ~20 windows. FF has been open for about a week and a half according to Process Explorer.

Compositing WebRender

WEBRENDER default available

Not sure what that means.

Compositing WebRender means it is rendering using D3D11, so I am not sure why swgl is showing up there with significant memory usage, but it may not be relevant to the larger memory issue.

The decoder device issue increases memory usage each time you open and close a video, so it isn't a matter of how many videos you currently have open, but how many have been opened and closed since the browser was started, ads can be videos so that can be a significant source of videos depending on what you encounter on the web.

(In reply to Ashley Hale [:ahale] from comment #60)

Compositing WebRender means it is rendering using D3D11, so I am not sure why swgl is showing up there with significant memory usage, but it may not be relevant to the larger memory issue.

The decoder device issue increases memory usage each time you open and close a video, so it isn't a matter of how many videos you currently have open, but how many have been opened and closed since the browser was started, ads can be videos so that can be a significant source of videos depending on what you encounter on the web.

I see. Either way this session did not likely see any significant amount of videos, the mentioned profile is primarily used for CSS styling (hence the devtools) and none of the projects have ads or videos.

I've been seeing this in the log too and the GPU driver seems to crash. Windows page file is set to 500gb.

Do not set the windows page file so low, the recommended maximum is still 1.5-2x available physical memory for a reason. (Commit charge is affected by address space fragmentation and that 500MB page file could be effectively as useless as a 4MB one.)

(In reply to Danial Horton from comment #62)

I've been seeing this in the log too and the GPU driver seems to crash. Windows page file is set to 500gb.

Do not set the windows page file so low, the recommended maximum is still 1.5-2x available physical memory for a reason. (Commit charge is affected by address space fragmentation and that 500MB page file could be effectively as useless as a 4MB one.)

500gb is more like 8x the 64gb of ram in the machine.

i read it as MB for some reason! doh

Blocks: 1902569
No longer blocks: wr-gpu-memory
See Also: hw-ffvpx →
No longer blocks: gfx-triage

Flagging as stalled until we receive a responses to outstanding needinfos.

Keywords: stalled

@ahale
regarding the "needinfo from AMD" and looking at this from our side side, it seems that as far as the driver is concerned, the video contexts are still held open and not destroyed, which means that from the driver point of view the associated resources are still kept around (and may be made resident by WDDM when the context is processed).
We will be looking at this on our side, but this may be something rooted in the application or runtime level, given that the destruction of the video device contexts doesn't occur for some reason or another.

Flags: needinfo?(paul.blinzer)
Priority: -- → P1

Sotaro, if these leaked decode devices are being held open on our side, do we have any other ways to track them?

Summary: firefox uses too much vram and doesn't release it if i close tabs → firefox uses too much vram and doesn't release it if i close tabs (decode device memory leak)

Redirect a needinfo that is pending on an inactive user to the triage owner.
:bhood, since the bug has high priority, high severity and recent activity, could you have a look please?

For more information, please visit BugBot documentation.

Flags: needinfo?(gagauzanatoly) → needinfo?(bhood)

Redirect a needinfo that is pending on an inactive user to the triage owner.
:bhood, since the bug has high priority and high severity, could you have a look please?

For more information, please visit BugBot documentation.

Flags: needinfo?(iceman50) → needinfo?(bhood)

Sotaro, adding a NI for the question in comment 67.

Flags: needinfo?(sotaro.ikeda.g)
Flags: needinfo?(bhood)

From the tooling I'm familiar with, I would probably be able to find them with gpuview because it was mentioned that they show up there, but that doesn't necessarily bring me any closer to finding the leak, since it's probably a refcounting issue on our side and thus wouldn't have any relevant events in gpuview after its creation.

It's also possible this was fixed in the past few months, so current status would be nice to know.

Duplicate of this bug: 1879521

(In reply to Ashley Hale [:ahale] from comment #67)

Sotaro, if these leaked decode devices are being held open on our side, do we have any other ways to track them?

With AMD GPUs, gecko always use compositor device as decoder device in DeviceManagerDx::CreateDecoderDevice(). When compositor device was not used as decoder device, we saw memory leak like Bug 1897006. Using the compositor device as decoder device is the same as for chromium.

gecko does not use ID3D11VideoContext directly. Instead gecko uses high level IMFTransform api for video decoding like MFTDecoder::Create()

Though chromium uses D3D video API directly like D3D11VideoDecoderWrapperImpl.

Then it might be possible that current IMFTransform usage might cause leaking ID3D11VideoContext internally. When I checked in the past MFTDecoder was not leaking.

Bug 1893427 is going to add a capability to use low level d3d11 video api directly.

Flags: needinfo?(sotaro.ikeda.g)
See Also: → 1897006, hw-ffvpx

Problem also shows with Firefox 133.0.3 on macOS Ventura 13.7.1, with Intel Core i5 and Intel Iris Plus Graphics 640.
Streaming H.264 with hardware acceleration eats up memory constantly, at a rate of about 50GB per half hour.

  • Superficial effect for the user: responsiveness of the computer goes down, eventually making the computer practically unusable (more than 10 seconds between mouse click or key press and seeing the effect).
  • Underlying effect: the growing size of virtual memory requiring more and more time to swap memory to/from the rotating hard-disk.

The only way to release the memory (taken up by the Firefox process) is to quit Firefox. Stopping the stream or closing the tab or window does not release the memory.

Workaround: switch off hardware acceleration in Firefox Settings (Firefox>Settings...>General>Performance, "Use recommended performance settings" OFF > "Use hardware acceleration when available" OFF). This keeps memory use of Firefox within about 2GB.

(In reply to Domo Viridi from comment #74)
It's better to open an additional issue for Mac, both of these issues are a part of hw-ffvpx.

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: