Very high gpu memory usage with many windows
Categories
(Core :: Graphics: WebRender, defect)
Tracking
()
People
(Reporter: lapineige, Unassigned)
References
(Blocks 1 open bug)
Details
Attachments
(6 files)
User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:89.0) Gecko/20100101 Firefox/89.0
Steps to reproduce:
After upgrading to Firefox 89, this issue happens all the time if Hardware Acceleration is active (webrender enable/disable as no influence).
No new extension was installed, but in safe-mode (with no extension enable) this issue seems to be reduced.
Actual results:
Firefox use much more RAM and VRAM than before: up to +7GB of RAM used, and 3,5GB of VRAM (up to 98% VRAM usage).
According to about:memory report, all that memory is used by a massive 6~7GB texture-cache in the webrender category.
Disabling webrender changes nothing.
Doing a Minimize Memory Usage instantly reduces memory usage back to normal. In few seconds after that, memory usage starts to increase regularly again, up to the previous extra +6GB in less than 30s.
Expected results:
Firefox should not consume massive extra memory "for nothing".
Note that this user agent is wrong (it is spoofed), I'm using Firefox 89 on Linux.
As a complement, here is an extract from the about:memory report:
5,758.98 MB (100.0%) -- explicit
├──4,914.71 MB (85.34%) ── heap-unclassified
5,560.60 MB (100.0%) -- gfx
└──5,560.60 MB (100.0%) -- webrender
├──5,364.41 MB (96.47%) -- textures
│ ├──3,893.37 MB (70.02%) ── texture-cache
│ ├────879.22 MB (15.81%) ── swap-chains
Note that right now I have disabled webrender.
Additional note: disabling Hardware Acceleration solves that issue.
|
||
Comment 3•3 years ago
|
||
The Bugbug bot thinks this bug should belong to the 'Core::Graphics: WebRender' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.
|
||
Comment 4•3 years ago
|
||
Nical do you need anything from the user to be able to debug understand the issue ?
|
||
Comment 5•3 years ago
•
|
||
Hi,
- Could you list your enabled extensions?
- Do you have multiple windows open when the problem is happening (and if so how many)? You have about 12 times as much swap-chain memory allocated than I do on a single full screen window (4k screen), that suggests that you have a lot of windows open. If that's the case It would be useful to check whether memory usage also explodes with a single open window.
- Could you save a full memory report of the issue happening with webrender and one without webrender, and attach them to this bug?
Hello,
Here is part of the answer, I did not have to time to do everything at once:
Do you have multiple windows open when the problem is happening (and if so how many)?
I don't have the figures with me right now, but that's something like: 50 windows, 1000 tabs.
Note: I know it's an extreme situation, yet before Firefox 89 it was running fine with a memory consumption under 5GB (in worst case scenario), but now it's more a 10-13GB memory consumption "for nothing" (garbage collecting it doesn't change anything). That's why I suppose it's a bug, not a performance issue.
If I close all windows but the main one (around 700 tabs), and open and load around 100 tabs (to be on pair with the number of active tabs), it goes back to a reasonable value (webrender enabled):
350.41 MB (100.0%) -- gfx
└──350.41 MB (100.0%) -- webrender
├──279.50 MB (79.76%) -- textures
│ ├──139.76 MB (39.88%) ── texture-cache
│ ├───69.71 MB (19.89%) ── render-texture-hosts
│ ├───35.00 MB (09.99%) ── depth-targets
│ ├───27.66 MB (07.89%) ── swap-chains
Could you save a full memory report of the issue happening with webrender and one without webrender, and attach them to this bug?
I will attache both. Note that I did enable webrender with about:config, then disable it the same way.
I have the feeling that "without" webrender it is even worse, but that was during normal use (webrender) compared to fresh startup (+ load a few tabs and wait 1m, "without" webrender).
I did not do that without hardware acceleration (it takes 10 min to start…).
Same problem. A couple of tabs open. Using inspector in one of the tabs and refreshing the page every now and then. After a while -10gigs of ram. I have 20 tabs in safari and it doesn't take even a half of that. There were almost no addonds installed because I don't use ff on regular basis.
89.0 Mojave
|
Reporter | |
Comment 10•3 years ago
|
||
(In reply to krq from comment #9)
Same problem. A couple of tabs open. Using inspector in one of the tabs and refreshing the page every now and then. After a while -10gigs of ram. I have 20 tabs in safari and it doesn't take even a half of that. There were almost no addonds installed because I don't use ff on regular basis.
Could you try to disable Hardware Acceleration (in Settings, search for that keyword) and see if this also "solves" the issue in your situation ?
|
|
||
Comment 11•3 years ago
|
||
Thanks a lot for the additional information. 50 windows definitely explains the situation, we have a rather high per-window overhead because each window has its own textures caches and pools. There are probably some low hanging fruits, but the biggest win here would be from having a single render backend thread for all windows and sharing the caches and pools between them. There is a lot of work to get there though. In the mean time a good thing to do would be to keep track of which windows have been rendered last and aggressively flush caches and pools from least recently shown windows when memory gets to a certain level (I filed bug 1716525).
|
|
Reporter | |
Comment 12•3 years ago
|
||
(In reply to Nicolas Silva [:nical] from comment #11)
we have a rather high per-window overhead because each window has its own textures caches and pools.
That's interesting, because in the past I believed I choose to split the tabs in more windows because (apart from organizational reason) I noticied better performance than a few windows with more tabs. But maybe not for memory consumption, I'm not sure…
In my case (and I suppose it's the most common use case for many (ex: >10) windows ?) most of them are rendered once after starting Firefox, then not used for a while before I need to interact with them. Hence those windows (at least most of them, maybe not the media playing ones) could be "paused" for a while without any issue.
For tabs I use Auto Tab Discard extension to discard them from memory (and CPU/GPU load), but that doesn't work for the last tab of the window.
I hope if it helps to find a temporary solution.
In the mean time a good thing to do would be to keep track of which windows have been rendered last and aggressively flush caches and pools from least recently shown windows when memory gets to a certain level
I don't know what that implies on the technical side, but what I notice right now is that I can minimize memory usage (and that works massively, and it goes back to normal), but the memory usage re-increase a lot in less than 1 minute after that (but not instantly, it's quite linear). After minimizing I noticed no performance issue.
|
|
Reporter | |
Comment 13•3 years ago
|
||
For the record, here is the precise current figures:
- 62 windows
- 1057 tabs
- main firefox process: 6.1 GB of RAM. That's much lower than the previous time (WebRender is still activated), I don't know why…
- for some reason, "only" 700MB of VRAM used (according to nvidia-smi) for firefox main process. 2900 for XOrg, and that's related (if I close firefox, it goes back to <500MB). Last times I did check it was more or less 3GB of VRAM for Firefox (98% VRAM used). I don't know why this time it doesn't fill the VRAM.
|
||
Comment 14•3 years ago
|
||
(In reply to lapineige from comment #13)
For the record, here is the precise current figures:
- 62 windows
- 1057 tabs
- main firefox process: 6.1 GB of RAM. That's much lower than the previous time (WebRender is still activated), I don't know why…
- for some reason, "only" 700MB of VRAM used (according to nvidia-smi) for firefox main process. 2900 for XOrg, and that's related (if I close firefox, it goes back to <500MB). Last times I did check it was more or less 3GB of VRAM for Firefox (98% VRAM used). I don't know why this time it doesn't fill the VRAM.
:lapineige, while we are investigating this, would you be willing to share some more information about your workflow. Mainly, why do you need 62 separate browser windows?
|
|
Reporter | |
Comment 15•3 years ago
|
||
That would be a long off-topic story, but I can share the main points.
Firstly that 62 windows is a bit of an extreme situation (I was currently doing some research for my work, opening lot of windows (about one particular topic/window). It went back to 44 right now.
Normal usage is around 30-40 windows.
I would say it's a combination of heavy multitasking and a lot set aside tasks, kept in a particular windows so I can see them easily. I'm working with 9 virtual desktops, one by "task" or "them" (2 being dedicated to working time, one for the main task, another one for a "side task"), with 0-10 Firefox windows/desktop (it's highly dependent of what I'm doing and when).
This have some key advantages for me :
- I can use the grid view (KDE's one if you see what it is) to visualize all "workspaces" (namely desktops), and know where I was working on [this], [this] and [that]
- inside one particular desktop, I can visualize all windows (~ sub-tasks) together at once without moving them, with a single shortcut (I think it's called "windows presentation" effect in KDE) and come back to what I was doing/switch to another window in an instant
- I can finely separate several tasks into one dedicated "workspace"
Most of these windows are 1-tab (or very few tabs) windows. They are mainly used to avoid having a single windows with dozens of (mostly) unrelated tabs, which is a mess to organize and handle.
About 50% of this is set aside as a "keep for later" content. A kind of read-it-later use case (but not that much (I'm using a read-it-later app for that), more precisely a "handle that later" use case). It can stay untouched for a few weeks in some (rare) cases. Apart from the main window (the one with pinned tabs, and most tabs), the other 50% is very ephemeral content that lives often less than a (few) day(s), like one shoot web searches. There is also quite a bunch of private browsing windows in the set, or windows with a special container.
I never did some real statistics, but I believe I open/close something like 10 to 20 windows a day… a bit like we usually use tabs ^^
Apart from memory usage (which is pretty well handled by tab lazy loading + unloading), there is only one main limit to that system: there is no thumbnails of the windows after opening Firefox, except if I focus them one by one (when it's fully loaded).
That's not a workflow that could be handled in such a practical way thought bookmarking, "tab group" extensions and so on. The old panorama view was a bit useful for that, but not that much (for instance: I can't ctrl+tab to another group).
Final note: I'm aware that a very particular use case, quite intense for the browser, but I'm not asking for Firefox to be that adapted to such an use case - yet it does the job pretty well (with a high memory usage, but not that much for such a big load), in particular since there are multiple processes to handle tabs and browser stuff, and since all the improvements to startup times (it pays an order of magnitude more than usual in my case). I was concerned because suddenly, for no apparent reason, it was consuming 3-4 times more memory.
|
|
||
Comment 16•3 years ago
|
||
I experimented with clearing pools and caches from windows that have not been rendered for 30 seconds. There's a build withe the patch here https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/ZTQsMPBjRVW-cDQ_mZrctA/runs/0/artifacts/public/build/target.tar.bz2 if you want to see give it a try.
As far as my local testing went it was able to remove a substantial amount of gpu memory usage when a lot of windows are open but not interacted with (and with nothing animating in them).
I was concerned because suddenly, for no apparent reason [..]
The reason is we rewrote the rendering engine. The new engine offloads a lot of work to the GPU and generally performs better than the old one but isn't designed with opening a large amount of windows in mind.
|
|
Reporter | |
Comment 17•3 years ago
|
||
(In reply to Nicolas Silva [:nical] from comment #16)
I experimented with clearing pools and caches from windows that have not been rendered for 30 seconds. There's a build withe the patch here https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/ZTQsMPBjRVW-cDQ_mZrctA/runs/0/artifacts/public/build/target.tar.bz2 if you want to see give it a try.
As far as my local testing went it was able to remove a substantial amount of gpu memory usage when a lot of windows are open but not interacted with (and with nothing animating in them).I was concerned because suddenly, for no apparent reason [..]
The reason is we rewrote the rendering engine. The new engine offloads a lot of work to the GPU and generally performs better than the old one but isn't designed with opening a large amount of windows in mind.
Great, thank you for explaining what's the reason behind.
I will give a try to that build.
Question: would that affect (audio) media playing tabs (when no animation is drawn) ?
Also do you want me to measure anything regarding performance ? Apart from memory usage I mean.
|
|
||
Comment 18•3 years ago
|
||
Question: would that affect (audio) media playing tabs (when no animation is drawn) ?
As long as nothing triggers a repaint of something (web content or UI) in the window, a lot of the window's gpu memory should get reclaimed after 30 seconds of inactivity. So sound shouldn't prevent that from happening.
Also do you want me to measure anything regarding performance ? Apart from memory usage I mean.
Thanks, this patch shouldn't affect performance in a noticeable way. If you notice slowdowns that you weren't seeing on a regular nightly build then please let me know but that's rather unexpected. The most interesting data point is whether the amount of graphics memory reported in about:memory looks better than with a regular firefox build on a similar wokload (a large-ish number of windows) after inactive windows have had time to cool down and be flushed.
|
|
Reporter | |
Comment 19•3 years ago
|
||
As long as nothing triggers a repaint of something (web content or UI) in the window, a lot of the window's gpu memory should get reclaimed after 30 seconds of inactivity. So sound shouldn't prevent that from happening.
Sorry it wasn't clear: in was looking the other way, would that operation on GPU cache affect media playing (preventing a background windows to play some music for instance) ? I suppose it would not be the case (it doesn't need any GPU stuff, right ?), but just in case…
The most interesting data point is whether the amount of graphics memory reported in about:memory looks better than with a regular firefox build on a similar workload (a large-ish number of windows) after inactive windows have had time to cool down and be flushed.
Ok then I will measure that. I'll try to open both at the same time, but I'm not sure if that would be possible without swapping…
|
|
||
Comment 20•3 years ago
|
||
would that operation on GPU cache affect media playing
Ah, no that's another part of the browser, this won't affect the audio stack.
Ok then I will measure that. I'll try to open both at the same time, but I'm not sure if that would be possible without swapping…
You could try them separately with similar wokloads and see in about:memory whether the custom build has a lower amount of memory. No need to go as far as 60 windows if it puts your system on its knees, but trying with a rather high window count like 20 is useful. Make sure to wait for a little bit (more than 30 seconds) before measuring.
|
|
Reporter | |
Comment 21•3 years ago
|
||
I tried your build. I did not notice any real change.
I will attach both memory report before and after the patch, and also before and after memory minimization (as a reference point for both versions).
I forgot that this would be a nightly build, thus v91 instead of my v89. I had some trouble converting my profile back to v89, but that's solved. I will make a copy next time.
|
|
Reporter | |
Comment 22•3 years ago
|
||
|
|
Reporter | |
Comment 23•3 years ago
|
||
|
|
Reporter | |
Comment 24•3 years ago
|
||
|
|
Reporter | |
Comment 25•3 years ago
|
||
|
|
Reporter | |
Comment 26•3 years ago
|
||
First of all, I will link that comment here: https://bugzilla.mozilla.org/show_bug.cgi?id=1701643#c10 (sorry, I have got myself mixed up between both tickets).
It documents some framerate issues.
Another finding : If I pause Firefox main process ("firefox") but not the others, all my framerate issues are gone, instantly. They re-appears instantly if I "unlock" it.
It doesn't depend on the (V)RAM usage, I have the issue regardless of any "minimize memory usage".
While watching a video, I also realized the issue comes back in "cycles": video starts to play rather smoothly (but not 60 fps), then starts to shutter a bit, then a lot, then it skips a few frame and it goes back to the first step.
|
||
Comment 27•3 years ago
|
||
(In reply to lapineige from comment #10)
(In reply to krq from comment #9)
Same problem. A couple of tabs open. Using inspector in one of the tabs and refreshing the page every now and then. After a while -10gigs of ram. I have 20 tabs in safari and it doesn't take even a half of that. There were almost no addonds installed because I don't use ff on regular basis.
Could you try to disable Hardware Acceleration (in Settings, search for that keyword) and see if this also "solves" the issue in your situation ?
Sorry for the delayed response. I think it was related to the dev-tolls memory leak issue(different issue)
|
||
Updated•2 years ago
|
Description
•