1798074 - High commit, leading to system instability

Status: UNCONFIRMED

(Core :: Graphics, defect, P1)

Description

[jkaelin]

Attachment: memory-report.json.gz

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:106.0) Gecko/20100101 Firefox/106.0

Steps to reproduce:

Browse web with lots of tabs.

Actual results:

System hit the upper limit of commit at 47.9GB out of 48GB and starts refusing allocation requests.

Expected results:

Firefox releases commit

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:glandium, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 2

[Mike Hommey [:glandium]]

There's only about 16.7GB of heap-mapped in the attached report, + about 1.5GB associated with the GPU, so I'm not sure where the extra 29GB used would be coming from. It would seem plausible that there could be some leak in on the GPU driver side. Moving this over to graphics for possibly more digging.

Comment 3

[jkaelin]

Attachment: Firefox 120.0.1 anonymized memory report

Current memory report. Still seems to be an issue, since I have to restart the browser occasionally when the swapping gets too bad. Currently at 55GB/102GB commit, and I only have 32GB RAM.

Some of the anonymized processes are just Youtube. I also run the NoScript and uBlock Origins extensions.

webIsolated=https://youtube.com (pid 7884)
─2,095.79 MB (00.00%) ++ commit

977.73 MB (100.0%) -- heap-committed
├──916.12 MB (93.70%) ── allocated
└───61.60 MB (06.30%) ── overhead

Comment 4

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:bhood, could you have a look please?

For more information, please visit BugBot documentation.

Comment 5

[Bob Hood [:bhood]]

Dropping into Graphics triage for a team look-see.

Comment 6

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:bhood, could you have a look please?

For more information, please visit BugBot documentation.

Comment 7

[Bob Hood [:bhood]]

Can you launch Task manager (taskmgr.exe) on your machine and give us the result (as a screen shot) of the memory consumption of all the Firefox processes when it is consuming all this RAM? Please make sure you enable all of the memory column types (View -> Select Columns) so we get values for all of the following in the capture:

• Memory - Working Set
• Memory - Peak Working Set
• Memory - Working Set Delta
• Memory - Private Working Set (probably already selected)
• Memory - Commit Size
• Memory - Paged Pool
• Memory Non-paged Pool

Comment 8

[jkaelin]

Attachment: firefox90of102CommitCapture.PNG

I recently closed out all my tabs using the OneTab extension, so attempting to recreate it by loading ~800 tabs back up.

Basically turned my computer into a slideshow with all the swapping at 90GB/102GB commit. But didn't hit the commit limit this time(would have hit my old commit limit of 48GB and started crashing things).

File > Exit took about 20 minutes after taking the screenshot.

Then the computer is at 22GB/102GB commit when Restore previous session is used. (Firefox at about 800MB RAM, 1.3GB commit with the 800 tabs present but unloaded.)

Comment 9

[Kelsey Gilbert [:jgilbert]]

Can you add the "Dedicated GPU memory" and "Shared GPU memory" columns and give us another screenshot?
Columns can be added by right-clicking one of the existing column headings.

Comment 10

[jkaelin]

Attachment: firefox41of102CommitCapture.PNG

I haven't completely blown up the PC this browsing session, so the 800+ tabs are mostly unloaded.

Comment 11

[Ashley Hale [:ahale]]

Attachment: Shared GPU memory with lots of tabs on a dev machine

On one of my dev machines I can see that having a lot of tabs and windows open leads to a rather alarming amount of Shared GPU memory (66GiB) on a long-running Firefox Nightly process, attaching it for comparison. That is slightly more memory than this machine physically has.

Comment 12

[Ashley Hale [:ahale]]

Attachment: memory report for lots of tabs and windows on a dev machine

Here is the about:memory report for that dev machine, only 2.0GiB commit, so I don't think I'm encountering the same issue here.

Comment 13

[Ashley Hale [:ahale]]

I can get a decent increase in commit by playing a youtube video and a twitch stream at the same time (+300MiB) along with +3GiB of shared GPU memory reported on Task Manager - Details mode.

My guess is that ublock origin may be causing youtube to leak js objects for some RPCs that never complete, as I have seen that happen before, which can cause a rather rapid increase in commit I think, though I can't repro that leak when I try currently, it may be an issue of that nature.

Troubleshooting - you might want to try toggling it off only for youtube and see what happens to commit, as it may be that issue, if so at least it can be narrowed down a bit.

Comment 14

[Ashley Hale [:ahale]]

One more idea - if you close some of the tabs (or windows with lots of tabs), does the commit go down reasonably or does it stay high?

I just closed a bunch of tabs and my commit went from 2.0GiB to 1.3GiB, which is approximately what I expected to happen, but if there is a significant drop when closing certain tabs it may be that those tabs are doing something we should be testing for.

Oddly when closing a bunch of tabs and windows, the GPU Shared Memory did not go down in the task manager Details view, still at 66.9GiB, but the total GPU shared memory usage reported in the Performance tab of task manager is only 2.1/31.9GiB, so there is something odd about how the Details view counts GPU memory for processes.

Comment 15

[jkaelin]

Commit does go down as I close active tabs, yes.

Disabling uBlock Origin and closing Youtube tabs does seem to release commit from the GPU process at least. Was seeing drop of 24MB vs 1MB without disabling.

Comment 16

[Glenn Watson [:gw]]

Bumping to S2 as this is a major issue and we don't know how many users are affected.

Comment 17

[Ashley Hale [:ahale]]

I'm still trying to figure out a way to repro this and dig into the root cause.

Comment 18

[Bob Hood [:bhood]]

Needs a summary here, Ashley.

Comment 19

[Ashley Hale [:ahale]]

Can you go to about:support and copy the text (either of the two buttons at the top) and paste in an attachment on this bug?

I see RadeonSoftware in the task manager screenshot so I assume you have an AMD GPU, I've been investigating a decoder memory leak on AMD 24.* drivers in https://bugzilla.mozilla.org/show_bug.cgi?id=1847453#c47 so this may be a duplicate of that bug, but it would help to know what driver you are using.

I'll probably put a summary on that bug rather than this bug as the investigation there has been much more fruitful.

Comment 20

[jkaelin]

Attachment: about:support

Requested by :ahale

Comment 21

[xse]

Firefox 126.0 (Build ID 20240509170740) mozilla-MSIX Windows 11
AMD Ryzen 7 PRO 6850U with Radeon Graphics

Firefox open for a while, browsing, watching videos when computer goes

"You're running low on disk space"

gpu process sitting at over 20GB reserved memory -> about:memory screenshot

Comment 22

[Ashley Hale [:ahale]]

(In reply to jkaelin from comment #20)

Given that the D3D driver version shows up as 31.0.22023.1014 in that about:support (whereas my latest system shows 31.0.24033.1003), I am assuming that is 22.* driver series which didn't seem to have the memory leak problem on decoder devices when I tested it at one point, that said it is also Ryzen PRO graphics whereas mine is regular Ryzen graphics, so the drivers may behave differently.

I am still wondering if this memory usage is a result of that leak I described in https://bugzilla.mozilla.org/show_bug.cgi?id=1847453#c47 however.

Comment 23

[Ashley Hale [:ahale]]

(In reply to xse from comment #21)

Firefox 126.0 (Build ID 20240509170740) mozilla-MSIX Windows 11
AMD Ryzen 7 PRO 6850U with Radeon Graphics

Firefox open for a while, browsing, watching videos when computer goes

"You're running low on disk space"

gpu process sitting at over 20GB reserved memory -> about:memory screenshot

That roughly matches what I saw in https://bugzilla.mozilla.org/show_bug.cgi?id=1847453#c47 , so it is probably the same issue with decoder devices.

Comment 24

[jkaelin]

(In reply to Ashley Hale [:ahale] from comment #22)

(In reply to jkaelin from comment #20)

Given that the D3D driver version shows up as 31.0.22023.1014 in that about:support (whereas my latest system shows 31.0.24033.1003), I am assuming that is 22.* driver series which didn't seem to have the memory leak problem on decoder devices when I tested it at one point, that said it is also Ryzen PRO graphics whereas mine is regular Ryzen graphics, so the drivers may behave differently.

I am still wondering if this memory usage is a result of that leak I described in https://bugzilla.mozilla.org/show_bug.cgi?id=1847453#c47 however.

Driver is Adrenaline 23.11.1, and my GPU is a Radeon RX 6600 8GB, on Windows 10.

https://www.amd.com/en/resources/support-articles/release-notes/RN-RAD-WIN-23-11-1.html

Comment 25

[Ashley Hale [:ahale]]

NeedInfo - Can you take a look at the reporting guide I wrote in https://bugzilla.mozilla.org/show_bug.cgi?id=1902566#c0 and see if this memory leak is repeatable (i.e. gets worse each time you do the actions that trigger it)?

So far we're keeping track of a decoder device leak in https://bugzilla.mozilla.org/show_bug.cgi?id=1847453 but I am not sure if this is the same issue so it would be extremely useful to know if it's a different root cause that we need to track down.

Comment 26

[jkaelin]

Attachment: dec2024memory-report.json.gz

The bulk of the commit isn't tied up by the GPU process, it seems to be tied up by processes attached to youtube.com. Which probably tracks with orphan DOM nodes from Ublock Origin.

Possibly related, when I Restore Previous Session, it seems to be trying to load up ALL the tabs at the same time. Which leads to me getting NoScript XSS redirect notifications about "possible DDOS attempt?" from sites being hit. This is also when commit tends to spike highest, and has led to crashing out as I run out of pagefile.sys.

Steps to reproduce are probably just keep opening Youtube tabs and watching videos. Reddit may do it too, with their in-line video player.

Firefox 133.0
Ublock Origin 1.61.2
NoScript 11.5.2