1828587 - Animated theme produces steady memory leak in occluded or minimized windows [leaking render-texture-hosts via RecvAddSharedSurface]

Status: VERIFIED FIXED

(Core :: Graphics: WebRender, defect, P1)

Description

[Avwon]

Attachment: memory-report.json.gz

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/112.0

Steps to reproduce:

Opened firefox, tabs restored w/ Youtube/Videos immediately | OR | Opened Firefox with no tabs loaded with video, then opened Twitch VOD | OR | Opened Firefox with no tabs loaded with video, then opened Kick VOD | OR | Opened a separate Firefox Profile and watched a video on there

Actual results:

In all cases above, seemingly, when Video related content is loaded after 30 or so minutes Firefox will consume all RAM. With Youtube at a rate of 13 megabytes a second. With other video providers at different rates. This RAM usage is not correctly reported in Task Manager, but is visible in Resource Monitor under Firefox. I don't have to watch anything. Just have the Video tab loaded and stopped the video and it will continue to rise even when buffer percentage is reached, as if some infinite cycle going on. Firefox seems to run normally for the most part, sometimes delayed input. RAM is freed when Firefox is closed. When closing Firefox, video seems to trail on for a second or two before shutting down, as if it is unloading a bunch of data in the background first. https://i.imgur.com/L46Fubn.png

Expected results:

RAM usage reflects tabs loaded and memory gets unloaded or stays steady depending on usage. Normal use around 3-6 gigs depending on workload. Not rising until 100% of memory is reached (22 gigs of in-use memory).

Comment 1

[Avwon]

Let me know if you need any more data. Happy to help!

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Performance' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

This bug was moved into the Performance component.

:kuriosevwon, could you make sure the following information is on this bug?

• For slowness or high CPU usage, capture a profile with http://profiler.firefox.com/, upload it and share the link here.
• ✅ For memory usage issues, capture a memory dump from about:memory and attach it to this bug.
• Troubleshooting information: Go to about:support, click "Copy raw data to clipboard", paste it into a file, save it, and attach the file here.

If the requested information is already in the bug, please confirm it is recent.

Thank you.

Comment 4

[Avwon]

Attachment: firefox about:support raw data

Requested about:support data dump

Comment 5

[Avwon]

https://share.firefox.dev/3GY1G8s Requested profile capture. Not that the application really runs slow. Just memory usage keeps growing and overflows to physical drives.

Also, additional community discussion can be found at https://www.reddit.com/r/firefox/comments/12q6m1z/firefox_hoarding_massive_amount_of_memoryram/

Comment 6

[Yannis Juglaret [:yannis]]

Hello, thank you for your report! Can you try the following steps:

• Download VMMap.
• Reproduce the issue.
• Navigate to about:processes and identify the process ID for the GPU process, it should show as GPU (XXXX) where XXXX is the process ID.
• Launch VMMap64.exe and select the Firefox GPU process by process ID.
• Capture a screenshot of the VMMap window that opens and share the screenshot here.
• Ideally, if you can also click File, Save As to save the data into a file, then share that file with me at [e-mail], that is even better.
• In the VMMap window, if you click Tools, Empty Working Set, does this temporarily solve the issue?

Comment 7

[Aelux]

Attachment: Aelux_VMMap.png

Comment 8

[Aelux]

Attachment: Aelux_firefox.mmp

Comment 9

[Aelux]

Hi Yannis,

I believe that I am experiencing the same bug and the cause of the bug I have found is simply having a window minimized, possibly with media on it I am unsure of that detail. And for me it clears itself out when I focus on that window again but doing that every 20 minutes or simply having to make sure no windows are minimized ever isn't ideal.

I have performed your last steps if you wish to check mine and have more data on this. See attached.

Comment 10

[Aelux]

Additionally emptying the working set didn't resolve the issue and as per usual, I need to open any minimized windows

Comment 11

[Sylvestre Ledru [:Sylvestre]]

[Tracking Requested - why for this release]: might be a big deal according to Yannis

Comment 12

[Aelux]

Mine was in fact v112.0.1 and for me I didn't notice it until this update

Comment 13

[Gabriele Svelto [:gsvelto]]

The attached about:memory shows massive amounts of committed memory which are unused. That's why Task Manager doesn't show it, the memory has been committed but hasn't been touched so it's not backed by physical memory. Unfortunately we'll end up running out of commit space anyway which will eventually lead to a crash, it seems we have a commit-space leak somewhere.

Comment 14

[Gabriele Svelto [:gsvelto]]

One more piece of information: the commited memory appears in the main process and in the GPU process in similar amounts, this could be related to passing data between the two.

Comment 15

[Gabriele Svelto [:gsvelto]]

Here's our likely culprit from the report:

28,605.56 MB (47.38%) ── render-texture-hosts

It seems we're leaking textures that have been shared from the main process to the GPU process (the mappings are read/write in the main process and read-only in the GPU process). We need to get hang of somebody from the graphics team.

Comment 16

[Gabriele Svelto [:gsvelto]]

Confirming the issue.

Comment 18

[Gabriele Svelto [:gsvelto]]

I'm reproducing on my machine by watching a YouTube video. On Windows commit-space creeps up continuously in both the main process and GPU process. Closing the tab containing the video releases all the committed memory in the main process but not in the GPU process which remains at the same level.

Comment 19

[Gabriele Svelto [:gsvelto]]

FYI I'm running a bisection, if someone else who knows the graphics code better than me wants to do a more direct investigation just go ahead and don't wait for me.

Comment 20

[Emilio Cobos Álvarez (:emilio)]

Is the minimized window a requirement? I wonder if it is bug 1768495? That involves the compositor and minimized/occluded windows.

Comment 21

[Yannis Juglaret [:yannis]]

(In reply to Gabriele Svelto [:gsvelto] from comment #15)

Here's our likely culprit from the report:

28,605.56 MB (47.38%) ── render-texture-hosts

These textures are stored in mozilla::wr::RenderThread::mRenderTextures in case you need to track them.

Comment 22

[nasko235679]

Having exact same issue, can confirm it occurs when a window that has recently played media (for example a YT video that has ended) is staying minimized. Task manager doesn't show the massive RAM leak, but software such as Process Explorer pinpoints the leak to firefox. When about:memory is run I'm getting massive leakage in the web-renderer section.
https://bugzilla.mozilla.org/show_bug.cgi?id=1828793 for reference.

Comment 23

[Gabriele Svelto [:gsvelto]]

I've not been able to reproduce reliably on my machine, can you share the exact steps that lead to the problem (including the video causing it in case it only happens on some videos rather than on others)?

Comment 24

[Gabriele Svelto [:gsvelto]]

Also, the original reporter had several YouTube- and Twitch-related extensions, is that also the case?

Comment 25

[Aelux]

I have tried without any extensions at all and still encounter my issue, and then on that Reddit thread the original poster shared a lot of people say they have the issue without any Add-ons as well.

Comment 26

[Gabriele Svelto [:gsvelto]]

Thanks. If anyone experiencing this issue consistently has some time on their hands it would be super-helpful if they could run mozrgression for us:

• Install mozregression-gui from here: https://github.com/mozilla/mozregression/releases
• Launch it
• Click on the scissors icon in the top left (Run a new bisection)
• In the wizard leave everything to its default values except for "Repository" which needs to be mozilla-central
• Click next, this will take you to the "Profile selection" page
• Click next again, this will test using a new profile and won't pollute your main Firefox profile
• In the "Build selection" page set "Last known good build" to release 111 and "First known bad build" to release 112 then click next
• mozregression will download and launch a version of Firefox, try reproducing the problem. If you can reproduce close Firefox and click on the "Bad" button in mozregression window, if the problem is gone click on the "Good" button
• Following this procedure should eventually lead you to the version that first contained the problem, attach this information here.

If anybody can do this it would help us immensely in identifying the issue.

Comment 29

[BugBot [:suhaib / :marco/ :calixte]]

The bug is marked as tracked for firefox112 (release), tracked for firefox113 (beta) and tracked for firefox114 (nightly). We have limited time to fix this, the soft freeze is in 13 days. However, the bug still isn't assigned.

:fdoty, could you please find an assignee for this tracked bug? If you disagree with the tracking decision, please talk with the release managers.

For more information, please visit auto_nag documentation.

Comment 30

[nasko235679]

(In reply to Gabriele Svelto [:gsvelto] from comment #24)

Also, the original reporter had several YouTube- and Twitch-related extensions, is that also the case?

(In reply to Gabriele Svelto [:gsvelto] from comment #23)

I've not been able to reproduce reliably on my machine, can you share the exact steps that lead to the problem (including the video causing it in case it only happens on some videos rather than on others)?

It just happened again. So here is what happens:

1.I have multiple monitors. I have multiple firefox instances (windows) open with a few tabs in each window. I minimize one of the windows on one monitor and that's the window that starts to leak memory. SHOCKINGLY this time there was no video in the window that was leaking memory. There was reddit (I use the classic grid so no videos autoplay and there are no video players in the open tab), my gmail and this bug's url. Once I maximized the window the memory leak stopped and memory and page file went back to normal.

In terms of extensions I have YouTube Enhancer, AdBlocker Ultimate, uBlock Origin, betterTTV and Return YouTube Dislike.
Hope this helps.

Comment 31

[Yannis Juglaret [:yannis]]

Edit: No more required thanks to comment 45.

Windows only: Are able to reproduce the issue with this custom build? If yes, please navigate to about:memory while the memory is still high, and click Measure. That should create a report.txt file on your Desktop. If you could then share that file here as an attachment, that should help us diagnose this issue. Thank you!

Note: This is custom behavior that is specific to this custom build. I am not asking for the contents of about:memory. The report.txt file will contain specific information that is relevant for the investigation.

Comment 34

[Andrew McCreight [:mccr8]]

Bug 1828075 has the interesting detail that they are seeing the issue with a second instance of Firefox opened, but minimized.

Comment 35

[Emilio Cobos Álvarez (:emilio)]

(In reply to Andrew McCreight [:mccr8] from comment #34)

Bug 1828075 has the interesting detail that they are seeing the issue with a second instance of Firefox opened, but minimized.

That does make me suspect strongly about bug 1768495 again fwiw... Can someone repro without another window minimized or occluded by another window?

Comment 36

[Aelux]

Attachment: Aelux_memory-report.json.gz

Comment 37

[Aelux]

Attached the report with your custom build, also worth mentioning I too have multiple screens and multiple windows open when getting the issue.

Comment 38

[Andrew McCreight [:mccr8]]

Aelux_memory-report looks similar to the other one.

Parent process:
4,927.30 MB ── resident
4,472.57 MB ── shmem-allocated
4,472.57 MB ── shmem-mapped

GPU process:
4,541.99 MB (47.95%) ── render-texture-hosts
└──4,480.06 MB (47.30%) -- images/mapped_from_owner
├──4,473.41 MB (47.23%) -- pid=39604
│ ├──4,461.33 MB (47.10%) ── image(640x360, compositor_ref:2, creator_ref:1)/decoded-nonheap [5076]

  4,666.75 MB ── resident
      0.00 MB ── shmem-allocated
  4,505.23 MB ── shmem-mapped

Comment 39

[Andrew McCreight [:mccr8]]

(In reply to Aelux from comment #37)

Attached the report with your custom build, also worth mentioning I too have multiple screens and multiple windows open when getting the issue.

We need the report.txt from your Desktop, not the about:memory report please. Thanks!

Comment 40

[Andrew McCreight [:mccr8]]

I'm moving this to WebRender, as the leak seems like some sort of graphics thing.

Comment 41

[Aelux]

Hmm I don't seem to get that file created, it is the button under show memory reports?

Comment 42

[Aelux]

actually one sec, it went to my other drive's "desktop" which isn't normally visible to me, I'll do a fresh one and upload

Comment 43

[Jeff Muizelaar [:jrmuizel]]

Bug 1816026 seems like a possible cause. For those who can reproduce this can you try setting gfx.webrender.dcomp-video-sw-overlay-win=false in about:config, restarting Firefox and checking whether you can still reproduce?

Comment 44

[Andrew McCreight [:mccr8]]

The 3 reports I've seen with the issue (the two in this bug plus the one in bug 1828075) have an entry in the GPU process under explicit/gfx/webrender/swgl, for whatever that's worth, so I assume that means they are all using SWGL.

Comment 45

[Aelux]

Attachment: Aelux_custom_report.txt

Comment 46

[Aelux]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #43)

Bug 1816026 seems like a possible cause. For those who can reproduce this can you try setting gfx.webrender.dcomp-video-sw-overlay-win=false in about:config, restarting Firefox and checking whether you can still reproduce?

It is still happening with that set and after a restart of FF sadly

Comment 47

[Yannis Juglaret [:yannis]]

Thank you for the report.txt file (comment 45). This suggests that you have 9146 entries that were added to mozilla::wr::RenderThread::mRenderTextures through the following path, which are still alive:

xul!mozilla::wr::RenderThread::RegisterExternalImage+0xfe [/builds/worker/checkouts/gecko/gfx/webrender_bindings/RenderThread.cpp @ 891]
xul!mozilla::layers::SharedSurfacesParent::Add+0x17a [/builds/worker/checkouts/gecko/gfx/layers/ipc/SharedSurfacesParent.cpp @ 243]
xul!mozilla::layers::CompositorManagerParent::RecvAddSharedSurface+0x23 [/builds/worker/checkouts/gecko/gfx/layers/ipc/CompositorManagerParent.cpp @ 266]
xul!mozilla::layers::PCompositorManagerParent::OnMessageReceived+0x88d [/builds/worker/workspace/obj-build/ipc/ipdl/PCompositorManagerParent.cpp @ 306]
xul!mozilla::ipc::MessageChannel::DispatchAsyncMessage+0x71 [/builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp @ 1800]
xul!mozilla::ipc::MessageChannel::DispatchMessage+0x15c [/builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp @ 1729]
xul!mozilla::ipc::MessageChannel::RunMessage+0xf6 [/builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp @ 1526]
xul!mozilla::ipc::MessageChannel::MessageTask::Run+0x70 [/builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp @ 1632]
xul!nsThread::ProcessNextEvent+0x58c [/builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp @ 1233]
xul!NS_ProcessNextEvent+0x68 [/builds/worker/checkouts/gecko/xpcom/threads/nsThreadUtils.cpp @ 477]
xul!mozilla::ipc::MessagePumpForNonMainThreads::Run+0xfe [/builds/worker/checkouts/gecko/ipc/glue/MessagePump.cpp @ 330]
xul!MessageLoop::RunHandler+0x50 [/builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc @ 375]
xul!MessageLoop::Run+0x57 [/builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc @ 357]
xul!nsThread::ThreadFunc+0xe6 [/builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp @ 393]

Comment 48

[Kelsey Gilbert [:jgilbert]]

This sounds a lot like an S1.

Comment 49

[Gabriele Svelto [:gsvelto]]

I'm adding another bit of info: I've tried reproducing on my machine with dual monitors but couldn't. I've tried playing video in a separate window on the second monitor, minimizing it, restoring it, etc... and couldn't trigger that runaway behavior. However I noticed a difference between the memory report I get and the ones in the reports from the affected users: if I look at the entries under gfx > webrender > images/mapped_from_owner > pid > image(...)/decoded-nonheap I always have compositor_ref: 1 and creator_ref: 0. In their reports compositor_ref is always >1, sometimes 10+. I don't know how that happens but maybe it's part of the reason why I'm not seeing the problem with the same setup.

Comment 50

[Gabriele Svelto [:gsvelto]]

So, taking the info from comment 47 it seems we're adding more consumers to those surfaces here. If you dig into that function you'll see that it bumps the mConsumers variable here. So the question is: why does this surface have more than one consumer and could that cause the leak?

Comment 51

[Kelsey Gilbert [:jgilbert]]

As much as I love watching youtube videos for work, this doesn't repro on nightly for me, even matching the webrender-compositor:disabled, dcomp-present:disabled config. (Win11 Nightly114)
Leaking texture hosts is a great lead.

Comment 52

[Aelux]

I am also on W10 still, not sure if that matters? just noticed that about Gilbert's comment and not sure if the other people with the issue are as well?

Comment 53

[Bob Hood [:bhood]]

Aelux, I tested Fx112 (Release) & Fx114 (Nightly) on Win10 64 22H2 with an NVIDIA 2080Ti (driver 531.18) using one and two monitors, and could not reproduce. So, hard to say that the specific version of the OS is actually a factor.

Comment 54

[Yannis Juglaret [:yannis]]

Windows only: Here is a new custom build which could help us explore further the idea from comment 50. Similar to the one from comment 31: Please navigate to about:memory with this custom build while the memory is still high, and click Measure. That should create a report.txt file on your Desktop which could help us diagnose this issue. Thank you!

This new custom build should help us visualize which calls to AddConsumer and DeleteConsumer did happen on the surfaces that are still alive. We may be able to deduce which ones did not happen, for example by comparing with what happens on computers that don't reproduce the issue.

Comment 55

[Aelux]

Attachment: Aelux_Custom_report_2.txt

Comment 56

[Timothy Nikkel (:tnikkel)]

To potential reproducers: note comment 44 suggests swgl might be necessary to reproduce. The pref gfx.webrender.software should turn that on.

Comment 57

[Aelux]

(In reply to Timothy Nikkel (:tnikkel) from comment #56)

To potential reproducers: note comment 44 suggests swgl might be necessary to reproduce. The pref gfx.webrender.software should turn that on.

These are my config settings around that and they are unmodified, it claims
gfx.webrender.software=false
gfx.webrender.software.d3d11=true
gfx.webrender.software.d3d11.upload-mode=4
gfx.webrender.software.opengl=false

Comment 58

[Andrew McCreight [:mccr8]]

Could you please attach your about:support (I guess the "Copy raw data to clipboard") as an attachment? I think that might indicate what is going on with your WebRender. Thanks.

Comment 59

[Aelux]

Attachment: Aelux_about:support_raw.txt

Comment 60

[Andrew McCreight [:mccr8]]

Thanks! There are a few things I noticed in that log that sounds relevant to some of the discussion around multiple monitors and refresh rates.

"name": "WEBRENDER_COMPOSITOR",
"status": "unavailable",
"log": [{"type": "default", "status": "available"},
{"type": "runtime", "status": "unavailable",
"failureId": "NVIDIA_REFRESH_RATE_MIXED",
"message": "No DirectComposition usage"}]

"name": "WEBRENDER_DCOMP_PRESENT",
"description": "WebRender DirectComposition",
"status": "blocked",
"log": [{"type": "default", "status": "available"},
{"type": "env", "status": "blocked",
"failureId": "NVIDIA_REFRESH_RATE_MIXED",
"message": "Monitor refresh rate too high/mixed }]

Comment 61

[Aelux]

Just as a note: On a downgraded version I am using for the moment as a workaround, the graphics section of that page appears to be identical including things you mentioned such as mixed refresh rate and no composition usage. I am curious how that looks on different machine using 112.0.1 however

Comment 62

[Aelux]

I noticed that the original posters windows build was also NT and not sure if that is some correlation or not? are those of you that can't reproduce running standard builds? I know that NT builds require manual installation of a media codec pack or some such whereas it is baked into regular installs. Maybe some change with the new version of FF that uses something that us on NT don't have or that isn't updated? (just wild speculation)

Comment 63

[nasko235679]

I'm running Windows 11 Pro 22H2 perfectly standard OS, I have no idea why the bug reporter identified it as Windows 10 / NT.

Comment 64

[Haik Aftandilian [:haik]]

If anyone that can reproduce this would like to try using Firefox Nightly and report back if the problem is reproducible, that would also be a helpful data point. Nightly can be downloaded from https://www.mozilla.org/en-US/firefox/channel/desktop/. But make sure to use a separate profile when Nightly starts up so that there is no risk to your existing profile.

Comment 65

[Avwon]

@Andrew McCreight [:mccr8]

Framerate does not seem to be the issue., I just set refresh rate to the same values which resulted in no error. Same issue.

Comment 66

[Avwon]

Attachment: firefox_about-support-nightly.txt

Using Yannis Nightly custom build, comment 54, framerate of displays set to 60hz to avoid "NVIDIA_REFRESH_RATE_MIXED" failure. Memory issue persists.

Comment 67

[Ashley Hale [:ahale]]

A few ideas to explore if you have the time:

• Try turning on the pref gfx.webrender.compositor.force-enabled in the about:config page - this may directly fix the issue but it may cause flickering issues when a browser window is on the border between monitors on nvidia drivers if a high refresh rate monitor is present (we currently disable the compositor to keep this flickering from occuring on affected systems with an nvidia gpu and high refresh rate and regular refresh rate monitors side by side, but not using the compositor can cause video playback code to behave very differently, which might be at the heart of this memory issue).
• Try installing Firefox Nightly - it is two major versions ahead of the current stable and if we happen to have fixed something in that time range it would be good to confirm (and it would give you a workaround of sorts as you can sync your account in Nightly to your regular release channel browser if you wish).
• Try mozregression if you have a few hours to spend on this issue - it runs several different versions of Firefox with a temporary profile (i.e. don't expect it to stick around, it's not going to touch your regular browser experience) and prompts you to reproduce the issue in each and tell it whether the issue is present ('bad') or works fine ('good'), it would then tell you exactly which date and time the firefox code changes happened that broke this for you. Knowing this date and time when things broke would let us immediately fix the issue.

If I am able to repro this issue I would be following these same steps to figure out what is going on, but I have not been able to reproduce it yet, so your help is much appreciated if you wish to spend the time :)

Comment 68

[Aelux]

So as soon as I tried to use mozregression the issue vanished and the test builds I was using was using my profile but I disabled or removed all addons..or so I thought, an addon that doesn't show as an addon, not even in all these debug logs we provided is a theme. In my case, Dark-Space-Theme which is a highly recommended one on the store. Low and behold I see a bug on the git for that theme and they have linked an old issue for it. They recommend simply making the gif they are using for the theme into an animated png.

https://addons.mozilla.org/en-US/firefox/addon/nicothin-space/
https://github.com/nicoth-in/Dark-Space-Theme/issues/11
https://bugzilla.mozilla.org/show_bug.cgi?id=1786247

It is still interesting to me that the theme only broke with 112.0.1
why did it work fine before that update?
is it localised to just this theme or other animated ones using the same technique? and is this why more than just one person reported it?
is there just my issue which could be unrelated and a second similar more serious one? or is everyone experiencing the same one as me?

Additionally: using gfx.webrender.compositor.force-enabled didn't resolve the issue so it doesn't seem like that is related.

Comment 69

[Ashley Hale [:ahale]]

I can confirm that using that dark space theme and opening 50 maximized windows causes a steady rise in committed readwrite pages in the address space section of the main process report in about:memory, it barely shows up on Windows task manager in process view and the Firefox task manager, but the memory view in task manager shows a steadily rising amount of memory usage, if I hold alt-tab and let it cycle through all tabs the memory usage goes entirely back to normal.

What this tells me is that the WebRender thread isn't looking at these hidden windows and dealing with consuming pending pictures pushed by the main process.

Comment 70

[Yannis Juglaret [:yannis]]

(In reply to Avwon from comment #66)

Created attachment 9329780 [details]
firefox_about-support-nightly.txt

Using Yannis Nightly custom build, comment 54, framerate of displays set to 60hz to avoid "NVIDIA_REFRESH_RATE_MIXED" failure. Memory issue persists.

(Custom builds are branded as Nightly but this was based on 112 Release.)

Comment 71

[Ashley Hale [:ahale]]

Steps to Reproduce for my example in comment #69:

• Enable theme https://addons.mozilla.org/en-US/firefox/addon/nicothin-space/
• maximize window
• hold Ctrl-N (or Cmd-N on Mac) until you have some hundreds of Firefox windows all overlapping the same monitor
• open Task Manager in Windows (ctrl-shift-escape), switch to Performance tab, select the Memory tab in that
• watch memory usage steadily grow for as long as you like
• hold alt-tab until it has cycled through previews of all of the Firefox windows
• watch memory usage return to normal and start climbing again

Deactivating the theme and then using alt-tab to cycle all windows will return Firefox to normal memory usage and does not grow over time.

It's not clear to me if this can be done with paused videos in hidden windows instead of animated gifs, but we should try to confirm that, I wasn't able to repro on videos in my testing but maybe there's a trick to it.

Comment 72

[nasko235679]

DAMN!!! You guys are crazy. I'm using the exact same theme. You actually found it ya crazy bastards. Okay, I guess the solution is switching to another theme?

Comment 73

[Yannis Juglaret [:yannis]]

Attachment: consumers.txt

For what it's worth, here are the symbolized stacks from comment 55 explaining why these surfaces are still alive. It seems that only half of the calls to add consumers originating from WebRenderBridgeParent::RecvUpdateResources have a matching call to mozilla::layers::WebRenderBridgeParent::ObserveSharedSurfaceRelease. Note that there are two stacks for WebRenderBridgeParent::RecvUpdateResources: one going through this path and the other through that path, so the total number of calls is the sum of the two paths.

Edit: For more clarity, this should probably read as something like: we added 3650 surfaces, for which, out of the two "things" (threads? processes?) that registered additional consumers for them, only one has successfully executed the code path to unregister those consumers through ObserveSharedSurfaceRelease, hence the underlying textures for these surfaces have not been released.

Comment 74

[Yannis Juglaret [:yannis]]

(In reply to Emilio Cobos Álvarez (:emilio) from comment #35)

That does make me suspect strongly about bug 1768495 again fwiw... Can someone repro without another window minimized or occluded by another window?

At this point though bug 1768495 sounds very suspicious like [:emilio] suggested. Maybe pausing the compositor while we are not active prevents it from doing the job that triggers the path to reach ObserveSharedSurfaceRelease?

Comment 75

[Yannis Juglaret [:yannis]]

(In reply to nasko235679 from comment #72)

DAMN!!! You guys are crazy. I'm using the exact same theme. You actually found it ya crazy bastards. Okay, I guess the solution is switching to another theme?

For the moment that is the best workaround, but we will do our best to fix this with Firefox 112.0.2.

Comment 76

[nasko235679]

That's a good idea considering this is one of the most used themes (and highest rated ones).

Comment 77

[Aelux]

Ayy, glad I got it!
I would also suggest that in the fix it is made so themes which actually exist under the add-on tab show up in logs like all these support output files we generated because someone would have found it much sooner than me if we all could see right away we all had one "add-on" left that we all had in common that wasn't disabled

Comment 78

[Kestrel]

The Comment 0 memory report includes an animated theme, ANIMATED | Neurons so this is definitely the issue.

Regression window:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=dc16834bdd96e0fc1c355848d5c79a02d9424943&tochange=1d55841947c3560b45c9b63344c0564cb153030c

Regressed by Bug 1768495.

Comment 79

[BugBot [:suhaib / :marco/ :calixte]]

:bradwerth, since you are the author of the regressor, bug 1768495, could you take a look?

For more information, please visit auto_nag documentation.

Comment 80

[Emilio Cobos Álvarez (:emilio)]

Attachment: Bug 1828587 - Don't paint or refresh animated images on throttled refresh drivers. r=smaug

As the compositor might be paused. We don't really want to queue a bunch
of paints that are never going to be visible.

Comment 81

[Emilio Cobos Álvarez (:emilio)]

I sent a tentative patch. Try run: https://treeherder.mozilla.org/jobs?repo=try&revision=286a65ddbc10ca378f0cb502840d87543b9f06a5

Comment 82

[Emilio Cobos Álvarez (:emilio)]

An alternative approach would be to throttle rather than pause the compositor, which is what Wayland does.

Comment 83

[Emilio Cobos Álvarez (:emilio)]

Attachment: Bug 1828587 - [release] Back out bug 1768495. r=smaug,bradwerth

This backs out the regressing bug. Backout applied cleanly, so even
though a targetted change in CanonicalBrowsingContext would've probably
been enough, just an automated backout seems preferable.

Comment 84

[Emilio Cobos Álvarez (:emilio)]

Attachment: Bug 1828587 - [beta] Back out bug 1768495. r=smaug,bradwerth

This backs out the regressing bug. Backout applied cleanly, so even
though a targetted change in CanonicalBrowsingContext would've probably
been enough, just an automated backout seems preferable.

Comment 85

[Emilio Cobos Álvarez (:emilio)]

Okay, so we decided to revert the offender on beta/release, and land the "proper" fix on nightly (since it's not risk-free):

• Nightly: patch | try
• Beta: patch | try
• Release: patch | try

Comment 86

[Emilio Cobos Álvarez (:emilio)]

Comment on attachment 9329810 [details]
Bug 1828587 - [release] Back out bug 1768495. r=smaug,bradwerth

Beta/Release Uplift Approval Request

• User impact if declined: See comment 0. When users have animated themes and one window is occluded or minimized, we'll appear to leak memory until the window is painted (which might be never).
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: comment 71 has reliable STR
• List of other uplifts needed: none
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Backs out regressing bug.
• String changes made/needed: none
• Is Android affected?: No

Comment 87

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6f2d406b412b
Don't paint or refresh animated images on throttled refresh drivers. r=smaug

Comment 88

[Emilio Cobos Álvarez (:emilio)]

Comment on attachment 9329811 [details]
Bug 1828587 - [beta] Back out bug 1768495. r=smaug,bradwerth

Beta/Release Uplift Approval Request

• User impact if declined: See comment 0. When users have animated themes and one window is occluded or minimized, we'll appear to leak memory until the window is painted (which might be never).
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: Comment 71
• List of other uplifts needed: none
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Cleanly reverts regressing patches.
• String changes made/needed: none
• Is Android affected?: No

Comment 89

[Dianna Smith [:diannaS]]

Comment on attachment 9329811 [details]
Bug 1828587 - [beta] Back out bug 1768495. r=smaug,bradwerth

Approved for 113.0b7

Comment 90

[Dianna Smith [:diannaS]]

https://hg.mozilla.org/releases/mozilla-beta/rev/e6bc65cb318a

Comment 91

[Dianna Smith [:diannaS]]

Comment on attachment 9329810 [details]
Bug 1828587 - [release] Back out bug 1768495. r=smaug,bradwerth

Approved for 112.0.2 dot release

Comment 92

[Dianna Smith [:diannaS]]

https://hg.mozilla.org/releases/mozilla-release/rev/6b8f889688ed

Comment 93

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/6f2d406b412b

Comment 94

[Avwon]

Darn, now I feel terrible for the misdirection. All of my profiles had different themes for me to differentiate them. My barebones profile also had a animated theme to it. Thank you for isolating the issue Aelux and for the devs work on this.

Comment 95

[Sylvestre Ledru [:Sylvestre]]

Darn, now I feel terrible for the misdirection.

Don't :) It is part of the investigate process!

Comment 96

[Sylvestre Ledru [:Sylvestre]]

If folks want to test, 113.0b7 is live on beta with the change

Comment 100

[Avwon]

(In reply to Sylvestre Ledru [:Sylvestre] from comment #96)

If folks want to test, 113.0b7 is live on beta with the change

Thank you, I have used it and it appears to be resolved.

I may have ran into a separate performance issue yesterday but will confirm. I was doing some other things as well at the time which might be the actual cause.

Comment 101

[Bogdan Maris, Desktop Test Engineering]

I managed to reproduce the issue described here using steps from comment 71 and an old Nightly build from 2023-04-20. I also verified that the memory remains at a constant rate using latest Nightly 114.0a1, Firefox 113.0b8 and Firefox 112.0.2 across platforms (macOS 13, Ubuntu 22.04 and Windows 10).

Comment 105

[Avwon]

I had Firefox beta version 113.0b7 installed to fix the issue for me since it was pointed out. Since then my Firefox has automatically updated to version 114.0b3 and this exact animated theme/gpu memory issue has returned. I can provide logs again if needed.

Comment 106

[Julien Wajsberg [:julienw]]

avwon, can you please file a separate bug so that this would get the right attention?

Comment 107

[Avwon]

(In reply to Julien Wajsberg [:julienw] from comment #106)

avwon, can you please file a separate bug so that this would get the right attention?

Thanks, reported: https://bugzilla.mozilla.org/show_bug.cgi?id=1834246
Sorry, for late report, a bit busy :)

Comment 108

[Brad Werth [:bradwerth] | Out until March 3]

comment 73 indicates that this has the exact same presentation at the issue being tracked in Bug 1830753. What we've landed here, though useful, wasn't sufficient to fix the issue. Let's use Bug 1830753 to build the rest of the fix.