1962210 - Outdated ANGLE library! (GPU process extreme memory leak / RAM and VRAM)

Status: REOPENED

(Core :: Graphics, defect, P1)

Description

[Jen1]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:137.0) Gecko/20100101 Firefox/137.0

Steps to reproduce:

Starting from Firefox version 137, the GPU process exhibits excessive memory usage, both in RAM and VRAM, which continues to increase over time. This issue seems to be related to the use of an outdated ANGLE library (libEGL.dll and libGLESv2.dll) that Firefox has been using for an extended period. The current version in use is 2.1.19739, while the oldest certified version for OpenGL ES 3.2 is ANGLE 2.1.2.21688, as per the information available at Chromium's ANGLE repository https://chromium.googlesource.com/angle/angle. In contrast, Microsoft Edge is utilizing a much newer version of the ANGLE library, which may contribute to better performance and stability.

Using about:processes, I can manually close the GPU process, which causes it to restart with significantly lower memory usage. This action temporarily releases the excessive memory consumption.

Actual results:

The GPU process memory usage increases continuously, leading to approximately 3GB of RAM and 2GB of VRAM being consumed (or more, depending on usage duration).

Expected results:

The GPU process should maintain stable memory usage without significant increases over time.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Graphics' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[nova_2k1]

same issue on nightly for at least 2 weeks
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:139.0) Gecko/20100101 Firefox/139.0

GPU process memory usage slowly rises and after around 20 hours video playback gets sluggish and the process crashes and restarts.

Comment 3

[Jen1]

I would like to confirm that I have also observed the extreme GPU process memory leak in version 138. Additionally, I noticed that Firefox has been using ANGEL library version 2.1.1973X since April 2023 (starting from version 112), which may be related to the ongoing problem.

In contrast, Microsoft Edge continues to update its ANGEL libraries and is currently using version 2.1.4XXXX. I want to highlight that I have not experienced any similar memory leak behavior in Edge. The GPU process in Edge maintains normal RAM and VRAM usage, and it consistently evicts memory when closing tabs.

Comment 4

[Ashley Hale [:ahale]]

I've seen instability on my computer that fits this as well, where if Firefox Nightly had been running for more than a day my PC gaming framerates would sometimes be terrible (120->30) until I minimized all desktop windows (and could then unminimize them again and it would be okay), or restarted Firefox, if I didn't do either of those things then eventually the monitors would turn off and the computer would reboot, which seems to happen a lot sooner (within a few hours) if I have two USB audio devices connected (normally I have a Schitt Hel2 with headphones plugged into it, but when I also plug in a Focusrite Solo with no instrument plugged into it, my computer tends to freeze like that and reboot within a few hours every time - I am not sure how USB audio is involved).

Updating ANGLE isn't an easy option as it takes a couple weeks of solid effort, so we'll be looking for more timely mitigations while considering that.

Comment 5

[Jen1]

A quick solution can be achieved by setting webgl.disable-angle to true. This will disable ANGLE rendering.

Comment 6

[Ashley Hale [:ahale]]

That would indeed disable ANGLE for WebGL, but not for WebRender, I had assumed this leak was in WebRender on ANGLE from your description above - do you have an example WebGL page that is triggering this leak?

I'm aware of a video memory leak bug when playing multiple videos in a session described in https://bugzilla.mozilla.org/show_bug.cgi?id=1952340 that can leak enough memory to be a problem, so in my repro attempts I've mostly seen that leak (in a single day with videos playing on music.youtube.com and some other random youtube things I have seen more than 7GB of vram in used by Firefox itself and another 2GB by Dwm.exe on shared surfaces owned by Firefox).

Comment 7

[Jen1]

I am testing two profiles and have set the webgl.disable-angle setting to true in one of them. I've noticed a significant decrease in both RAM and VRAM usage for the GPU process in the profile with WebGL ANGLE rendering disabled. Additionally, memory (both RAM and VRAM) appears to be evicted more frequently. I have someone who can confirm this observation.

I haven't focused on any specific websites; my browsing is mostly normal, with a lot of YouTube. I don't play games or use 3D websites while browsing.

However, when I tested the ANGLE library for Edge in Firefox, I observed a significant increase in memory leaks—almost doubling the usage to 6 GB of RAM and 3 GB of VRAM. This leads me to believe that the ANGLE library could be the cause of the memory leak.

Comment 8

[Kelsey Gilbert [:jgilbert]]

It is really unclear how an older version of ANGLE might negatively affect what you say. Usually it is updates to libraries that cause new issues, while older but stable versions is generally safer from the perspective of regressions.
In particular, you shouldn't ANGLE causing new problems this year since we've had it for years.

I don't think there's evidence here that proves a causation between an older version of ANGLE, and I'm unsure how you came to this conclusion.
The issue you're having that instead sounds like bug 1952340, so I'm going to dupe this against that.

Comment 9

[Jen1]

I initially reported this bug due to a significant memory leak observed while testing the ANGLE library for Edge (exact version not recalled) in Firefox version 137. At that time, I suspected the issue was related to the ANGLE library. For further context, please refer to this comment: Bugzilla Comment.

Since then, I have tested the ANGLE library for Edge version 2.1.41232.0 in Firefox version 139, and I am pleased to report that the memory leak for the GPU process is much less pronounced. Additionally, memory is sometimes evicted, which never happened with the ANGLE library in Firefox.

Based on these findings, I am confident that the issue is affected partially in some way due to the outdated version of the ANGLE library used by Firefox. Testing different versions of the ANGLE library from Edge will change the behavior and performance of the Firefox GPU process.

Thank you for your attention to this matter.

Comment 10

[Jen1]

Attachment: ANGLE.zip

I will attach a screenshot as proof of the memory eviction issue related to the GPU process. Additionally, I am also attaching the ANGLE library version 2.1.41232.0 for anyone who wants to test this further

Comment 11

[Jen1]

I have tested the ANGLE library version 2.1.41232.0 for a long time now, and I can confirm that the memory leak issue has been resolved. The performance has noticeably improved, with resource usage significantly reduced. It is now on par with, if not better than, the performance prior to version 137, particularly in terms of both RAM and VRAM usage.

However, I would like to point out that despite these improvements, the GPU process in Firefox still consumes more RAM and VRAM compared to Edge. This remains a concern that may require further investigation.

Comment 12

[Jen1]

I have conducted a comparison of the ANGLE library extensions between versions 2.1.41232.0 and 2.1.19739. Below are my findings regarding the differences in extensions:

Extensions Missing from ANGLE 2.1.41232.0

WebGL 1 Driver Extensions:

GL_ANGLE_base_vertex_base_instance
GL_ANGLE_multi_draw
GL_EXT_base_instance
GL_EXT_multi_draw_indirect

WebGL 1 Driver WSI Info:

EGL_ANGLE_windows_ui_composition

WebGL 2 Driver Extensions:

GL_ANGLE_base_vertex_base_instance
GL_ANGLE_multi_draw
GL_EXT_base_instance
GL_EXT_multi_draw_indirect

WebGL 2 Driver WSI Info:

EGL_ANGLE_windows_ui_composition

Extensions Added to ANGLE 2.1.19739

WebGL 1 Driver Extensions:

GL_ANGLE_blob_cache
GL_ANGLE_clip_cull_distance
GL_ANGLE_polygon_mode
GL_ANGLE_program_binary_readiness_query
GL_ANGLE_renderability_validation
GL_ANGLE_shader_pixel_local_storage
GL_ANGLE_shader_pixel_local_storage_coherent
GL_ANGLE_stencil_texturing
GL_APPLE_clip_distance
GL_ARM_rgba8
GL_EXT_clip_cull_distance
GL_EXT_conservative_depth
GL_EXT_depth_clamp
GL_EXT_polygon_offset_clamp
GL_EXT_render_snorm
GL_EXT_texture_border_clamp
GL_EXT_texture_mirror_clamp_to_edge
GL_KHR_robustness
GL_NV_shader_noperspective_interpolation
GL_OES_required_internalformat
GL_OES_sample_variables
GL_OES_shader_multisample_interpolation
GL_OES_texture_storage_multisample_2d_array

WebGL 1 Driver WSI Info:

EGL_ANGLE_memory_usage_report
EGL_ANGLE_platform_angle_d3d_luid
EGL_ANGLE_platform_angle_device_type_swiftshader
EGL_ANGLE_platform_angle_vulkan
EGL_ANGLE_platform_angle_vulkan_device_uuid
EGL_ANGLE_no_error

WebGL 1 Extensions:

EXT_depth_clamp

WebGL 2 Driver Extensions:

GL_ANGLE_blob_cache
GL_ANGLE_clip_cull_distance
GL_ANGLE_polygon_mode
GL_ANGLE_program_binary_readiness_query
GL_ANGLE_renderability_validation
GL_ANGLE_shader_pixel_local_storage
GL_ANGLE_shader_pixel_local_storage_coherent
GL_ANGLE_stencil_texturing
GL_APPLE_clip_distance
GL_ARM_rgba8
GL_EXT_clip_cull_distance
GL_EXT_conservative_depth
GL_EXT_depth_clamp
GL_EXT_polygon_offset_clamp
GL_EXT_render_snorm
GL_EXT_texture_border_clamp
GL_EXT_texture_mirror_clamp_to_edge
GL_KHR_robustness
GL_NV_shader_noperspective_interpolation
GL_OES_required_internalformat
GL_OES_sample_variables
GL_OES_shader_multisample_interpolation
GL_OES_texture_storage_multisample_2d_array

WebGL 2 Driver WSI Info:

EGL_ANGLE_memory_usage_report
EGL_ANGLE_platform_angle_d3d_luid
EGL_ANGLE_platform_angle_device_type_swiftshader
EGL_ANGLE_platform_angle_vulkan
EGL_ANGLE_platform_angle_vulkan_device_uuid
EGL_ANGLE_no_error

WebGL 2 Extensions:

EXT_depth_clamp

I suspect that the missing extension, EGL_ANGLE_memory_usage_report, may be the cause of the memory leak According to the documentation here. Although the documentation pertains to Vulkan, the extension shares the same name and plays a crucial role in managing GPU memory, particularly in freeing up GPU resources.

Comment 13

[Jen1]

Bug 1971159

Comment 14

[Bob Hood [:bhood]]

Setting to stalled until we have cycles available to continue working the issue.

Comment 15

[Jen1]

Attachment: ANGEL.zip

ANGEL version 2.1.25519.0 from Chrome seems to work smoother with Firefox 140.

Comment 16

[Glenn Watson [:gw]]

We will working on an ANGLE update for Firefox this half.

Comment 17

[Brad Werth [:bradwerth]]

We'll update our ANGLE library, and when we've created the Bug referencing the specific ANGLE version, I'll link the Bug here.

Comment 18

[Jen1]

Attachment: ANGLE binary 2.1.25655.0.zip

If updating the binaries requires significant time and effort, have you considered using Google's libraries? The source is open, and it might be worthwhile to explore a partnership with Google, as they may be open to collaboration. I am currently using ANGLE binary version 2.1.25655.0 from Google, and I've found it to be quite effective—better than the previous version 2.1.25519.0. Additionally, I've noticed that many open-source applications utilize binaries from Google, although I'm not certain if they have obtained explicit permission.

Comment 19

[jonathan]

Just for info, I tried Angle versions 2.1.25655.0 and 2.1.41232.0 by replacing the libEGL.dll and libGLESv2.dll files in C:\Program Files\Mozilla Firefox (then restarting Firefox), but it changed nothing. The GPU thread rapidly takes 3GB, and its memory consumption remains high, even after a night of inactivity.
Windows 10 x64. Firefox 140.0.4. Intel IGP - HUD Graphics 630. Drivers are up-to-date. I also tried to downgrade the drivers of my IGP, but it doesn't help.

Comment 20

[Jen1]

You wanna check also bug 1971159

Comment 21

[Revertron]

I experience the same as @Jen1, but I have Radeon RX 5700XT on the same system.

The only thing that I notice with those new ANGLE libraries that Firefox becomes more snappier, I think. But vRAM leaks the same.

Comment 22

[Jen1]

You seem to be mixing up two issues. If you're watching videos in one tab, this might resolve your problem. However, if you're frequently opening and closing new tabs, you should check bug 1971159

Comment 23

[Revertron]

(In reply to Jen1 from comment #22)

You seem to be mixing up two issues. If you're watching videos in one tab, this might resolve your problem. However, if you're frequently opening and closing new tabs, you should check bug 1971159

Yes, I know. I'm watching all the bugs related to vRAM leaks and this one too :)

Comment 24

[Jen1]

@Revertron In my case, it is not just snappier; it definitely fixed the memory leak issue. The GPU process used to reach 6GB of RAM and 3GB of VRAM, which required me to restart it. Now, I am watching hundreds of YouTube videos (on a list in one tab), and the GPU process stays at 500MB of RAM and almost the same amount of VRAM. However, it has been a long time since I tried Firefox's ANGLE binaries.

Comment 25

[Jen1]

This reminds me that watching YouTube videos in a list can lead to high I/O between the GPU process and the web process. There is no such issue when watching just one video, so I might consider starting a new bug report.

Comment 26

[antzpantz]

(In reply to Jen1 from comment #25)

This reminds me that watching YouTube videos in a list can lead to high I/O between the GPU process and the web process. There is no such issue when watching just one video, so I might consider starting a new bug report.

I'm getting this too.

Windows 10 x64. Firefox 141.0, AMD Ryzen 5 5600X - Intel ARC A750 8GB (latest graphics)
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:141.0) Gecko/20100101 Firefox/141.0

Comment 27

[Jen1]

Attachment: ANGLE 2.1.25769.0.zip

(In reply to antzpantz from comment #26)

I'm getting this too.

Windows 10 x64. Firefox 141.0, AMD Ryzen 5 5600X - Intel ARC A750 8GB (latest graphics)
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:141.0) Gecko/20100101 Firefox/141.0

It's funny, but since I made that comment, I haven't encountered the issue again. The only thing I remember doing is updating to ANGLE version 2.1.25769.0. If you'd like to give it a try and let me know if the issue is resolved, it seems that this ANGLE library might be causing another problem. As long as I'm not experiencing the issue, I won't be filing a bug report.

Comment 28

[antzpantz]

Hi there, so I've been running this for a while and have noticed some better performance difference but some other tab(?) is chewing up my RAM. How do I try to trace which tab is overutilising the GPU and memory? The profiler only seems to show calls but not where they originate?

(In reply to Jen1 from comment #27)

Created attachment 9504699 [details]
ANGLE 2.1.25769.0.zip

(In reply to antzpantz from comment #26)

I'm getting this too.

Windows 10 x64. Firefox 141.0, AMD Ryzen 5 5600X - Intel ARC A750 8GB (latest graphics)
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:141.0) Gecko/20100101 Firefox/141.0

It's funny, but since I made that comment, I haven't encountered the issue again. The only thing I remember doing is updating to ANGLE version 2.1.25769.0. If you'd like to give it a try and let me know if the issue is resolved, it seems that this ANGLE library might be causing another problem. As long as I'm not experiencing the issue, I won't be filing a bug report.

Comment 29

[Ashley Hale [:ahale]]

(In reply to antzpantz from comment #28)

Hi there, so I've been running this for a while and have noticed some better performance difference but some other tab(?) is chewing up my RAM. How do I try to trace which tab is overutilising the GPU and memory? The profiler only seems to show calls but not where they originate?

The only attribution we have is the GPU process in about:memory which doesn't distinguish which window or tabs are responsible, but you can at least see the totals such as gpu-dedicated there.