1654013 - Resident memory climbs when (at least) switching tabs and WebRender is enabled

Status: RESOLVED DUPLICATE of bug 1640564

(Core :: Graphics: WebRender, defect)

Description

[Paul Adenot (:padenot)]

Attachment: memory-report-wr.json.gz

• GPU
  Intel(R) HD Graphics 630
• Driver
  26.20.100.7986
• Windows 10 1909
  Build 18363.959

This is a DELL XPS15 9570. about:memory report attached, see the GPU Process section, where the resident and the rest of the numbers don't match at all.

I thought this was mapped GPU memory, but it's not, I currently have about 2.5GB resident in the GPU process, but Windows' Task Manager reports about 1.5GB graphics memory usage, out of a max of 7.9GB.

I can reproduce at will by rapidly switching tabs (for example having ctrl+tab held down).

Comment 1

[Paul Adenot (:padenot)]

Attachment: UMDH diff log

I followed this documentation from Microsoft to get the stacks that resulted in an allocation. In between the first and second invocation of the command umdh.exe -p:<PID> -f:<logfile>, I held down ctrl+tab, where <logfile> were log1.txt and log2.txt.

Then I ran umdh.exe log1.txt log2.txt after having set the environment variable with the Mozilla symbol server, and attached is the resulting file, that has a bunch of stacks that are graphics/webrender related.

Comment 2

[Jeff Muizelaar [:jrmuizel]]

I can reproduce this on my Intel HD 530

Comment 3

[Jeff Muizelaar [:jrmuizel]]

So I can get useful data out of this by using Windows Performance Recorder to trace VirtualAlloc and then viewing the result in WPA.
The "Impacting Stack" of the "VirtualAlloc Commit LifeTimes" view is the most useful part.

I see the total amount go up by 275MB during my trace time. Here are some notable parts. Identical Code Folding could make these lies
58MB of that is associated with RawVec<gpu_cache::GpuBlockData>
35MB with SceneBuilder::build_item, perhaps vec::from_iter<webrender_api::font::GlyphInstance>
26MB is from DCLayerTree::GetOrCreateFBO in CreateTexture2D
24MB is update_texture_cache

An alternative breakdown:
81MB of texture related stuff allocated with DxgkCreateAllocation
of which 27MB is DCLayerTree::GetOrCreateFBO, 24MB is update_texture_cache, 8Mb clear_target, 20MB is drawElementsInstanced,

174MB is malloc

Comment 4

[Rares Doghi, Desktop QA]

Hi Jeff, I'm trying to reproduce this issue in order to get a regression range but are there any steps you can help me with or maybe a reduced case that would help me reproduce the issue ?

Comment 5

[Jeff Muizelaar [:jrmuizel]]

STR: Use Process Explorer to monitor "Private Bytes" of the gpu process. Load a bunch of pages in different tabs. I used the all of the pocket suggestions from about:newtab. Use "Ctrl+Pg Up" to cycle through them. You should see "private bytes" continue to rise.

Comment 6

[Rares Doghi, Desktop QA]

Thanks for the extra info Jeff, I managed to get a regression range, Im not sure how accurate it is but I hope it helps:

First KNOWN BAD:
changeset: a9eaeeec414894e9968976cdf58ae390d0244970
pushlog_url: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=bb2d9b53ac6e5c462588725ac967e6c4f2ff810d&tochange=a9eaeeec414894e9968976cdf58ae390d0244970

Last KNOWN GOOD:
changeset: 41012e39a75d1e0a78bd4756ccc84902b50af309
pushlog_url: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=41012e39a75d1e0a78bd4756ccc84902b50af309&tochange=a9eaeeec414894e9968976cdf58ae390d0244970

Comment 7

[Jeff Muizelaar [:jrmuizel]]

It's possible the regression window is confused by bug 1640103 which was a memory regression caused by bug 1612440

Comment 8

[Jeff Muizelaar [:jrmuizel]]

Some more discoveries here:

• RawVec<gpu_cache::GpuBlockData> is probably not the real vec but is something that shows up because of identical code folding.
• I tried reproducing it with DMD and didn't really see the same thing.

Comment 9

[Jeff Muizelaar [:jrmuizel]]

I also confirmed that we seem to be leaking memory that is not allocated through VirtualAlloc. I was able to grow my GPU process to 1.1GB and had 530MB allocated through VirtualAlloc and 599MB through Unknown.

Comment 10

[Jeff Muizelaar [:jrmuizel]]

Bug 1654929 fixes some of this but I think there's still more leaking going on. More specifically we seem to be leaking the batching Vecs that we allocate here: https://searchfox.org/mozilla-central/rev/227f22acef5c4865503bde9f835452bf38332c8e/gfx/wr/webrender/src/batch.rs#328

Comment 11

[Dzmitry Malyshau [:kvark]]

ni me to look at this!

Comment 12

[Dzmitry Malyshau [:kvark]]

Looked at the batching code and haven't found any inherently wrong. The code in comment 10 reserves at most 128 instances per batch. Given that we typically have ~50 batches, this shouldn't show up as anything in memory profiles, unless, of course, it's truly leaked. The whole batch list is rebuilt every frame from scratch though, so I'm not seeing how it would leak.

Comment 13

[Andrew Osmond [:aosmond] (he/him)]

Given the original regression range in comment 6, I think bug 1640564 would have fixed this (or a large part thereof), notwithstanding Jeff's other concerns.

Comment 14

[Nicolas Silva [:nical]]

Paul can you still reproduce this ? It should be fixed by bug 1640564.

Comment 15

[Paul Adenot (:padenot)]

It's now seemingly bounded to about 850MB for the same scenario on the same machine, sorry for the delay in answering.