1686061 - 400% more heap-unclassified memory reported by AWSY on Linux (webrender vs non-webrender)

Status: RESOLVED INCOMPLETE

(Core :: Graphics: WebRender, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Check the following graph from perfherder:

https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&series=autoland,1959227,1,4&series=mozilla-central,2851262,1,4&series=mozilla-central,1960003,1,4&series=mozilla-central,2223974,1,4&timerange=1209600

As it can be seen in the graph there is 50MB of extra heap-unclassified memory reported on windows-qr compared to MacOS and Linux. Also non-WebRender builds on Windows have only 1.5MB of heap-unclassified.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Greg, do you know how these values are actually computed by AWSY? It would be good to get the appropriate DMD logs.

Comment 2

[Greg Mierzwinski [:sparky]]

Those values are parsed from about:memory reports:
(1): https://searchfox.org/mozilla-central/source/testing/awsy/awsy/process_perf_data.py#134-166
(2): https://searchfox.org/mozilla-central/source/testing/awsy/awsy/parse_about_memory.py#99-128

Comment 3

[Markus Stange [:mstange]]

Interesting, there seem to be some traces of DMD support there: https://searchfox.org/mozilla-central/rev/014fe72eaba26dcf6082fb9bbaf208f97a38594e/testing/awsy/awsy/process_perf_data.py#128-129

Greg, do you know if it's possible to obtain DMD dumps from those tests?
https://developer.mozilla.org/en-US/docs/Mozilla/Performance/DMD

Comment 4

[Markus Stange [:mstange]]

Oh, apparently there is an awsy-dmd job, see bug 1395540 and bug 1628527, albeit with broken stacks on Windows (see that latter bug).

Comment 5

[Markus Stange [:mstange]]

I've triggered an "sy-d" job on the latest mozilla-central push: https://treeherder.mozilla.org/jobs?repo=mozilla-central&tier=1%2C2%2C3&revision=6da943baaccf738a40f52a3c481801668371d34c&searchStr=shippable%2Cawsy%2Cdmd

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Markus Stange [:mstange] from comment #5)

I've triggered an "sy-d" job on the latest mozilla-central push: https://treeherder.mozilla.org/jobs?repo=mozilla-central&tier=1%2C2%2C3&revision=6da943baaccf738a40f52a3c481801668371d34c&searchStr=shippable%2Cawsy%2Cdmd

There are a huge amount of DMD logs attached as artifacts . I wonder how the value on Perfherder is actually computed. Markus, have you had a chance to check those DMD reports?

Comment 7

[Markus Stange [:mstange]]

I haven't had a chance yet, and I'm also not sure which report I would need to look at.

Comment 8

[Markus Stange [:mstange]]

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+1] from comment #0)

https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&series=autoland,1959227,1,4&series=mozilla-central,2851262,1,4&series=mozilla-central,1960003,1,4&series=mozilla-central,2223974,1,4&timerange=1209600

That URL is showing both "Base Content Heap Unclassified" and "Heap Unclassified" graphs. The two have very different numbers.

Graph of just "Base Content Heap Unclassified"
Graph of just "Heap Unclassified"

In both of these graphs, the numbers are Windows < Linux < macOS.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Oh, good catch! I actually didn't notice that I accidentally selected the wrong measurement. That would indeed make this bug invalid. Sorry for that.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Markus Stange [:mstange] from comment #8)

Graph of just "Base Content Heap Unclassified"
Graph of just "Heap Unclassified"

Actually for those graphs you also selected the wrong platforms (non-webrender) once for Windows and then for Linux.

Here the real graphs:

Graph of just "Base Content Heap Unclassified"
Graph of just "Heap Unclassified"

In both cases Windows is indeed the best platform, and Linux / Mac have worse numbers, especially Heap Unclassified on Linux, which are 4x the amount of non-webrender.

I'm going to morph this bug so it's about Linux and not Windows.

Comment 11

[Markus Stange [:mstange]]

I've triggered some Linux DMD jobs.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Markus Stange [:mstange] from comment #11)

I've triggered some Linux DMD jobs.

Mind sharing the (try) build where you triggered those jobs? We could then ask Greg which of the DMD files we actually would have to look at.

Comment 13

[Markus Stange [:mstange]]

Turns out I triggered the wrong jobs - I triggered non-DMD awsy. Same push as before: https://treeherder.mozilla.org/jobs?repo=mozilla-central&tier=1%2C2%2C3&revision=6da943baaccf738a40f52a3c481801668371d34c&searchStr=awsy&selectedTaskRun=cNNFueHKSqSDaGWEr-8R3A.0

I tried to trigger awsy-DMD just now but the infrastructure seems to be in a bad state and triggering jobs is currently impossible.

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I used the staging instance of Treeherder to trigger the job:

https://treeherder.allizom.org/jobs?repo=mozilla-central&tier=1%2C2%2C3&revision=6da943baaccf738a40f52a3c481801668371d34c&searchStr=awsy

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Greg, mind telling us which specific DMD reports we need here?

Comment 16

[Greg Mierzwinski [:sparky]]

Looking at the subtests for Heap Unclassified, I'm seeing that all of them have increased significantly.

I think you'll want to look at these DMD reports because they are the most obvious regressions (however, you should see the issue in most/all of them):

• dmd-TabsClosed-*.json.gz
• dmd-TabsClosedExtraProcesses-*.json.gz
• dmd-TabsClosedForceGC-*.json.gz

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: dmd-report.txt

When taking the following job I can see a lot of unreported heap-unclassified in webrender:

https://treeherder.allizom.org/jobs?repo=mozilla-central&tier=1%2C2%2C3&revision=6da943baaccf738a40f52a3c481801668371d34c&searchStr=awsy&selectedTaskRun=eNomKr90SAWc38WWsXHFCA.0

The parent process pid is actually 1177.

The largest piece seems to come from webrender::device::gl::Device::create_texture:

Unreported {
  68 blocks in heap block record 1 of 16,098
  117,600,256 bytes (117,463,296 requested / 136,960 slop)
  Individual block sizes: 16,777,216 x 5; 4,194,304 x 2; 2,420,736; 2,179,072; 2,097,152 x 4; 1,318,912; 1,048,576; 1,044,480 x 2; 999,424; 876,544; 745,472 x 2; 729,088; 393,216; 258,048; 225,280; 221,184; 184,320; 126,976; 114,688; 86,016; 77,824; 73,728 x 3; 69,632; 65,536 x 24; 40,960; 36,864 x 2; 32,768; 24,576; 16,384 x 3; 12,288
  42.14% of the heap (42.14% cumulative)
  57.77% of unreported (57.77% cumulative)
  Allocated at {
    #01: webrender::device::gl::Device::create_texture::ha427ee5acf7e061b (/builds/worker/workspace/build/application/firefox/libxul.so + 0x3fcc839)
    #02: ??? (/builds/worker/workspace/build/application/firefox/libxul.so + 0x40574de)
    #03: ??? (/builds/worker/workspace/build/application/firefox/libxul.so + 0x4051ca9)
    #04: webrender::renderer::Renderer::render::h1efb399faaa43b4f (/builds/worker/workspace/build/application/firefox/libxul.so + 0x4051455)
    #05: ??? (/builds/worker/workspace/build/application/firefox/libxul.so + 0x3eb0cd4)
    #06: ??? (/builds/worker/workspace/build/application/firefox/libxul.so + 0x1854516)
    #07: ??? (/builds/worker/workspace/build/application/firefox/libxul.so + 0x1853c92)
    #08: ??? (/builds/worker/workspace/build/application/firefox/libxul.so + 0x18537b7)
    #09: ??? (/builds/worker/workspace/build/application/firefox/libxul.so + 0x185905f)
    #10: ??? (/builds/worker/workspace/build/application/firefox/libxul.so + 0x4954f6e)
    #11: ??? (/builds/worker/workspace/build/application/firefox/libxul.so + 0x540551f)
    #12: ??? (/builds/worker/workspace/build/application/firefox/libxul.so + 0x540b77f)
    #13: ??? (/builds/worker/workspace/build/application/firefox/libxul.so + 0x5409c4a)
    #14: ??? (/lib/x86_64-linux-gnu/libpthread.so.0 + 0x76db)
  }
}

(I assume that I need a Linux machine to actually process the dmd log correctly)

Comment 18

[Markus Stange [:mstange]]

Attachment: DMD report from dmd-TabsClosedSettled-1-1177.json.gz

I resymbolicated the report.

Comment 19

[Markus Stange [:mstange]]

Attachment: DMD report from dmd-TabsOpenSettled-2-1177.json.gz

Comment 20

[Markus Stange [:mstange]]

I resymbolicated the reports with a custom patch to fix-stacks to change the regex, and with this command line:

% /Applications/Firefox\ Nightly.app/Contents/Resources/dmd.py /Users/mstange/Downloads/dmd-TabsOpenSettled-2-1177.json.gz | sed 's$/builds/worker/workspace/build/application/firefox$/Users/mstange/Downloads/firefox-awsy-dmd$' | ~/code/fix-stacks/target/release/fix-stacks -b /Users/mstange/Downloads/target.crashreporter-symbols\(7\) > ~/Downloads/awsy-dmd-TabsOpenSettled-2-1177.txt

(I also had to download the Linux build and the crashreporter-symbols zip.)

Comment 21

[Markus Stange [:mstange]]

Most of the heap unclassified seems to come from the graphics driver and from system freetype.

The graphics driver probably allocates CPU backing memory for the GPU textures.

Comment 22

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Markus Stange [:mstange] from comment #21)

Most of the heap unclassified seems to come from the graphics driver and from system freetype.
The graphics driver probably allocates CPU backing memory for the GPU textures.

Is that something we can track via a memory reporter, or isn't it possible for allocations as done by the kernel (modules)?

Comment 23

[Markus Stange [:mstange]]

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+1] from comment #22)

(In reply to Markus Stange [:mstange] from comment #21)

Most of the heap unclassified seems to come from the graphics driver and from system freetype.
The graphics driver probably allocates CPU backing memory for the GPU textures.

Is that something we can track via a memory reporter, or isn't it possible for allocations as done by the kernel (modules)?

It is probably not possible to track this via a memory reporter, no. (FWIW, it's a system library, not the kernel. The allocation happens in user space, not in kernel space.)

Comment 24

[Jeff Muizelaar [:jrmuizel]]

If most of this memory is coming from system freetype and the GPU driver, as comment 21 suggests, that's pretty expected with WebRender vs non-WebRender.

Without WebRender we weren't using the GPU so that new memory is not surprising.

With WebRender we're now rasterizing all fonts in the parent process instead of the child processes so it's also expected that we'd have more FreeType memory in the parent process.

Comment 25

[Henrik Skupin [:whimboo][⌚️UTC+2]]

How does that work on other platforms like Windows and MacOS? Is that also system memory that is used there? Why do we have such a huge difference on Linux only?

Comment 26

[Jeff Muizelaar [:jrmuizel]]

On macOS and Windows we were using the GPU before WebRender. On Linux we were not so it's hard to compare Linux with and without WebRender.

Comment 27

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I see. But there is still the >250% higher values in heap unclassified values compared to MacOS and Windows when for all WebRender is enabled:

https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&series=mozilla-central,2855108,1,4&series=mozilla-central,2224083,1,4&series=mozilla-central,1961103,1,4&timerange=1209600

Comment 28

[Jeff Muizelaar [:jrmuizel]]

That's a good question. I'm not sure why it would be so different.

Comment 29

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Note that nothing has been changed for the values on Linux after the patch on bug 1690956 has been landed. There is still the same amount of heap unclassified reported.

Comment 30

[gwarser]

If it's graphic driver issue, should it disappear with "WebRender (Software)"?

Comment 31

[Glenn Watson [:gw]]

There is only WR.