Open Bug 1686061 Opened 9 months ago Updated 6 months ago

400% more heap-unclassified memory reported by AWSY on Linux (webrender vs non-webrender)

Categories

(Core :: Graphics: WebRender, defect)

defect

Tracking

()

REOPENED

People

(Reporter: whimboo, Unassigned)

References

Details

(Keywords: memory-footprint)

Attachments

(2 files, 1 obsolete file)

Check the following graph from perfherder:

https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&series=autoland,1959227,1,4&series=mozilla-central,2851262,1,4&series=mozilla-central,1960003,1,4&series=mozilla-central,2223974,1,4&timerange=1209600

As it can be seen in the graph there is 50MB of extra heap-unclassified memory reported on windows-qr compared to MacOS and Linux. Also non-WebRender builds on Windows have only 1.5MB of heap-unclassified.

Flags: needinfo?(jmuizelaar)

Greg, do you know how these values are actually computed by AWSY? It would be good to get the appropriate DMD logs.

Flags: needinfo?(gmierz2)

Interesting, there seem to be some traces of DMD support there: https://searchfox.org/mozilla-central/rev/014fe72eaba26dcf6082fb9bbaf208f97a38594e/testing/awsy/awsy/process_perf_data.py#128-129

Greg, do you know if it's possible to obtain DMD dumps from those tests?
https://developer.mozilla.org/en-US/docs/Mozilla/Performance/DMD

Flags: needinfo?(gmierz2)

Oh, apparently there is an awsy-dmd job, see bug 1395540 and bug 1628527, albeit with broken stacks on Windows (see that latter bug).

(In reply to Markus Stange [:mstange] from comment #5)

I've triggered an "sy-d" job on the latest mozilla-central push: https://treeherder.mozilla.org/jobs?repo=mozilla-central&tier=1%2C2%2C3&revision=6da943baaccf738a40f52a3c481801668371d34c&searchStr=shippable%2Cawsy%2Cdmd

There are a huge amount of DMD logs attached as artifacts . I wonder how the value on Perfherder is actually computed. Markus, have you had a chance to check those DMD reports?

Flags: needinfo?(jmuizelaar) → needinfo?(mstange.moz)

I haven't had a chance yet, and I'm also not sure which report I would need to look at.

Flags: needinfo?(mstange.moz)

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+1] from comment #0)

https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&series=autoland,1959227,1,4&series=mozilla-central,2851262,1,4&series=mozilla-central,1960003,1,4&series=mozilla-central,2223974,1,4&timerange=1209600

That URL is showing both "Base Content Heap Unclassified" and "Heap Unclassified" graphs. The two have very different numbers.

Graph of just "Base Content Heap Unclassified"
Graph of just "Heap Unclassified"

In both of these graphs, the numbers are Windows < Linux < macOS.

Oh, good catch! I actually didn't notice that I accidentally selected the wrong measurement. That would indeed make this bug invalid. Sorry for that.

Status: NEW → RESOLVED
Closed: 8 months ago
Resolution: --- → INVALID

(In reply to Markus Stange [:mstange] from comment #8)

Graph of just "Base Content Heap Unclassified"
Graph of just "Heap Unclassified"

Actually for those graphs you also selected the wrong platforms (non-webrender) once for Windows and then for Linux.

Here the real graphs:

Graph of just "Base Content Heap Unclassified"
Graph of just "Heap Unclassified"

In both cases Windows is indeed the best platform, and Linux / Mac have worse numbers, especially Heap Unclassified on Linux, which are 4x the amount of non-webrender.

I'm going to morph this bug so it's about Linux and not Windows.

Status: RESOLVED → REOPENED
Resolution: INVALID → ---
Summary: 50MB more heap-unclassified memory reported by AWSY on Windows → 400% more heap-unclassified memory reported by AWSY on Linux (webrender vs non-webrender)
See Also: → 1690956

I've triggered some Linux DMD jobs.

(In reply to Markus Stange [:mstange] from comment #11)

I've triggered some Linux DMD jobs.

Mind sharing the (try) build where you triggered those jobs? We could then ask Greg which of the DMD files we actually would have to look at.

Flags: needinfo?(mstange.moz)

Turns out I triggered the wrong jobs - I triggered non-DMD awsy. Same push as before: https://treeherder.mozilla.org/jobs?repo=mozilla-central&tier=1%2C2%2C3&revision=6da943baaccf738a40f52a3c481801668371d34c&searchStr=awsy&selectedTaskRun=cNNFueHKSqSDaGWEr-8R3A.0

I tried to trigger awsy-DMD just now but the infrastructure seems to be in a bad state and triggering jobs is currently impossible.

Flags: needinfo?(mstange.moz)

Greg, mind telling us which specific DMD reports we need here?

Flags: needinfo?(gmierz2)

Looking at the subtests for Heap Unclassified, I'm seeing that all of them have increased significantly.

I think you'll want to look at these DMD reports because they are the most obvious regressions (however, you should see the issue in most/all of them):

  • dmd-TabsClosed-*.json.gz
  • dmd-TabsClosedExtraProcesses-*.json.gz
  • dmd-TabsClosedForceGC-*.json.gz
Flags: needinfo?(gmierz2)
Attached file dmd-report.txt (obsolete) —

When taking the following job I can see a lot of unreported heap-unclassified in webrender:

https://treeherder.allizom.org/jobs?repo=mozilla-central&tier=1%2C2%2C3&revision=6da943baaccf738a40f52a3c481801668371d34c&searchStr=awsy&selectedTaskRun=eNomKr90SAWc38WWsXHFCA.0

The parent process pid is actually 1177.

The largest piece seems to come from webrender::device::gl::Device::create_texture:

Unreported {
  68 blocks in heap block record 1 of 16,098
  117,600,256 bytes (117,463,296 requested / 136,960 slop)
  Individual block sizes: 16,777,216 x 5; 4,194,304 x 2; 2,420,736; 2,179,072; 2,097,152 x 4; 1,318,912; 1,048,576; 1,044,480 x 2; 999,424; 876,544; 745,472 x 2; 729,088; 393,216; 258,048; 225,280; 221,184; 184,320; 126,976; 114,688; 86,016; 77,824; 73,728 x 3; 69,632; 65,536 x 24; 40,960; 36,864 x 2; 32,768; 24,576; 16,384 x 3; 12,288
  42.14% of the heap (42.14% cumulative)
  57.77% of unreported (57.77% cumulative)
  Allocated at {
    #01: webrender::device::gl::Device::create_texture::ha427ee5acf7e061b (/builds/worker/workspace/build/application/firefox/libxul.so + 0x3fcc839)
    #02: ??? (/builds/worker/workspace/build/application/firefox/libxul.so + 0x40574de)
    #03: ??? (/builds/worker/workspace/build/application/firefox/libxul.so + 0x4051ca9)
    #04: webrender::renderer::Renderer::render::h1efb399faaa43b4f (/builds/worker/workspace/build/application/firefox/libxul.so + 0x4051455)
    #05: ??? (/builds/worker/workspace/build/application/firefox/libxul.so + 0x3eb0cd4)
    #06: ??? (/builds/worker/workspace/build/application/firefox/libxul.so + 0x1854516)
    #07: ??? (/builds/worker/workspace/build/application/firefox/libxul.so + 0x1853c92)
    #08: ??? (/builds/worker/workspace/build/application/firefox/libxul.so + 0x18537b7)
    #09: ??? (/builds/worker/workspace/build/application/firefox/libxul.so + 0x185905f)
    #10: ??? (/builds/worker/workspace/build/application/firefox/libxul.so + 0x4954f6e)
    #11: ??? (/builds/worker/workspace/build/application/firefox/libxul.so + 0x540551f)
    #12: ??? (/builds/worker/workspace/build/application/firefox/libxul.so + 0x540b77f)
    #13: ??? (/builds/worker/workspace/build/application/firefox/libxul.so + 0x5409c4a)
    #14: ??? (/lib/x86_64-linux-gnu/libpthread.so.0 + 0x76db)
  }
}

(I assume that I need a Linux machine to actually process the dmd log correctly)

Flags: needinfo?(mstange.moz)

I resymbolicated the report.

Attachment #9203249 - Attachment is obsolete: true
Flags: needinfo?(mstange.moz)

I resymbolicated the reports with a custom patch to fix-stacks to change the regex, and with this command line:

% /Applications/Firefox\ Nightly.app/Contents/Resources/dmd.py /Users/mstange/Downloads/dmd-TabsOpenSettled-2-1177.json.gz | sed 's$/builds/worker/workspace/build/application/firefox$/Users/mstange/Downloads/firefox-awsy-dmd$' | ~/code/fix-stacks/target/release/fix-stacks -b /Users/mstange/Downloads/target.crashreporter-symbols\(7\) > ~/Downloads/awsy-dmd-TabsOpenSettled-2-1177.txt

(I also had to download the Linux build and the crashreporter-symbols zip.)

Most of the heap unclassified seems to come from the graphics driver and from system freetype.

The graphics driver probably allocates CPU backing memory for the GPU textures.

(In reply to Markus Stange [:mstange] from comment #21)

Most of the heap unclassified seems to come from the graphics driver and from system freetype.
The graphics driver probably allocates CPU backing memory for the GPU textures.

Is that something we can track via a memory reporter, or isn't it possible for allocations as done by the kernel (modules)?

Flags: needinfo?(jmathies)
Blocks: gfx-triage
Flags: needinfo?(jmathies)

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+1] from comment #22)

(In reply to Markus Stange [:mstange] from comment #21)

Most of the heap unclassified seems to come from the graphics driver and from system freetype.
The graphics driver probably allocates CPU backing memory for the GPU textures.

Is that something we can track via a memory reporter, or isn't it possible for allocations as done by the kernel (modules)?

It is probably not possible to track this via a memory reporter, no. (FWIW, it's a system library, not the kernel. The allocation happens in user space, not in kernel space.)

If most of this memory is coming from system freetype and the GPU driver, as comment 21 suggests, that's pretty expected with WebRender vs non-WebRender.

Without WebRender we weren't using the GPU so that new memory is not surprising.

With WebRender we're now rasterizing all fonts in the parent process instead of the child processes so it's also expected that we'd have more FreeType memory in the parent process.

How does that work on other platforms like Windows and MacOS? Is that also system memory that is used there? Why do we have such a huge difference on Linux only?

On macOS and Windows we were using the GPU before WebRender. On Linux we were not so it's hard to compare Linux with and without WebRender.

That's a good question. I'm not sure why it would be so different.

No longer blocks: gfx-triage

Note that nothing has been changed for the values on Linux after the patch on bug 1690956 has been landed. There is still the same amount of heap unclassified reported.

If it's graphic driver issue, should it disappear with "WebRender (Software)"?

You need to log in before you can comment on or make changes to this bug.