Closed Bug 1441667 Opened 3 years ago Closed 3 years ago

WebRender leaks over 3GB of memory during awsy testing

Categories

(Core :: Graphics: WebRender, enhancement, P1)

enhancement

Tracking

()

RESOLVED FIXED

People

(Reporter: erahm, Assigned: lsalzman)

References

Details

(Whiteboard: [MemShrink])

Attachments

(1 file)

We can see during awsy testing that over 3GB of memory is leaked during testing with webrender enabled [1]:

> 3,367.49 MB (100.0%) -- explicit
> ├──3,256.20 MB (96.70%) ── heap-unclassified
> ├─────69.43 MB (02.06%) ++ (29 tiny)
> └─────41.85 MB (01.24%) ++ heap-overhead

Compared with webrender disabled:

> 115.00 MB (100.0%) -- explicit
> ├───33.31 MB (28.97%) ++ js-non-window
> ├───20.04 MB (17.42%) ++ heap-overhead
> ├───16.67 MB (14.50%) ── heap-unclassified

In theory bug 1432708 should have helped but it looks like there are more leaks.

[1] https://treeherder.mozilla.org/perf.html#/graphs?series=mozilla-central,1660725,1,4&series=mozilla-central,1651973,1,4&series=mozilla-central,1651977,1,4&series=mozilla-central,1660729,1,4&zoom=1519210409403.4749,1519730854000,0,5000000000
Is it possible to do an AWSY run with DMD enabled to get some stacks of where things are being allocated?
Flags: needinfo?(erahm)
(In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #1)
> Is it possible to do an AWSY run with DMD enabled to get some stacks of
> where things are being allocated?

Yeah I can kick off a run. Patch 5 in bug 1395540 adds support for dmd in automation, but hasn't landed yet. Locally you can run |mach awsy-test --dmd|
Flags: needinfo?(erahm)
Hopefully a run with webrender + DMD, it's going to be pretty slow though:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=bf49ebc634f1386880e4730ca19c9d39afa11914
Attached file Processed DMD report
It looks like we're leaking font stuff and maybe gl stuff. Fonts seem to dominate over time.
Fonts:

> Unreported {
>   26,272 blocks in heap block record 1 of 5,098
>   1,073,032,256 bytes (1,048,415,656 requested / 24,616,600 slop)
>   Individual block sizes: 241,664 x 2,826; 135,168 x 102; 114,688 x 1,224; 110,592 x 1,500; 16,384 x 1,224; 12,288 x 1,602; 4,096 x 2,826; 2,048 x 8,361; 1,024 x 99; 432 x 1,096; 384 x 812; 352 x 72; 256 x 800; 224 x 51; 208 x 558; 192 x 645; 176 x 529; 160 x 35; 112 x 429; 96 x 356; 80 x 19; 64 x 462; 48 x 7; 32 x 491; 16 x 146
>   35.55% of the heap (35.55% cumulative)
>   36.04% of unreported (36.04% cumulative)
>   Allocated at {
>     #01: ft_mem_qalloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #02: ft_mem_alloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #03: ft_mem_qrealloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #04: ft_mem_realloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #05: ??? (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #06: ??? (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #07: TT_New_Context (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #08: FT_Done_GlyphSlot (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>   }
> }
> 
> Unreported {
>   2,826 blocks in heap block record 3 of 5,098
>   219,930,624 bytes (214,411,446 requested / 5,519,178 slop)
>   Individual block sizes: 131,072 x 1,413; 24,576 x 1,413
>   7.29% of the heap (70.31% cumulative)
>   7.39% of unreported (71.27% cumulative)
>   Allocated at {
>     #01: ft_mem_qalloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #02: ft_mem_alloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #03: ft_mem_qrealloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #04: ft_mem_realloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #05: TT_New_Context (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #06: FT_Done_GlyphSlot (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #07: FT_Open_Face (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #08: FT_New_Face (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>   }
> }
> 
> Unreported {
>   10,749 blocks in heap block record 4 of 5,098
>   198,801,408 bytes (186,506,540 requested / 12,294,868 slop)
>   Individual block sizes: 135,168 x 1,143; 28,672 x 37; 16,384 x 686; 8,192 x 1,192; 4,096 x 3,168; 2,048 x 4,523
>   6.59% of the heap (76.90% cumulative)
>   6.68% of unreported (77.95% cumulative)
>   Allocated at {
>     #01: ft_mem_qalloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #02: ft_mem_alloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #03: af_get_coverage (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #04: af_get_coverage (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #05: FT_Load_Glyph (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #06: webrender::platform::unix::font::{{impl}}::load_glyph (/var/dev/erahm/mozilla-unified/gfx/webrender/src/platform/unix/font.rs:320)
>   }
> }
> 
> Unreported {
>   11,871 blocks in heap block record 5 of 5,098
>   145,870,848 bytes (103,325,184 requested / 42,545,664 slop)
>   Individual block sizes: 12,288 x 11,871
>   4.83% of the heap (81.73% cumulative)
>   4.90% of unreported (82.85% cumulative)
>   Allocated at {
>     #01: ft_mem_qalloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #02: ft_mem_alloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #03: af_get_coverage (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #04: FT_Load_Glyph (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #05: webrender::platform::unix::font::{{impl}}::load_glyph (/var/dev/erahm/mozilla-unified/gfx/webrender/src/platform/unix/font.rs:320)
>   }
> }
> 
> Unreported {
>   29,342 blocks in heap block record 6 of 5,098
>   120,184,832 bytes (115,959,584 requested / 4,225,248 slop)
>   Individual block sizes: 4,096 x 29,342
>   3.98% of the heap (85.71% cumulative)
>   4.04% of unreported (86.89% cumulative)
>   Allocated at {
>     #01: ft_mem_qalloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #02: ft_mem_alloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #03: af_get_char_index (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #04: TT_New_Context (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #05: FT_New_Size (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #06: FT_Open_Face (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #07: FT_New_Face (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #08: webrender::platform::unix::font::{{impl}}::add_native_font (/var/dev/erahm/mozilla-unified/gfx/webrender/src/platform/unix/font.rs:205)
>   }
> }
> 
> Unreported {
>   3,366 blocks in heap block record 7 of 5,098
>   75,550,720 bytes (67,891,200 requested / 7,659,520 slop)
>   Individual block sizes: 61,440 x 51; 49,152 x 51; 40,960 x 1,020; 24,576 x 102; 16,384 x 952; 12,288 x 68; 8,192 x 1,122
>   2.50% of the heap (88.21% cumulative)
>   2.54% of unreported (89.42% cumulative)
>   Allocated at {
>     #01: ft_mem_qalloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #02: ft_mem_alloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #03: FT_Stream_OpenBzip2 (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #04: TT_New_Context (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #05: TT_New_Context (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #06: TT_New_Context (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #07: TT_New_Context (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #08: FT_Done_GlyphSlot (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>   }
> }
> 
> Unreported {
>   3,445 blocks in heap block record 8 of 5,098
>   47,265,520 bytes (36,773,864 requested / 10,491,656 slop)
>   Individual block sizes: 24,576 x 102; 16,384 x 2,724; 208 x 619
>   1.57% of the heap (89.78% cumulative)
>   1.59% of unreported (91.01% cumulative)
>   Allocated at {
>     #01: ft_mem_qalloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #02: ft_mem_alloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #03: ft_mem_qrealloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #04: ft_mem_realloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #05: ??? (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #06: TT_New_Context (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #07: FT_Done_GlyphSlot (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #08: FT_Open_Face (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>   }
> }

etc
gl is harder, I don't have a stack:

> Unreported {
>   4,940 blocks in heap block record 2 of 5,098
>   829,060,416 bytes (826,131,598 requested / 2,928,818 slop)
>   Individual block sizes: 67,112,960; 16,781,312 x 3; 8,392,704; 4,198,400 x 2; 3,854,336 x 50; 3,354,624 x 2; 3,153,920 x 2; 3,080,192 x 71; 3,059,712 x 5; 2,101,248; 2,097,152; 2,093,056; 1,576,960 x 2; 1,277,952 x 2; 1,196,032 x 2; 1,089,536; 1,052,672 x 3; 999,424 x 7; 966,656 x 50; 962,560 x 9; 950,272 x 2; 839,680 x 2; 819,200 x 2; 786,432 x 2; 770,048 x 71; 765,952 x 5; 602,112 x 4; 540,672 x 9; 528,384 x 4; 524,288 x 7; 483,328 x 4; 466,944 x 2; 397,312; 393,216 x 2; 319,488 x 2; 299,008 x 3; 290,816; 274,432; 266,240 x 8; 262,144 x 2; 249,856 x 8; 241,664 x 59; 208,896 x 2; 196,608 x 2; 192,512 x 76; 188,416; 172,032; 163,840 x 3; 159,744 x 2; 151,552 x 4; 139,264; 135,168 x 11; 131,072 x 7; 122,880 x 4; 118,784 x 2; 106,496; 102,400; 98,304; 90,112 x 8; 81,920 x 2; 77,824 x 4; 73,728 x 5; 69,632 x 7; 65,536 x 433; 61,440 x 61; 57,344 x 2; 53,248 x 6; 49,152 x 78; 45,056 x 5; 40,960 x 7; 36,864 x 16; 32,768 x 22; 28,672 x 5; 24,576 x 5; 20,480 x 23; 16,384 x 80; 12,288 x 112; 8,192 x 136; 4,096 x 228; 2,048 x 916; 1,024 x 241; 512 x 93; 496 x 6; 464 x 210; 448 x 6; 432 x 8; 400 x 7; 384 x 4; 352 x 4; 336 x 8; 320 x 8; 304 x 241; 288 x 14; 272 x 8; 256 x 72; 240 x 9; 224 x 6; 208 x 25; 192 x 37; 176 x 11; 160 x 2; 144 x 24; 128 x 34; 112 x 20; 96 x 607; 80 x 55; 64 x 184; 48 x 146; 32 x 133; 16 x 12; 8 x 8
>   27.47% of the heap (63.02% cumulative)
>   27.85% of unreported (63.89% cumulative)
>   Allocated at {
>     #01: vk_icdGetInstanceProcAddr (/usr/lib/nvidia-384/libGL.so.1)
>   }
> }
Sorry DMD was clamping to 8 frames in that run, most of the allocations originate in:

webrender::platform::unix::font::{{impl}}::load_glyph
webrender::platform::unix::font::{{impl}}::add_native_font
Lee, how are font allocations supposed to be freed with WR? See allocation stacks above - we're leaking lots of memory.
Flags: needinfo?(lsalzman)
Assignee: nobody → lsalzman
Priority: -- → P1
See Also: → 1438021
This seems to be a WebRender bug. On tab teardown, rather than explicitly deleting each font, we rely on WR to clear out all fonts in the tab's id namespace. However, WR was not properly doing this. This will be fixed by WR PR https://github.com/servo/webrender/pull/2516
Depends on: 1444946
Flags: needinfo?(lsalzman)
Thanks! I did a try push [1] with the PR applied. The awsy job shows the resident memory as 1,377,407,401.72 which is ~2 GB better than the previous try push (without that PR) at [2]. So yeah, the PR seems to have worked, \o/

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=e1c9ac9bf5f755d809d08ea81fc0536c179b2216&group_state=expanded&selectedJob=167929414
[2] https://treeherder.mozilla.org/#/jobs?repo=try&revision=dd8ae9480fb28ff31b42a3eb415918d4240a9f23&group_state=expanded&selectedJob=167745027
I'm working on another WR PR to limit the over all size of the glyph cache which should further improve this: https://github.com/servo/webrender/pull/2543
Given that both PRs (2516/2543) I wrote to address this have now been merged into Gecko, I am going to mark this as resolved. If we have further leaks other than just the glyph cache or behavior that needs addressing, let's just set up a new bug to deal with it.
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.