1441667 - WebRender leaks over 3GB of memory during awsy testing

Status: RESOLVED FIXED

(Core :: Graphics: WebRender, enhancement, P1)

Description

[Eric Rahm [:erahm]]

We can see during awsy testing that over 3GB of memory is leaked during testing with webrender enabled [1]:

> 3,367.49 MB (100.0%) -- explicit
> ├──3,256.20 MB (96.70%) ── heap-unclassified
> ├─────69.43 MB (02.06%) ++ (29 tiny)
> └─────41.85 MB (01.24%) ++ heap-overhead

Compared with webrender disabled:

> 115.00 MB (100.0%) -- explicit
> ├───33.31 MB (28.97%) ++ js-non-window
> ├───20.04 MB (17.42%) ++ heap-overhead
> ├───16.67 MB (14.50%) ── heap-unclassified

In theory bug 1432708 should have helped but it looks like there are more leaks.

[1] https://treeherder.mozilla.org/perf.html#/graphs?series=mozilla-central,1660725,1,4&series=mozilla-central,1651973,1,4&series=mozilla-central,1651977,1,4&series=mozilla-central,1660729,1,4&zoom=1519210409403.4749,1519730854000,0,5000000000

Comment 1

[Kartikaya Gupta (email:[e-mail])]

Is it possible to do an AWSY run with DMD enabled to get some stacks of where things are being allocated?

Comment 2

[Eric Rahm [:erahm]]

(In reply to Kartikaya Gupta (email:[e-mail]) from comment #1)
> Is it possible to do an AWSY run with DMD enabled to get some stacks of
> where things are being allocated?

Yeah I can kick off a run. Patch 5 in bug 1395540 adds support for dmd in automation, but hasn't landed yet. Locally you can run |mach awsy-test --dmd|

Comment 3

[Eric Rahm [:erahm]]

Hopefully a run with webrender + DMD, it's going to be pretty slow though:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=bf49ebc634f1386880e4730ca19c9d39afa11914

Comment 4

[Eric Rahm [:erahm]]

Attachment: Processed DMD report

It looks like we're leaking font stuff and maybe gl stuff. Fonts seem to dominate over time.

Comment 5

[Eric Rahm [:erahm]]

Fonts:

> Unreported {
>   26,272 blocks in heap block record 1 of 5,098
>   1,073,032,256 bytes (1,048,415,656 requested / 24,616,600 slop)
>   Individual block sizes: 241,664 x 2,826; 135,168 x 102; 114,688 x 1,224; 110,592 x 1,500; 16,384 x 1,224; 12,288 x 1,602; 4,096 x 2,826; 2,048 x 8,361; 1,024 x 99; 432 x 1,096; 384 x 812; 352 x 72; 256 x 800; 224 x 51; 208 x 558; 192 x 645; 176 x 529; 160 x 35; 112 x 429; 96 x 356; 80 x 19; 64 x 462; 48 x 7; 32 x 491; 16 x 146
>   35.55% of the heap (35.55% cumulative)
>   36.04% of unreported (36.04% cumulative)
>   Allocated at {
>     #01: ft_mem_qalloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #02: ft_mem_alloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #03: ft_mem_qrealloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #04: ft_mem_realloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #05: ??? (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #06: ??? (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #07: TT_New_Context (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #08: FT_Done_GlyphSlot (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>   }
> }
> 
> Unreported {
>   2,826 blocks in heap block record 3 of 5,098
>   219,930,624 bytes (214,411,446 requested / 5,519,178 slop)
>   Individual block sizes: 131,072 x 1,413; 24,576 x 1,413
>   7.29% of the heap (70.31% cumulative)
>   7.39% of unreported (71.27% cumulative)
>   Allocated at {
>     #01: ft_mem_qalloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #02: ft_mem_alloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #03: ft_mem_qrealloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #04: ft_mem_realloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #05: TT_New_Context (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #06: FT_Done_GlyphSlot (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #07: FT_Open_Face (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #08: FT_New_Face (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>   }
> }
> 
> Unreported {
>   10,749 blocks in heap block record 4 of 5,098
>   198,801,408 bytes (186,506,540 requested / 12,294,868 slop)
>   Individual block sizes: 135,168 x 1,143; 28,672 x 37; 16,384 x 686; 8,192 x 1,192; 4,096 x 3,168; 2,048 x 4,523
>   6.59% of the heap (76.90% cumulative)
>   6.68% of unreported (77.95% cumulative)
>   Allocated at {
>     #01: ft_mem_qalloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #02: ft_mem_alloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #03: af_get_coverage (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #04: af_get_coverage (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #05: FT_Load_Glyph (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #06: webrender::platform::unix::font::{{impl}}::load_glyph (/var/dev/erahm/mozilla-unified/gfx/webrender/src/platform/unix/font.rs:320)
>   }
> }
> 
> Unreported {
>   11,871 blocks in heap block record 5 of 5,098
>   145,870,848 bytes (103,325,184 requested / 42,545,664 slop)
>   Individual block sizes: 12,288 x 11,871
>   4.83% of the heap (81.73% cumulative)
>   4.90% of unreported (82.85% cumulative)
>   Allocated at {
>     #01: ft_mem_qalloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #02: ft_mem_alloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #03: af_get_coverage (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #04: FT_Load_Glyph (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #05: webrender::platform::unix::font::{{impl}}::load_glyph (/var/dev/erahm/mozilla-unified/gfx/webrender/src/platform/unix/font.rs:320)
>   }
> }
> 
> Unreported {
>   29,342 blocks in heap block record 6 of 5,098
>   120,184,832 bytes (115,959,584 requested / 4,225,248 slop)
>   Individual block sizes: 4,096 x 29,342
>   3.98% of the heap (85.71% cumulative)
>   4.04% of unreported (86.89% cumulative)
>   Allocated at {
>     #01: ft_mem_qalloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #02: ft_mem_alloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #03: af_get_char_index (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #04: TT_New_Context (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #05: FT_New_Size (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #06: FT_Open_Face (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #07: FT_New_Face (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #08: webrender::platform::unix::font::{{impl}}::add_native_font (/var/dev/erahm/mozilla-unified/gfx/webrender/src/platform/unix/font.rs:205)
>   }
> }
> 
> Unreported {
>   3,366 blocks in heap block record 7 of 5,098
>   75,550,720 bytes (67,891,200 requested / 7,659,520 slop)
>   Individual block sizes: 61,440 x 51; 49,152 x 51; 40,960 x 1,020; 24,576 x 102; 16,384 x 952; 12,288 x 68; 8,192 x 1,122
>   2.50% of the heap (88.21% cumulative)
>   2.54% of unreported (89.42% cumulative)
>   Allocated at {
>     #01: ft_mem_qalloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #02: ft_mem_alloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #03: FT_Stream_OpenBzip2 (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #04: TT_New_Context (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #05: TT_New_Context (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #06: TT_New_Context (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #07: TT_New_Context (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #08: FT_Done_GlyphSlot (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>   }
> }
> 
> Unreported {
>   3,445 blocks in heap block record 8 of 5,098
>   47,265,520 bytes (36,773,864 requested / 10,491,656 slop)
>   Individual block sizes: 24,576 x 102; 16,384 x 2,724; 208 x 619
>   1.57% of the heap (89.78% cumulative)
>   1.59% of unreported (91.01% cumulative)
>   Allocated at {
>     #01: ft_mem_qalloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #02: ft_mem_alloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #03: ft_mem_qrealloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #04: ft_mem_realloc (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #05: ??? (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #06: TT_New_Context (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #07: FT_Done_GlyphSlot (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>     #08: FT_Open_Face (/usr/lib/x86_64-linux-gnu/libfreetype.so.6)
>   }
> }

etc

Comment 6

[Eric Rahm [:erahm]]

gl is harder, I don't have a stack:

> Unreported {
>   4,940 blocks in heap block record 2 of 5,098
>   829,060,416 bytes (826,131,598 requested / 2,928,818 slop)
>   Individual block sizes: 67,112,960; 16,781,312 x 3; 8,392,704; 4,198,400 x 2; 3,854,336 x 50; 3,354,624 x 2; 3,153,920 x 2; 3,080,192 x 71; 3,059,712 x 5; 2,101,248; 2,097,152; 2,093,056; 1,576,960 x 2; 1,277,952 x 2; 1,196,032 x 2; 1,089,536; 1,052,672 x 3; 999,424 x 7; 966,656 x 50; 962,560 x 9; 950,272 x 2; 839,680 x 2; 819,200 x 2; 786,432 x 2; 770,048 x 71; 765,952 x 5; 602,112 x 4; 540,672 x 9; 528,384 x 4; 524,288 x 7; 483,328 x 4; 466,944 x 2; 397,312; 393,216 x 2; 319,488 x 2; 299,008 x 3; 290,816; 274,432; 266,240 x 8; 262,144 x 2; 249,856 x 8; 241,664 x 59; 208,896 x 2; 196,608 x 2; 192,512 x 76; 188,416; 172,032; 163,840 x 3; 159,744 x 2; 151,552 x 4; 139,264; 135,168 x 11; 131,072 x 7; 122,880 x 4; 118,784 x 2; 106,496; 102,400; 98,304; 90,112 x 8; 81,920 x 2; 77,824 x 4; 73,728 x 5; 69,632 x 7; 65,536 x 433; 61,440 x 61; 57,344 x 2; 53,248 x 6; 49,152 x 78; 45,056 x 5; 40,960 x 7; 36,864 x 16; 32,768 x 22; 28,672 x 5; 24,576 x 5; 20,480 x 23; 16,384 x 80; 12,288 x 112; 8,192 x 136; 4,096 x 228; 2,048 x 916; 1,024 x 241; 512 x 93; 496 x 6; 464 x 210; 448 x 6; 432 x 8; 400 x 7; 384 x 4; 352 x 4; 336 x 8; 320 x 8; 304 x 241; 288 x 14; 272 x 8; 256 x 72; 240 x 9; 224 x 6; 208 x 25; 192 x 37; 176 x 11; 160 x 2; 144 x 24; 128 x 34; 112 x 20; 96 x 607; 80 x 55; 64 x 184; 48 x 146; 32 x 133; 16 x 12; 8 x 8
>   27.47% of the heap (63.02% cumulative)
>   27.85% of unreported (63.89% cumulative)
>   Allocated at {
>     #01: vk_icdGetInstanceProcAddr (/usr/lib/nvidia-384/libGL.so.1)
>   }
> }

Comment 7

[Eric Rahm [:erahm]]

Sorry DMD was clamping to 8 frames in that run, most of the allocations originate in:

webrender::platform::unix::font::{{impl}}::load_glyph
webrender::platform::unix::font::{{impl}}::add_native_font

Comment 8

[Kartikaya Gupta (email:[e-mail])]

Lee, how are font allocations supposed to be freed with WR? See allocation stacks above - we're leaking lots of memory.

Comment 9

[Lee Salzman [:lsalzman]]

This seems to be a WebRender bug. On tab teardown, rather than explicitly deleting each font, we rely on WR to clear out all fonts in the tab's id namespace. However, WR was not properly doing this. This will be fixed by WR PR https://github.com/servo/webrender/pull/2516

Comment 10

[Kartikaya Gupta (email:[e-mail])]

Thanks! I did a try push [1] with the PR applied. The awsy job shows the resident memory as 1,377,407,401.72 which is ~2 GB better than the previous try push (without that PR) at [2]. So yeah, the PR seems to have worked, \o/

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=e1c9ac9bf5f755d809d08ea81fc0536c179b2216&group_state=expanded&selectedJob=167929414
[2] https://treeherder.mozilla.org/#/jobs?repo=try&revision=dd8ae9480fb28ff31b42a3eb415918d4240a9f23&group_state=expanded&selectedJob=167745027

Comment 11

[Lee Salzman [:lsalzman]]

I'm working on another WR PR to limit the over all size of the glyph cache which should further improve this: https://github.com/servo/webrender/pull/2543

Comment 12

[Lee Salzman [:lsalzman]]

Given that both PRs (2516/2543) I wrote to address this have now been merged into Gecko, I am going to mark this as resolved. If we have further leaks other than just the glyph cache or behavior that needs addressing, let's just set up a new bug to deal with it.