1267663 - Unreported heap-unclassified blocks coming from asm.js parsing (detected by DMD)

Status: RESOLVED FIXED

(Core :: JavaScript Engine, defect)

Description

[Jukka Jylänki]

Profiling UE4 PlatformerGame at

https://s3.amazonaws.com/mozilla-games/tmp/2016-04-23-PlatformerGame/PlatformerGame.html?cpuprofiler&playback

which has about 12% of its about:memory footprint in heap-unclassified, DMD shows that the single largest callstack source of unreported memory comes from asm.js parsing, and looks like

Unreported {
  44,976 blocks in heap block record 2 of 3,690
  5,170,144 bytes (4,825,149 requested / 344,995 slop)
  Individual block sizes: 1,024; 512 x 34; 496 x 218; 480 x 51; 464 x 9; 448 x 2; 432 x 54; 416 x 13; 400 x 6; 384 x 27; 368 x 16; 352 x 7; 336 x 12; 320 x 15; 304 x 25; 288 x 46; 272 x 119; 256 x 139; 240 x 252; 224 x 587; 208 x 1,213; 192 x 1,562; 176 x 2,309; 160 x 2,856; 144 x 3,445; 128 x 4,101; 112 x 4,906; 96 x 6,054; 80 x 7,021; 64 x 5,465; 48 x 3,483; 32 x 868; 16 x 60
  0.87% of the heap (4.44% cumulative)
  3.84% of unreported (19.55% cumulative)
  Allocated at {
    #01: js::MallocProvider<js::ExclusiveContext>::pod_malloc<unsigned char> (d:\gecko-dev\js\src\vm\mallocprovider.h:90)
    #02: JS::CharsToNewUTF8CharsZ<unsigned char const > (d:\gecko-dev\js\src\vm\characterencoding.cpp:147)
    #03: js::StringToNewUTF8CharsZ (d:\gecko-dev\js\src\vm\string.h:1158)
    #04: ModuleValidator::finish (d:\gecko-dev\js\src\asmjs\asmjs.cpp:2218)
    #05: CheckModule (d:\gecko-dev\js\src\asmjs\asmjs.cpp:7014)
    #06: js::CompileAsmJS (d:\gecko-dev\js\src\asmjs\asmjs.cpp:8281)
    #07: js::frontend::Parser<js::frontend::FullParseHandler>::asmJS (d:\gecko-dev\js\src\frontend\parser.cpp:3421)
    #08: js::frontend::Parser<js::frontend::FullParseHandler>::statements (d:\gecko-dev\js\src\frontend\parser.cpp:3561)
    #09: js::frontend::Parser<js::frontend::FullParseHandler>::functionBody (d:\gecko-dev\js\src\frontend\parser.cpp:1366)
    #10: js::frontend::Parser<js::frontend::FullParseHandler>::functionArgsAndBodyGeneric (d:\gecko-dev\js\src\frontend\parser.cpp:3189)
    #11: js::frontend::Parser<js::frontend::FullParseHandler>::functionArgsAndBody (d:\gecko-dev\js\src\frontend\parser.cpp:2994)
    #12: js::frontend::Parser<js::frontend::FullParseHandler>::functionDef (d:\gecko-dev\js\src\frontend\parser.cpp:2823)
    #13: js::frontend::Parser<js::frontend::FullParseHandler>::functionExpr (d:\gecko-dev\js\src\frontend\parser.cpp:3348)
    #14: js::frontend::Parser<js::frontend::FullParseHandler>::primaryExpr (d:\gecko-dev\js\src\frontend\parser.cpp:9473)
    #15: js::frontend::Parser<js::frontend::FullParseHandler>::memberExpr (d:\gecko-dev\js\src\frontend\parser.cpp:8722)
    #16: js::frontend::Parser<js::frontend::FullParseHandler>::unaryExpr (d:\gecko-dev\js\src\frontend\parser.cpp:8250)
    #17: js::frontend::Parser<js::frontend::FullParseHandler>::assignExpr (d:\gecko-dev\js\src\frontend\parser.cpp:7915)
    #18: js::frontend::Parser<js::frontend::FullParseHandler>::expr (d:\gecko-dev\js\src\frontend\parser.cpp:7564)
    #19: js::frontend::Parser<js::frontend::FullParseHandler>::primaryExpr (d:\gecko-dev\js\src\frontend\parser.cpp:9514)
    #20: js::frontend::Parser<js::frontend::FullParseHandler>::memberExpr (d:\gecko-dev\js\src\frontend\parser.cpp:8722)
    #21: js::frontend::Parser<js::frontend::FullParseHandler>::unaryExpr (d:\gecko-dev\js\src\frontend\parser.cpp:8250)
    #22: js::frontend::Parser<js::frontend::FullParseHandler>::assignExpr (d:\gecko-dev\js\src\frontend\parser.cpp:7915)
    #23: js::frontend::Parser<js::frontend::FullParseHandler>::initializerInNameDeclaration (d:\gecko-dev\js\src\frontend\parser.cpp:4798)
    #24: js::frontend::Parser<js::frontend::FullParseHandler>::declarationName (d:\gecko-dev\js\src\frontend\parser.cpp:4906)
  }
}

There was a related bug 1241233, I wonder if this might have gone from twice reported to not reported at all, or if this is something slightly different?

Comment 1

[Luke Wagner [:luke]]

Attachment: fix-pretty-name-reporting

Whoops, we're reporting the size of the Vector's buffer, but not transitively aggregating the size of each string in the vector.  Simple fix.  Great work with DMD Jukka!

Comment 2

[Benjamin Bouvier [:bbouvier] (inactive)]

Is there a way we could generalize these fixes? They seem very spot on at the moment, and maybe making a subclass of the Vector that implements their own sizeOfExcludingThis would be more convenient and less error-prone?

Thank you for the bug Jukka and for the patch Luke, btw.

Comment 3

[Luke Wagner [:luke]]

Yeah, that's an interesting idea; kindof like we did with CacheableChars.

Comment 4

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/50f12f65fc3e

Comment 5

[Jukka Jylänki]

Great work with the quick fix! The background here is that occassionally on that UE4 demo we're seeing about 400MB of extra memory being held on in heap-unclassified that is very rarely seen, and I'm trying to hunt down where that comes from exactly, so chipping away on individual DMD sources to try to pinpoint the root cause.

Comment 6

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/50f12f65fc3e