1087093 - large blocks of unreported memory from CodeGenerator::link

Status: NEW

(Core :: JavaScript Engine: JIT, defect, P5)

Description

[Nathan Froyd [:froydnj]]

Taking a DMD snapshot in my browser showed up these stacks:

Unreported {
  ~8,074 blocks in heap block record 2 of 8,862
  ~34,026,846 bytes (~33,203,306 requested / ~823,540 slop)
  5.03% of the heap (10.70% cumulative)
  11.87% of unreported (25.26% cumulative)
  Allocated at {
    #01: replace_malloc (/opt/src/repos/mozilla-central.hg/memory/replace/dmd/DMD.cpp:1113)
    #02: pod_malloc<unsigned char> (/opt/src/repos/mozilla-central.hg/js/src/vm/MallocProvider.h:63)
    #03: js::jit::CodeGenerator::link(JSContext*, js::types::CompilerConstraintList*) (/opt/src/repos/mozilla-central.hg/js/src/jit/CodeGenerator.cpp:7570)
    #04: ~AutoUnlockHelperThreadState (/opt/src/repos/mozilla-central.hg/js/src/vm/HelperThreads.h:424)
    #05: js::InvokeInterruptCallback(JSContext*) (/opt/src/repos/mozilla-central.hg/js/src/jscntxt.cpp:1007)
    #06: ??? (???:???)
  }
}

Unreported {
  ~7,306 blocks in heap block record 3 of 8,862
  ~29,936,289 bytes (~29,896,517 requested / ~39,772 slop)
  4.42% of the heap (15.12% cumulative)
  10.45% of unreported (35.71% cumulative)
  Allocated at {
    #01: replace_malloc (/opt/src/repos/mozilla-central.hg/memory/replace/dmd/DMD.cpp:1113)
    #02: pod_malloc<unsigned char> (/opt/src/repos/mozilla-central.hg/js/src/vm/MallocProvider.h:63)
    #03: js::jit::CodeGenerator::link(JSContext*, js::types::CompilerConstraintList*) (/opt/src/repos/mozilla-central.hg/js/src/jit/CodeGenerator.cpp:7570)
    #04: ~AutoUnlockHelperThreadState (/opt/src/repos/mozilla-central.hg/js/src/vm/HelperThreads.h:424)
    #05: js::InvokeInterruptCallback(JSContext*) (/opt/src/repos/mozilla-central.hg/js/src/jscntxt.cpp:1007)
    #06: ??? (???:???)
  }
}

Unreported {
  ~5,116 blocks in heap block record 4 of 8,862
  ~21,276,125 bytes (~20,971,725 requested / ~304,400 slop)
  3.14% of the heap (18.26% cumulative)
  7.42% of unreported (43.13% cumulative)
  Allocated at {
    #01: replace_malloc (/opt/src/repos/mozilla-central.hg/memory/replace/dmd/DMD.cpp:1113)
    #02: pod_malloc<unsigned char> (/opt/src/repos/mozilla-central.hg/js/src/vm/MallocProvider.h:63)
    #03: js::jit::CodeGenerator::link(JSContext*, js::types::CompilerConstraintList*) (/opt/src/repos/mozilla-central.hg/js/src/jit/CodeGenerator.cpp:7570)
    #04: ~AutoUnlockHelperThreadState (/opt/src/repos/mozilla-central.hg/js/src/vm/HelperThreads.h:424)
    #05: js::InvokeInterruptCallback(JSContext*) (/opt/src/repos/mozilla-central.hg/js/src/jscntxt.cpp:1007)
    #06: ??? (???:???)
  }
}

These are the biggest offenders, I can see several more with ~2.5MB allocated, and a long tail of what look like related stacks.

Surfacing this in about:memory would be nice.  Reducing this amount would also be nice.

Comment 1

[Nicolas B. Pierron [:nbp]]

(In reply to Nathan Froyd (:froydnj) from comment #0)
> Unreported {
>   ~8,074 blocks in heap block record 2 of 8,862
>   ~34,026,846 bytes (~33,203,306 requested / ~823,540 slop)
>   5.03% of the heap (10.70% cumulative)
>   11.87% of unreported (25.26% cumulative)
>   Allocated at {
>     #01: replace_malloc
> (/opt/src/repos/mozilla-central.hg/memory/replace/dmd/DMD.cpp:1113)
>     #02: pod_malloc<unsigned char>
> (/opt/src/repos/mozilla-central.hg/js/src/vm/MallocProvider.h:63)
>     #03: js::jit::CodeGenerator::link(JSContext*,
> js::types::CompilerConstraintList*)
> (/opt/src/repos/mozilla-central.hg/js/src/jit/CodeGenerator.cpp:7570)
>     #04: ~AutoUnlockHelperThreadState
> (/opt/src/repos/mozilla-central.hg/js/src/vm/HelperThreads.h:424)
>     #05: js::InvokeInterruptCallback(JSContext*)
> (/opt/src/repos/mozilla-central.hg/js/src/jscntxt.cpp:1007)
>     #06: ??? (???:???)
>   }
> }

h4writer: Since when are we linking code under an interruption callback?

> These are the biggest offenders, I can see several more with ~2.5MB
> allocated, and a long tail of what look like related stacks.
> 
> Surfacing this in about:memory would be nice.  Reducing this amount would
> also be nice.

I do not think that this would be the IonCode allocation since we have it since a long time.  On the other hand I wonder if this could be the fact that we inline generateCompactNativeToBytecodeMap?

djvj: 2 questions, Can you investigate this bug? Are you sure it is "Compact"?

For reporting memory, we need to instrument the code this way: https://wiki.mozilla.org/Platform/Memory_Reporting

Comment 2

[Hannes Verschore [:h4writer]]

(In reply to Nicolas B. Pierron [:nbp] from comment #1)
> (In reply to Nathan Froyd (:froydnj) from comment #0)
> > Unreported {
> >   ~8,074 blocks in heap block record 2 of 8,862
> >   ~34,026,846 bytes (~33,203,306 requested / ~823,540 slop)
> >   5.03% of the heap (10.70% cumulative)
> >   11.87% of unreported (25.26% cumulative)
> >   Allocated at {
> >     #01: replace_malloc
> > (/opt/src/repos/mozilla-central.hg/memory/replace/dmd/DMD.cpp:1113)
> >     #02: pod_malloc<unsigned char>
> > (/opt/src/repos/mozilla-central.hg/js/src/vm/MallocProvider.h:63)
> >     #03: js::jit::CodeGenerator::link(JSContext*,
> > js::types::CompilerConstraintList*)
> > (/opt/src/repos/mozilla-central.hg/js/src/jit/CodeGenerator.cpp:7570)
> >     #04: ~AutoUnlockHelperThreadState
> > (/opt/src/repos/mozilla-central.hg/js/src/vm/HelperThreads.h:424)
> >     #05: js::InvokeInterruptCallback(JSContext*)
> > (/opt/src/repos/mozilla-central.hg/js/src/jscntxt.cpp:1007)
> >     #06: ??? (???:???)
> >   }
> > }
> 
> h4writer: Since when are we linking code under an interruption callback?

Since always?! That is the normal way to link.

I recently only recently added lazy linking. In which case we adjust the JSScript::ion to point to a link stub. Which gets called before running the relevant JSScript::ion pointer.

Comment 3

[Kannan Vijayan [:djvj]]

(In reply to Nicolas B. Pierron [:nbp] from comment #1)
> djvj: 2 questions, Can you investigate this bug? Are you sure it is
> "Compact"?

I can, but not immediately.  I'm fixing a couple of slightly related memory leaks associated with the native2bytecode mapping code.  Adding memory reporting to that shouldn't be too difficult, but how high priority is this?

The overhead for these data structures should be somewhere along the line of 5% of jitcode size.