Last Comment Bug 846616 - unclassified heap block (DMD) results from a MemBench run
: unclassified heap block (DMD) results from a MemBench run
Status: NEW
[MemShrink:P2]
:
Product: Core
Classification: Components
Component: General (show other bugs)
: unspecified
: x86_64 Linux
: -- normal with 2 votes (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
:
:
Mentors:
Depends on: 847787
Blocks:
  Show dependency treegraph
 
Reported: 2013-02-28 19:38 PST by Nicholas Nethercote [:njn]
Modified: 2013-03-21 18:43 PDT (History)
9 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
GLContext stuff (20.96 KB, text/plain)
2013-02-28 19:39 PST, Nicholas Nethercote [:njn]
no flags Details
Font/text stuff (24.08 KB, text/plain)
2013-02-28 19:41 PST, Nicholas Nethercote [:njn]
no flags Details
Cache and nsFeedSniffer stuff (9.83 KB, text/plain)
2013-02-28 19:43 PST, Nicholas Nethercote [:njn]
no flags Details
memory report dump (JSON) (1.43 MB, application/json)
2013-02-28 19:55 PST, Nicholas Nethercote [:njn]
no flags Details
use no global context on glx (928 bytes, patch)
2013-02-28 19:59 PST, Benoit Jacob [:bjacob] (mostly away)
no flags Details | Diff | Splinter Review

Description Nicholas Nethercote [:njn] 2013-02-28 19:38:40 PST
I ran MemBench (http://gregor-wagner.com/tmp/mem) under DMD on Linux64.  It opens and closes 150 tabs.  I triggered DMD after they all closed.  The main sources of unreported memory:

- GLContext stuff.

- Font stuff.

- Cache and other stuff.

I'll attach files with details shortly.
Comment 1 Nicholas Nethercote [:njn] 2013-02-28 19:39:44 PST
Created attachment 719802 [details]
GLContext stuff

All up there was ~14 MiB of GLContext stuff;  this file shows most of it.
Comment 2 Nicholas Nethercote [:njn] 2013-02-28 19:41:29 PST
Created attachment 719804 [details]
Font/text stuff

There were some Chinese(?) pages in the 150 tabs, which I think explains the hyphenation stuff.  I don't think there were any Thai pages (is that what libthai.so relates to?) though I'm not certain.
Comment 3 Nicholas Nethercote [:njn] 2013-02-28 19:43:15 PST
Created attachment 719805 [details]
Cache and nsFeedSniffer stuff
Comment 4 Nicholas Nethercote [:njn] 2013-02-28 19:55:19 PST
Created attachment 719809 [details]
memory report dump (JSON)

Just for completeness, here's the memory report JSON dump for a similar run.  View it in about:memory via the buttons at the bottom.
Comment 5 Benoit Jacob [:bjacob] (mostly away) 2013-02-28 19:59:35 PST
Created attachment 719811 [details] [diff] [review]
use no global context on glx

For the GL part, try this please on Linux.

Not landable as-is as that would break the (disabled by default) GLX layers.

If that fixes it that would mean that this leak was induced by our usage of share groups, which might prevent the GL from releasing GL resources. Not clear to me then whether that would be our fault or the GL's but that's a simple fix.

If that does not fix it that almost certainly a plain leak in the GL.

jgilbert: the WebGL context already deletes all the GL objects associated with it on destruction; does the GL context likewise delete remaining objects (e.g. the framebuffer 0)?
Comment 6 Jeff Gilbert [:jgilbert] 2013-02-28 20:08:30 PST
Yes, we delete all the GLContext accessories. *However*, we deal somewhat strangely with objects held by OGL Layers on platforms with shared GL contexts. That's my guess.
Comment 7 John Daggett (:jtd) 2013-02-28 20:59:31 PST
(In reply to Nicholas Nethercote [:njn] from comment #2)
> Created attachment 719804 [details]
> Font/text stuff
> 
> There were some Chinese(?) pages in the 150 tabs, which I think explains the
> hyphenation stuff.  I don't think there were any Thai pages (is that what
> libthai.so relates to?) though I'm not certain.

Er, not sure what DMD is but stacks 1,2,5,6 are in OTS, the sanitizer
for downloadable fonts, stacks 4,7 is in Japanese line breaking code
(nsJISx4051LineBreaker) calling through to Pango, and stack 3 is the
hyphenation code. God knows why Pango is calling libthai in this instance.  I doubt Chinese pages have much to do with hyphenation.
Comment 8 Nicholas Nethercote [:njn] 2013-02-28 21:43:15 PST
> Er, not sure what DMD is

It's a tool for identifying memory allocations that aren't reported and so fall into the "heap-unclassified" bucket in about:memory.  See https://wiki.mozilla.org/Performance/MemShrink/DMD for more details.
Comment 9 Jonathan Kew (:jfkthame) 2013-03-01 02:14:22 PST
(In reply to John Daggett (:jtd) from comment #7)
> (In reply to Nicholas Nethercote [:njn] from comment #2)
> > Created attachment 719804 [details]
> > Font/text stuff
> > 
> > There were some Chinese(?) pages in the 150 tabs, which I think explains the
> > hyphenation stuff.  I don't think there were any Thai pages (is that what
> > libthai.so relates to?) though I'm not certain.
> 
> Er, not sure what DMD is but stacks 1,2,5,6 are in OTS, the sanitizer
> for downloadable fonts, 

Which implies that the tabs involved downloadable fonts, and those font entries are still present. It's possible that if you wait for a couple of minutes, these would expire from cache (they're managed by an expiration timer after the last reference to them is released).

Maybe we could add a memory reporter that would cover these.

> stacks 4,7 is in Japanese line breaking code
> (nsJISx4051LineBreaker) calling through to Pango, and stack 3 is the
> hyphenation code. God knows why Pango is calling libthai in this instance. 
> I doubt Chinese pages have much to do with hyphenation.

The hyphenation won't be for Chinese (or similar), it'll be a page that has hyphens:auto and is tagged with a language, very likely en-US, for which we have a hyphenation dictionary. We load the hyphenation resources on first use, but then we keep them in memory so that we won't kill performance by having to do the (expensive) dictionary-loading operation for every line-break.

I'm not sure if there'd be a good way to report this memory without hacking at libhyphen internals, which is not something I'd be keen to do.

(The hyphenation manager should probably listen for the memory-pressure notification and unload dictionaries when memory is tight; I don't think we have code in place to do that at present. But we definitely want to keep them loaded in the normal case.)
Comment 10 Jonathan Kew (:jfkthame) 2013-03-01 05:11:18 PST
FTR, I filed bug 846690 to discard hyphenation resources on memory-pressure. This won't remove them from DMD in the normal case, but it should mean that if you fire a memory-pressure notification, the memory will be freed (unless the hyphenation code actually leaks).
Comment 11 Nicholas Nethercote [:njn] 2013-03-01 12:27:05 PST
> Which implies that the tabs involved downloadable fonts, and those font
> entries are still present.
> 
> Maybe we could add a memory reporter that would cover these.

Can you file a bug?


> We load the hyphenation resources on first
> use, but then we keep them in memory so that we won't kill performance by
> having to do the (expensive) dictionary-loading operation for every
> line-break.
> 
> I'm not sure if there'd be a good way to report this memory without hacking
> at libhyphen internals, which is not something I'd be keen to do.

Fair enough.  Tracking memory in libraries is a pain, and the biggest remaining shortcoming in about:memory's coverage (Cairo is another example).


> FTR, I filed bug 846690 to discard hyphenation resources on memory-pressure.

Thanks!
Comment 12 John Daggett (:jtd) 2013-03-04 20:57:53 PST
(In reply to Nicholas Nethercote [:njn] from comment #11)
> > Which implies that the tabs involved downloadable fonts, and those font
> > entries are still present.
> > 
> > Maybe we could add a memory reporter that would cover these.
> 
> Can you file a bug?

Done.  Looks like gfxDownloadedFcFontEntry needs to add memory reporter methods.
Comment 13 Nicholas Nethercote [:njn] 2013-03-06 15:23:49 PST
> For the GL part, try this please on Linux.
> 
> Not landable as-is as that would break the (disabled by default) GLX layers.

I tried it and the browser crashed part-way through the MemBench run, presumably on a page that used WebGL.  Any other things I can try?
Comment 14 Benoit Jacob [:bjacob] (mostly away) 2013-03-06 17:58:55 PST
If my patch made it crash where it didn't crash before, then that's very surprising and I'd need to debug this myself to understand what's going on. Sorry, I'm out of ideas at the moment. A tip if you want to get this prioritized by the gfx team: try to see if by chance this reproduces also on Mac. If it does, that would up a lot the priority. We're swamped at the moment.
Comment 15 Nicholas Nethercote [:njn] 2013-03-06 20:24:15 PST
> try to see if by chance this reproduces also on Mac.

I tried that and DMD crashed while getting a stack trace, in a way I haven't seen before :(

Note You need to log in before you can comment on or make changes to this bug.