Open
Bug 563700
(DarkMatter)
Opened 15 years ago
Updated 2 years ago
[meta] Reduce about:memory's "heap-unclassified" number to (typically) 10%
Categories
(Core :: General, defect)
Core
General
Tracking
()
NEW
People
(Reporter: vlad, Unassigned)
References
(Depends on 31 open bugs, )
Details
(Keywords: meta)
Attachments
(2 files)
This is a tracking bug for the various about:memory reporters that need to be added.
Updated•14 years ago
|
Updated•14 years ago
|
Blocks: MemShrinkTools
Updated•14 years ago
|
Depends on: revampaboutmemory
Here is one testcase that about:memory currently doesn't account for:
1. open any website (e.g. http://google.com)
2. execute the following bookmarklet:
javascript:if(undefined == document.body.asdf){document.body.asdf = []};var a=document.body.asdf;var i,j,tmp;for(i=0;i<10000;i++){tmp=[];for(j=0;j<10000;j++){tmp[j]="a"} a.push(tmp)};alert("done");void(0);
This simply attaches a huge nested array to the DOM tree (yeah, i know, bad practice) and gobbles up about 1GB of memory which doesn't show up in about:memory.
If you don't get the "done" alert the script possibly failed due to an OOM abort, you'll have to close the tab and try again with lower numbers for the loop counts.
Comment 3•14 years ago
|
||
JSScripts should be tracked. With 10 gmail tabs open I saw them account for 17% of the live heap. Here's one call stack from Massif:
o1o> 17.03% (70,660,318B) 0x67648A4: JSScript::NewScript(JSContext*, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigne
d int, unsigned int, unsigned int, unsigned short, unsigned short, JSVersion) (jsutil.h:239)
| o2o> 16.64% (69,033,931B) 0x67675CE: JSScript::NewScriptFromCG(JSContext*, JSCodeGenerator*) (jsscript.cpp:1260)
| | o3o> 15.73% (65,262,175B) 0x66DBFCE: js_EmitFunctionScript (jsemit.cpp:3678)
| | | o4o> 15.67% (65,014,983B) 0x66D9BE7: js_EmitTree (jsemit.cpp:4640)
| | | | o5o> 08.47% (35,117,714B) 0x66D7723: js_EmitTree (jsemit.cpp:6176)
| | | | | o6o> 08.46% (35,109,464B) 0x66D585A: js_EmitTree (jsemit.cpp:5984)
| | | | | | o7o> 08.13% (33,744,883B) 0x66D8622: js_EmitTree (jsemit.cpp:5892)
| | | | | | | o8o> 08.07% (33,496,774B) 0x66D636D: js_EmitTree (jsemit.cpp:5451)
Comment 4•14 years ago
|
||
JSObject slots should be tracked, ie. in allocSlots(), growSlots() and shrinkSlots(). With 10 gmail tabs open I saw them account for 7% of the live heap.
Comment 5•14 years ago
|
||
Also various strings are important, eg:
o1o> 11.67% (48,410,287B): nsStringBuffer::Alloc() (nsSubstring.cpp:209)
| o2o> 10.55% (43,754,212B): nsXMLHttpRequest::ConvertBodyToText() (nsXMLHttpRequest.cpp:762)
| | o3o> 10.55% (43,754,212B): nsIXMLHttpRequest_GetResponseText() (dom_quickstubs.cpp:26975)
| | o4o> 10.06% (41,739,008B): js_NativeGet() (jscntxtinlines.h:326)
| | | o5o> 10.06% (41,739,008B): InlineGetProp() (StubCalls.cpp:1868)
| | | | o6o> 10.06% (41,739,008B): js::mjit::stubs::GetProp() (StubCalls.cpp:1894)
Not sure how best to record them -- lump all strings together, or try to be smarter than that.
Comment 6•14 years ago
|
||
Here's the raw data I'm getting these ideas from, BTW.
Comment 7•14 years ago
|
||
The Hunspell spell-check accounts for 2.7MB on my machine, it looks like it's a constant amount. See bug 653817 comment 7.
Comment 8•14 years ago
|
||
js::PropertyTable::init() and js::PropertyTable::change() together account for 4.3% in the gmail example.
Updated•14 years ago
|
My testcase from comment #2 still shows up under heap-unclassified. Considering that it is trivial to use up memory like that and many poorly written javascript libraries attach all their state to dom objects i would say it would be useful to add instrumentation for that case.
Comment 10•13 years ago
|
||
(In reply to comment #2)
> Here is one testcase that about:memory currently doesn't account for:
>
> 1. open any website (e.g. http://google.com)
> 2. execute the following bookmarklet:
> javascript:if(undefined == document.body.asdf){document.body.asdf = []};var
> a=document.body.asdf;var
> i,j,tmp;for(i=0;i<10000;i++){tmp=[];for(j=0;j<10000;j++){tmp[j]="a"}
> a.push(tmp)};alert("done");void(0);
Oh, bug 571249 will fix that. (Well, the follow-up bug 664647 will do so.)
I just tried your JS snippet in a build with the js/object-slots reporter enabled (plus per-compartment reporters from bug 661474):
1,624,547,222 B (100.0%) -- explicit
├──1,494,174,525 B (91.97%) -- js
│ ├──1,314,183,902 B (80.90%) -- compartment(http://www.google.com.au/)
│ │ ├──1,310,986,032 B (80.70%) -- object-slots
│ │ ├──────1,436,610 B (00.09%) -- scripts
│ │ ├────────983,040 B (00.06%) -- mjit-code
│ │ ├────────572,112 B (00.04%) -- tjit-data
│ │ │ ├──444,000 B (00.03%) -- allocators-reserve
│ │ │ └──128,112 B (00.01%) -- allocators-main
│ │ ├────────131,072 B (00.01%) -- tjit-code
│ │ ├─────────69,676 B (00.00%) -- mjit-data
│ │ └──────────5,360 B (00.00%) -- string-chars
So that's good! :)
Depends on: 664913
Updated•13 years ago
|
Alias: MemoryReporters
Updated•13 years ago
|
Component: General → about:memory
Product: Core → Toolkit
QA Contact: general → about.memory
Comment 11•13 years ago
|
||
I've changed this back to Core/General; although the numbers show up in about:memory, the reporters themselves are mostly in Core, and about:memory doesn't need changing for them to show up once they are added.
Component: about:memory → General
Product: Toolkit → Core
QA Contact: about.memory → general
Updated•13 years ago
|
Updated•13 years ago
|
Component: General → about:memory
Product: Core → Toolkit
QA Contact: general → about.memory
Comment 12•13 years ago
|
||
Oh, sorry for changing the component without reading this.
Updated•13 years ago
|
Component: about:memory → General
Product: Toolkit → Core
QA Contact: about.memory → general
Comment 13•13 years ago
|
||
I'm changing this bug's title and repurposing it slightly. Instead of being a simple tracking bug for new reporter's, it's a bug for getting the "heap-unclassified" number (which is currently typically around 35--50%) down as low as feasible. Obviously, bugs for new reporters will still block this bug.
All of the things identified above have been spun-off into separate bugs (some of which have been fixed) except for the nsXMLHttpRequest strings one in comment 5.
A couple more stack traces of interest from the Massif data:
o1o> 05.00% (20,748,856B) nsCSSExpandedDataBlock::Compress() (mozalloc.h:229)
| o2o> 04.99% (20,688,600B) (anonymous namespace)::CSSParserImpl::ParseDeclarationBlock() (Declaration.h:124)
| | o3o> 04.91% (20,371,668B) (anonymous namespace)::CSSParserImpl::ParseRuleSet() (nsCSSParser.cpp:2612)
| | | o4o> 04.91% (20,349,128B) nsCSSParser::Parse() (nsCSSParser.cpp:963)
| | | | o5o> 04.91% (20,349,128B) mozilla::css::Loader::ParseSheet() (Loader.cpp:1585)
| | | | o6o> 04.86% (20,156,400B) mozilla::css::Loader::LoadInlineStyle() (Loader.cpp:1807)
| | | | | o7o> 04.86% (20,156,400B) nsStyleLinkElement::DoUpdateStyleSheet() (nsStyleLinkElement.cpp:298)
| | | | | o8o> 04.76% (19,743,840B) nsStyleLinkElement::UpdateStyleSheetInternal() (nsStyleLinkElement.cpp:209)
o1o> 03.81% (15,797,388B) nsTextFragment::SetTo() (nsMemory.h:68)
| o2o> 03.81% (15,797,388B) nsGenericDOMDataNode::SetTextInternal() (nsGenericDOMDataNode.cpp:350)
| o3o> 02.50% (10,366,215B) nsHtml5TreeOperation::AppendText() (nsHtml5TreeOperation.cpp:197)
| | o4o> 02.50% (10,366,215B) nsHtml5TreeOperation::Perform() (nsHtml5TreeOperation.cpp:531)
| | o5o> 02.48% (10,300,277B) nsHtml5TreeOpExecutor::RunFlushLoop() (nsHtml5TreeOpExecutor.cpp:489)
| | | o6o> 02.45% (10,160,236B) nsHtml5ExecutorFlusher::Run() (nsHtml5StreamParser.cpp:153)
o1o> 02.92% (12,120,960B) ChangeTable (pldhash.c:563)
| o2o> 02.91% (12,068,864B) PL_DHashTableOperate (pldhash.c:649)
| | o3o> 01.02% (4,233,984B) AddSelector() (nsCSSRuleProcessor.cpp:2608)
| | | o4o> 01.02% (4,233,984B) nsCSSRuleProcessor::RefreshRuleCascade() (nsCSSRuleProcessor.cpp:2715)
| | |
| | o3o> 00.68% (2,839,040B) RuleHash::AppendRuleToTable() (nsCSSRuleProcessor.cpp:528)
| | | o4o> 00.68% (2,822,656B) RuleHash::AppendRule() (nsCSSRuleProcessor.cpp:560)
| | |
| | o3o> 00.33% (1,367,040B) mozilla::FramePropertyTable::Set() (nsTHashtable.h:188)
| | | o4o> 00.32% (1,329,152B) nsIFrame::GetOverflowAreasProperty() (FramePropertyTable.h:237)
| | | |
| | | o4o> 00.01% (37,888B) in 6 places, all below massif's threshold (00.10%)
| | |
| | o3o> 00.24% (1,009,152B) AppendRuleToTagTable() (nsCSSRuleProcessor.cpp:540)
| | | o4o> 00.22% (897,024B) RuleHash::AppendRule() (nsCSSRuleProcessor.cpp:565)
| | |
| | o3o> 00.21% (888,832B) nsDocument::AddToIdTable(mozilla::dom::Element*, nsIAtom*) (nsTHashtable.h:188)
o1o> 02.87% (11,917,512B) nsCSSSelectorList::AddSelector() (mozalloc.h:229)
| o2o> 02.87% (11,907,936B) (anonymous namespace)::CSSParserImpl::ParseSelectorGroup() (nsCSSParser.cpp:3644)
o1o> 01.63% (6,742,008B) (anonymous namespace)::CSSParserImpl::ParseRuleSet() (mozalloc.h:229)
| o2o> 01.62% (6,713,136B) nsCSSParser::Parse() (nsCSSParser.cpp:963)
| | o3o> 01.62% (6,713,136B) mozilla::css::Loader::ParseSheet() (Loader.cpp:1585)
| | o4o> 01.59% (6,613,920B) mozilla::css::Loader::LoadInlineStyle() (Loader.cpp:1807)
| | | o5o> 01.59% (6,613,920B) nsStyleLinkElement::DoUpdateStyleSheet() (nsStyleLinkElement.cpp:298)
| | | o6o> 01.56% (6,488,640B) nsStyleLinkElement::UpdateStyleSheetInternal() (nsStyleLinkElement.cpp:209)
See the attachment for full details.
Looks like we really need some CSS memory reporters! There were a couple of >1% stack traces for CSS things that I haven't copied in here either.
Alias: MemoryReporters
Summary: [tracking] about:memory reporters → Reduce about:memory's "heap-unclassified" number
Whiteboard: [MemShrink:P1]
Comment 14•13 years ago
|
||
It was suggested at today's MemShrink meeting that it might be worth someone digging into memory dumps or something else to push this along. But the relevant information is actually all there in Massif's output. The hard part of interpreting that output is knowing how the allocation traces Massif reports match up with existing memory reporters. But a similar problem would occur for anyone who dug through memory dumps.
I think a better way forward would be to identify sites for which our heap-unclassified number is unusually high. Comment 2 and bug 669005 are both good examples (and both are now fixed). I know Massif is a pain to run on Firefox, so I'm happy to run it on any such sites that people can find.
Or if you just know of any remaining decent-sized chunks of memory that aren't covered, please speak up. (Bug 669117 was identified this way by khuey, IIRC.)
Comment 15•13 years ago
|
||
If we want a snappy name for this bug, I propose "Dark Matter".
Alias: darkmatter
Updated•13 years ago
|
Summary: Reduce about:memory's "heap-unclassified" number → [meta] Reduce about:memory's "heap-unclassified" number
Comment 16•13 years ago
|
||
On Linux64, if I load just about:memory?verbose and http://blogs.technet.com/b/dmelanchthon/archive/2011/07/19/featurematrix-outlook-client-und-exchange-server-reloaded.aspx, I get a "heap-unclassified" of ~50%. The next highest bucket is "js" with 32%. Full output is attached. That page includes lots of tables.
Comment 17•13 years ago
|
||
That actually suggests an interesting approach to tracking down additional darkmatter: automatically check the heap-unclassified numbers for a bunch of webpages, and look at sites that have unusually large numbers to see what is weird about them.
Relatedly, in https://bugzilla.mozilla.org/show_bug.cgi?id=670689#c23 bz says: "And by the way, I'm guessing that in this case the remaining unclassified stuff is also somewhere in the js engine. Might be worth it to look up where."
Comment 18•13 years ago
|
||
(In reply to comment #17)
> That actually suggests an interesting approach to tracking down additional
> darkmatter: automatically check the heap-unclassified numbers for a bunch of
> webpages, and look at sites that have unusually large numbers to see what is
> weird about them.
Yes. See comment 14 :)
Comment 19•13 years ago
|
||
Nice catch with Bug 675132. Is it possible to use Massif (which I assume is recording the amount requesting of every malloc?) to see how much space is being wasted by allocations-slightly-more-than-powers-of-two? Or that something you already set up to ferret this out? The cycle collector allocates memory in chunks, and I could imagine it accidentally wasting a bit of space in the same way, so it could probably be happening all over the place.
Comment 20•13 years ago
|
||
(In reply to comment #19)
> Is it possible to use Massif (which I assume is
> recording the amount requesting of every malloc?) to see how much space is
> being wasted by allocations-slightly-more-than-powers-of-two?
Nope. Massif replaces the heap allocator with its own version, and the amount of rounding-up is a characteristic of the heap allocator. Well, I guess I could try to get Massif to round up the same as jemalloc, but there's an easier path forward -- see bug 675136 where I've found that jemalloc rounding could be accounting for something like 17% of "heap-allocated", i.e. roughly half of "heap-unclassified".
Comment 21•13 years ago
|
||
Just to clarify about Massif: it makes a half-hearted attempt to account for extra space needed by the allocator -- it adds 8 bytes to the amount recorded for each heap block, on the assumption that the average overhead is that size. You can change it with the --heap-admin option, but it's always a constant. So it's pretty hopeless. Also, that overhead-per-block is recorded separately from the requested-amount-per-block and it's not even shown in those allocation trees that I cut and paste into bugs all the time. So I haven't even considered jemalloc's rounding until just today.
tl;dr: Massif is hopeless for measuring extra space caused by jemalloc's rounding and book-keeping.
Comment 22•13 years ago
|
||
Yeah, I meant doing some kind of post-processing step in Massif to round up the same as jemalloc, but looks like you cut out the middle man.
Comment 23•13 years ago
|
||
I was thinking some more about how to get *serious* about hunting down all the heap-unclassified bytes.
You'd add instrumentation code to maintain a data structure that records 1 bit of information for every malloc'd byte. A 0 means "not reported", a 1 means "reported". You'd also add instrumentation code to record, for every heap block, the stack trace at its allocation point.
Then, you'd modify every heap memory reporter so that when it is queried, it sets the "reported" bits of all the heap bytes it counts.
Once about:memory was loaded, you'd iterate over all the heap blocks. Any heap block fully reported would be ignored. Any heap block partially or not-at-all reported would be recorded. You'd aggregate repeated stack traces for recorded heap blocks and present the stack traces in order so that the ones responsible for the most unreported bytes are shown first.
Also, while the memory reporters are doing their thing, you'd complain about any byte on the heap whose "reported" bit was set more than once -- such bytes are double-counted. You could print out its stack trace and the name of the reporter that was second to count it.
Tracking one-bit-per-byte metadata and per-heap-block stack traces is something that Valgrind excels at, BTW. A Valgrind tool that implemented this analysis would have to use its own heap allocator rather than jemalloc (that's just how Valgrind works), so it would have to be careful about the rounding up of request sizes, but it could definitely be made to work. Memory reporters would use client requests to tell the tool which bytes have been accounted for. A client request would also be used to tell the tool when all the memory reporters have been consulted.
There's a fairly obvious candidate for who might implement this :P
Comment 24•13 years ago
|
||
I spun the Valgrind tool idea off as bug 676724.
Comment 25•13 years ago
|
||
Bug 291643 should be blocking darkmatter as XML produces lots of heap-unclassified
Comment 26•13 years ago
|
||
I've set a specific goal of 10% dark matter for "typical" cases, so that this bug has an end goal. We can argue about the meaning of "typical" when we're getting close. (With Gmail and this bug open, I currently see 22.93% dark matter.)
Summary: [meta] Reduce about:memory's "heap-unclassified" number → [meta] Reduce about:memory's "heap-unclassified" number to (typically) 10%
Comment 27•13 years ago
|
||
A lot of bugs are of the form "this totally non-typical page generates lots of heap-unclassified." It seems to me that bugs of that form are still worth tracking (perhaps in a separate bug), since it's hard to understand what we're doing wrong on those pages if about:memory fails us.
Updated•13 years ago
|
Comment 28•13 years ago
|
||
Something that bug 699951 made me think of is that there's one situation that's hard to measure with DMD, because it's so slow -- long sessions with lots of browsing. (I've left DMD running overnight before, but that doesn't involve opening/closing lots of sites.)
Comment 29•13 years ago
|
||
Nicholas, what about Bug 646575?
Comment 30•13 years ago
|
||
(In reply to Phoenix from comment #29)
> Nicholas, what about Bug 646575?
What about it?
Comment 31•13 years ago
|
||
Sholdn't it be added to this list too?
Comment 32•13 years ago
|
||
It's unclear to me how bug 646575 relates to this bug. Can you explain?
Updated•13 years ago
|
Assignee: nobody → n.nethercote
Updated•13 years ago
|
Alias: darkmatter → DarkMatter
Comment 33•13 years ago
|
||
Nicholas, do you know if there are some memory reporters for the memory use of plugin-container.exe (I mean especially about Flash/Java/Silverlight)?
Does it count explicitly in 'Other Measurements' in about:memory?
Comment 34•13 years ago
|
||
(In reply to Loic from comment #33)
> Nicholas, do you know if there are some memory reporters for the memory use
> of plugin-container.exe (I mean especially about Flash/Java/Silverlight)?
> Does it count explicitly in 'Other Measurements' in about:memory?
We currently do not measure plugin-container.exe at all.
Comment 35•13 years ago
|
||
Bug 648415 is open for plugin-container measurement, but no progress has been made on it.
Updated•12 years ago
|
Whiteboard: [MemShrink:P1]
Updated•12 years ago
|
Depends on: B2GDarkMatter
Comment 36•12 years ago
|
||
FWIW, my "heap-unclassified" is regularly under 10% on my Linux64 box, running a trunk build. It was 6.4% at the end of the day a couple of days ago. And it's currently 10.9% on my Mac, which is on FF19 beta.
Comment 37•12 years ago
|
||
Install Stylish and this "issue" would be easily resolved :D
Comment 38•12 years ago
|
||
> Install Stylish and this "issue" would be easily resolved :D
I don't understand what you mean by 'this "issue"'. Is heap-unclassified high with Stylish installed?
Comment 39•12 years ago
|
||
(In reply to Nicholas Nethercote [:njn] from comment #38)
> Is heap-unclassified high with Stylish installed?
Right, you'll get something like 16% from beginning and growing to ~30% on usage
Comment 40•12 years ago
|
||
I'm just curious -- I recognize that this meta-bug is mostly tracking heap-unclassified data in Firefox, and that there's a separate meta-bug for B2G. Is there a comparable bug for Thunderbird? (On my machine at the moment, heap-unclassified is at 33%, which makes it tricky to help diagnose the memory leak I seem to be seeing...)
Comment 41•12 years ago
|
||
> Is there a comparable bug for Thunderbird?
Nope. You might want to talk to jcranmer, who was working on some Thunderbird memory reporters just the other day. Alternatively, DMD might help you: https://wiki.mozilla.org/Performance/MemShrink/DMD
Comment 42•12 years ago
|
||
(In reply to Nicholas Nethercote [:njn] from comment #41)
> > Is there a comparable bug for Thunderbird?
>
> Nope. You might want to talk to jcranmer, who was working on some
> Thunderbird memory reporters just the other day. Alternatively, DMD might
> help you: https://wiki.mozilla.org/Performance/MemShrink/DMD
I tried using DMD on Thunderbird to diagnose bug 844937. All it shows is that Thunderbird desperately needs memory reporters for mailnews/ code, which is the work jcranmer has started in bug 480843.
Updated•9 years ago
|
Updated•4 years ago
|
Assignee: n.nethercote → nobody
Updated•2 years ago
|
Severity: normal → S3
You need to log in
before you can comment on or make changes to this bug.
Description
•