Closed Bug 551477 Opened 15 years ago Closed 11 years ago

Add a built-in space profiler

Categories

(Core :: General, defect)

Product:
Core
Component:
General
Platform:
x86
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED WONTFIX

People

(Reporter: jseward, Unassigned)

References

(Depends on 1 open bug)

Details

Attachments

(4 files, 6 obsolete files)

Demo heap profile created by patch in comment #60
142.37 KB, text/plain
Details
Standalone heap profiler. Works on win32, linux and mac.
134.35 KB, patch
Details | Diff | Splinter Review
screenshot of about:heap
204.39 KB, image/png
Details
with much better resolution of allocations within jsengine
246.08 KB, patch
Details | Diff | Splinter Review
The following is a placeholder bug following a discussion on irc with jorendorff, bz, ted, w/ apologies in advance if I paraphrase it wrong. It would be nice to have some kind of space profiler facility built into Fx. It appears there is concern about the difficulty in determining what Fx is using memory for, especially in more complex use scenarios (many open tabs, many extensions, many different web domains displayed). Initial discussion assumed that the traversal problem was somehow solvable. Specifically, we would be able to traverse the entire JS graph (easy) and also the entire C++ object graph (conservative scan?) FTR, I am inclined to think of the traversal problem in a pretty abstract sense: we have a combined C++/JS object graph, plus some set of roots into the graph. We will have a traversal mechanism which allows us to visit all nodes in said graph, plus a query mechanism that can ask each node something about itself (what kind are you? who put you here?), although it might be that from C++ objects we can get no such data. The question of what kind of information should or could be collected proved less clear cut. I think Jason was after this: * for each (web) domain, total amount of storage attributed solely to that domain * for each extension, total amount of storage attributed solely to that extension I think I have a more general proposal for this, which I'll put in a following comment, so as to avoid cluttering this initial statement. There may be other metrics which are interesting. Two of the most basic are "what are you" (what kind of object) and "who put you here" (what call stack allocated you?) One other question is: how much can we do without having to add profiling metadata word(s) to each object?
(In reply to comment #0) > * for each (web) domain, total amount of storage attributed solely to > that domain > > * for each extension, total amount of storage attributed solely to > that extension Here's a strawman generalisation, since for sure we don't want to be hardwiring concepts like "domain X" or "extension Y" into the profiler. We have a graph G and a set of roots R, so that each 'r' in R is a pointer to some node 'nd' in G. Imagine we also have a completely arbitrary set T of tags. Each member of T characterises a root in some (arbitrary) way. Each root then has some subset of T associated with it. For example, T could be {"domain is bbc.co.uk", "extension is FlashBlock", "tab number is 57"} So then we can ask the profiler: how much graph is reachable from roots which have the tag "domain is bbc.co.uk" ? Or from roots which have the tag "extension is FlashBlock"? Or from roots which have both of those tags? Is that a useful generalisation? I suspect so. Is it implementable? I don't know.
FWIW, I think it would be fine to do the traversal only on request, as long as we keep enough information around to perform that at any time. I suspect we'd want to be able to gather this information and display it when a user visits about:memory. Maybe it makes sense to just have the underlying API provide the data you just proposed, and then the front-end UI can figure out a way to display it that makes sense?
(In reply to comment #2) > Maybe it makes sense to just have the underlying API provide the data you just > proposed, and then the front-end UI can figure out a way to display it that > makes sense? Yep, that'd be the right approach; I suspect a lot of tools would want to use this data in different ways. Note that exposing a simple query interface would also be perfectly ok, though, since getting the data could well be expensive. But a query could also just be 'give me all the data'.
It seems ok to have an expensive query, as long as we document that the call is expensive. If a user opens about:memory because their Firefox is already screwed, it's probably not a big deal if it takes a little bit to run.
Thinking a bit about the top level implementation choices and their space requirements and intrusiveness. Seems like there's at least three different things we can ask of an object, at heap-census time: (1) What are you? (== what kind of object?) Presumably we can find this out by examining the object itself. Does not require any extra storage. (2) Who put you here? (== what point in the program allocated you?) This requires having extra storage associated with the object, and this is needed for the entire lifetime of the object. Usually done by having an extra word in each object, containing some kind of allocation-point tag. (3) Why are you still alive? (== who's pointing at you, and/or from which root(s) are you reachable?) Requires constant extra storage per object (perhaps a root-set tag?) but that storage is only required transiently at census-time, not for the entire lifetime of the object. (1) and (3) might be acceptable. (2) is massively intrusive because we'd have to change the layout of every object (or do some nasty hack like having a shadow heap containing the allocation-point data), and we'd also have to mess with every allocation point, to insert a tag into the object/shadow-heap. Note that (3) is a generalisation of the tags idea in Comment #1.
Gregor has been working on (2).
(In reply to comment #6) Is there a tracking bug, or some other place I can learn about what he's doing?
(In reply to comment #7) > (In reply to comment #6) > Is there a tracking bug, or some other place I can learn about > what he's doing? There is bug 547327. My approach is to monitor objects that are created within a loop and try to optimize the allocation size for further objects that are created at the same program location.
Julian, is there anything I can do to help move this along? Do you need your car washed? Meals cooked? Heaven and earth moved? Is there anybody you want rubbed out? Andreas maybe? Andreas, we need this feature, you're going to have to take one from the team.
I meant "for the team", but either way.
Ted pointed me to this bug from here: http://groups.google.com/group/mozilla.dev.platform/browse_thread/thread/f2008793142b9abb/49e00f7d41b70ce1?hl=en&lnk=gst&q=memory+profiling+firebug#49e00f7d41b70ce1 One of the most requested features for Firebug is a support for web page memory profiling. Recently we have tried to implement a prototype (based on memory API included in Jetpack prototype). You can see more here: http://getfirebug.com/wiki/index.php/Firebug_Memory_Profiler The first question: what is the relation between the existing Jetpack APIs and this bug? Will the Jetpack be used as a base for the solution? --- So far, there is no way how to get precise size (row bytes) information for e.g. strings, which would be nice addition to some size info already provided. Some other more structured size info (related to images, HTML elements, etc) would be also useful. The prototype we have done for Firebug showed us that one of very useful things (except of the actual amount of memory consumed) is the list of referents (comment #5 - point #3). We have managed to get such list (using Jetpack APIs) since it provides a way how to access the entire graph of JS objects in the current firefox runtime. Note that we needed to filter the graph in order to get JS objects only for the current page (which is how Firebug works, always focusing on the current page) so, having some kind of query interface (as mentioned in comment #3) would help. Also agree with Vladimir that various tools would want to use different way how to present the data to the user. For example, in case of Firebug we also want to integrate the memory-info with existing panels. The second question. As I mentioned, using the Jetpack APIs, it's relatively easy to get the entire graph of meta-data, where each node has its unique ID. The problem is that there is no way how to get JS object ID and/or to get an existing JS object according to the provided ID (I know that JSD provides an APIs to get these IDs for script objects: jsdIScript.tag), which complicates "JS object <-> meta-data" relation. In Firebug it would be extremely helpful to show the meta data in context of real objects on the page (e.g. in the Watch panel, DOM panel, etc.) Could such methods (like: getJSObject(id), getJSObjectID(obj) ) also be part of the memory API? Honza
> The first question: what is the relation between the existing Jetpack APIs and > this bug? Will the Jetpack be used as a base for the solution? I had a look at the Firebug profiler links. It looks like a nice presentation and query environment. I think we're approaching the same problem from opposite ends, though, and I don't think I can answer your questions at present. Here's what I'm looking into. My goal atm is low level enumeration of the graph of objects, with the short-term aim of maximising coverage. I use "object" in the loosest sense here, to mean any lump of memory to which anyone has a pointer or some kind of reference. So I am aiming to cover the JS heap, the C++ heap, and not just stuff under control of the cycle collector, but ad-hoc malloc'd memory too. Given the great diversity of objects that covers, I am thinking about the idea of building a "shadow graph" -- a throwaway structure which is (mostly) constructed when we want a memory census, analysed, and discarded. The shadow graph is composed of shadow nodes, each of which is: start-address length arbitrary-description set-of-out-pointers plus a set of pointers which serve as roots. At this stage the arbitrary-description can be minimal or nothing. The shadow graph can be incomplete or inaccurate. Basically we forage around for information to build it with, then hand it off to an analyser. The analyser is tolerant of missing pointers, pointers to the middle of nodes, pointers to static objects, apparent inconsistencies like nested or overlapping blocks, etc. The graph would be constructed, for JS objects, by a hacked version of JS_DumpHeap. For C++ objects I am thinking to add hooks to memory/mozalloc/mozalloc.h which observe all mallocs/news and frees/deletes. At analysis time any such blocks would be scanned to find pointers to other blocks. This has various advantages: * it gives coverage of C++ and JS objects * without having to get involved with the Cycle Collector * all the analysis code can be in one place, rather than scattered around * the analysis code does not need to be aware of the layout of any of the objects * the only code that does need to be aware of object layouts (and be scattered around) are "abstractor methods", you might say, which take (eg) a JSSquirrellyThing* and give back one or more shadow nodes that represent it * we can incrementally increase the level of detail in the arbitrary-description field as required to support more detailed queries. * it extends naturally to anything else we write an "abstractor method" for, eg, mmap'd blocks * initially the description field can be empty. Even like that we get basic stats on the numbers and sizes of allocated objects, plus reachability information. * the shadow graph doesn't need to be constructed all at the same time, iow we don't necessarily need to stop the entire world to do it. We can do analyses of incomplete or inconsistent graphs if need be. * as soon as the shadow graph is built, the system can continue. We don't have to wait around for expensive reachability or whatever analyses to complete. That can be done by a separate thread. Or even by a different process, offline. * when profiling is inactive it's free in space terms and almost free in time terms. "Almost" because it would still require an extra conditional branch on the mozalloc malloc/new functions to decide not to collect the allocated address/lengths. Anyway. If anyone thinks this is the wrong thing to build, or simply won't work, please yell asap.
(In reply to comment #12) > Anyway. If anyone thinks this is the wrong thing to build, or simply > won't work, please yell asap. If it weren't you saying it, I would say, "you're vastly underestimating the complexity of what you're proposing." But since it *is* you, I'll say, "It sounds too good to be true."
I don't understand what you mean by "abstractor methods", and what they would do.
(In reply to comment #14) > I don't understand what you mean by "abstractor methods", I mean a function which takes an object and returns its essential details for inclusion into the shadow graph. For example, given a pointer "p" to one of these struct Foo { int a; Blah* b; char c; D* d; } the abstractor function must return at a minimum these 3 items: p sizeof(*p) {p->b, p->d} that is, the address range of the object and the set of pointers found within it. That is the minimum info needed to build the shadow graph. The point about intercepting moz_xmalloc et al is that we can extract the above info automatically, for malloc'd blocks. This is just as well since it would obviously be totally infeasible to manually annotate every C++ constructor in the tree. When an malloc-style allocation is made we note the returned pointer and the requested size. That means for a 2 words/block overhead we get a list of all the blocks in the C++ heap. Then at analysis time we can scan each block to find any pointers to other blocks it might hold. This is really just conservative C/C++ garbage collection stuff. I just verified that I can stick a hook in moz_xmalloc et al (thanks cjones!) and see all mallocs and frees going by. So this part of the plan at least looks like it might work. For JS objects and other stuff in the JS heap I will need to write custom abstractor functions. But that's not such a big deal because there's only a limited set of object formats to consider.
Julian, how much of a problem would tagged pointers be for this scheme? We use them in some places in the DOM, and I've been trying to decide whether we should stop doing so... Does it matter for this setup one way or another?
(In reply to comment #16) I don't think that would be a problem -- those bits can just be masked off. It weakens a bit the heuristic used to decide whether something is could be a pointer to a block, because we can no longer reject misaligned values as can't-possibly-be-a-pointer. But I don't think that's a particularly safe heuristic anyway, since it would assume that nobody in the entire system uses tagged pointers. Which I don't believe. So the short answer is: it makes no difference.
As a devil's advocate type question, what is the advantage of this approach versus forking an inert copy of the program or performing a non-fatal core-dump and then probing that static memory snapshot using a python-enabled gdb? Assuming debug symbols are available, gdb can introspect objects and tell when a pointer in memory amounts to an nsCOMPtr/nsRefPtr versus a presumed invariant-controlled backlink and otherwise get a lot of structure knowledge for free. The 'self-introspection' goal could be satisfied by taking a page from roc's chronicle playbook and having the python/gdb driver expose an HTTP loop that consumes and returns JSON. One might even argue that it's cleaner because the patient is never performing analytical surgery on itself. In any event, this is very exciting work you are undertaking and I look forward to the results! You may be interested in trace-malloc's leak-soup logic that does some of the conservative pointer analysis stuff already, apparently: http://mxr.mozilla.org/mozilla-central/source/tools/trace-malloc/
gdb, what's that? I'm a Windows-based JS developer who wants to know where all my memory is going! /be
The use case I proposed when I badgered Julian into filing this bug was "I'm a normal user, and my Firefox is using a lot of memory. How can I find out what's using all that memory?" Ideally we will provide an API that things like about:memory can use which could give a rough idea of how much memory is entrained by each extension and web site. We need to give users tools so that instead of saying "Firefox uses too much memory" they can say "AdBlock uses too much memory when I visit cnn.com". (Then, ideally, the parties responsible can fix their code!)
(In reply to comment #20) > [...] they can say "AdBlock uses too much memory when I visit cnn.com". Yes, I completely agree that's where we want to get to eventually and yes it will no doubt require an API to query the collected data. But first we need the low-level info-collecting machinery in place, since for sure I don't see how we can answer any interesting question without having at a minimum a graph, roots and reachability analysis. Also .. I'm reluctant to build a profiler which only has partial coverage (eg, JS heap only), hence the thinking about the C++ heap at this early stage.
One thing I'd like to ask the assembled wisdom here is, when should a heap census be performed? My current plan is: * immediately after the cycle collector has run, and * immediately after JSGC has run From sticking printfs in various places it appears that a run of CC necessarily causes a run of JSGC (yes?) but not vice versa. So I'm inclined to hack up some logic which does a census after every CC run and after every JSGC run that isn't bracketed by a CC run. Better suggestions / your-analysis-is-bogus style comments welcomed.
Right, I'm not suggesting you need to implement all that, just the underlying bits! It was more in response to Andrew's "why not just use gdb + python" question.
I raised the gdb solution because it seemed like a way to avoid a slippery slope on the C++ analysis front and because forking seems like the fastest way to get the system running again. Specifically, implementing the abstractor methods seems like manually reproducing the work that debug information already provides, and using gdb is much less risk-prone than rolling your own DWARF parser. (Of course, gdb-python or dehydra/treehydra could be used to automatically generate those if the need arose.) It obviously is not the ideal long-term solution for a web developer focused tool. It might be reasonable for a mozilla developer tool, especially as it could be viable for crash-dump analysis. Mainly, I was thinking it could be reasonable as a simpler-to-develop prototype given that I believe Jim Blandy already has the JS side of the house covered for gdb-python ( http://hg.mozilla.org/users/jblandy_mozilla.com/archer-mozilla/ ). The fundamental assumption that the python-gdb way is simpler might be very wrong on my part... fwiw, I believe gdb can run on windows, but I am unclear as to whether it understands the MS compiler's debugging symbols format, which would be the most important thing. In any event, an in-tree solution is even more exciting and useful than an external mechanism with dependencies; I eagerly await the fruits!
I have working, a first cut at machinery that creates a combined C++/JS shadow graph, as per proposal in comment #12. Will clean it up and post a wip patch later this week. This is still far from being useful. For a Fx idling and displaying news.bbc.co.uk, there are about 88k nodes in the graph, of which 84k are C++ and 4k are JS.
Depends on: 563305
What this patch demonstrates: * creation of a combined C++/JS shadow graph * done periodically, at the end of some JS garbage collections * thread-safe-ly, more or less, and without deadlocking the system * reasonable performance (resulting browser is quite usable) * without the profiler's own dynamic memory allocation skewing the results Example output: all C/C++ heap allocations via mozalloc are tracked, and running stats of live blocks maintained. Periodically a 1-liner summary is printed: HEAPPROF 20747: CH: 1605639/1444361/170M total adds/dels/alloc, 161278/14595k live blocks/bytes Distribution of ages at death is also computed but not displayed. After some but not all JS GCs, a full profile is initiated. A shadow graph is built, cleaned up, and some simple stats on nodes and edges computed: HEAPPROF 20747: PROFILE #8: BEGIN creating shadow graph HEAPPROF 20747: profile #8: analysis begin HEAPPROF 20747: raw: 161363 CH blocks, 4200 JS blocks, 207 dups HEAPPROF 20747: initial sanity checks ok HEAPPROF 20747: refinement done, 434094 mapped, 807262 unmapped HEAPPROF 20747: objects/bytes: 4003/46752 JS, 161353/16426453 CH HEAPPROF 20747: edges: 279 JS->JS, 225 JS->CH, 5647 CH->JS, 315904 CH->CH HEAPPROF 20747: outdegree: 504 from JS, 321551 from CH, 322055 total HEAPPROF 20747: indegree: 5926 to JS, 316129 to CH, 322055 total HEAPPROF 20747 18183 objects of 165356 total unreferenced from within the SG HEAPPROF 20747: PROFILE #8: END What next: As it stands, this is more or less useless. It mechanically extracts low level reachability, but knows nothing about each node. One direction is to try to answer "what kind of objects are in the heap?" questions. This could be provided by the JS world without difficulty. For the C++ heap we might be reduced to guessing object types by looking for pointers to vtables within the objects. Another direction is to answer "who put these objects in the heap?" questions. This would require tagging each object with its allocation point. For the C++ heap this could possibly by done by getting stack traces from Breakpad.
(In reply to comment #26) > Another direction is to answer "who put these objects in the heap?" > questions. This would require tagging each object with its allocation > point. For the C++ heap this could possibly by done by getting > stack traces from Breakpad. How about causality tracking? For example, when a DOM click event gets generated it would generate a 'frame' that says "DOM click calling foo.js/func bar()/line 53" and uses that as the tag. When that event-triggered JS initiates a setTimeout it creates a new frame to use when the setTimeout gets dispatched and links to the currently active (event-origin) frame. In order to avoid creating the world's longest linked list, a setTimeout from inside a setTimeout callback would reference the original progenitor instead and boost a counter to reflect that it's the result of a chained setTimeout. It seems like they could just be JS objects and the normal garbage collector would take care of it? The same mechanism could potentially be exposed to C++ consumers with default logic being provided for the event loop/native nsITimers/other low-hanging fruit using the breakpad (or external address->symbol resolution). The startup timeline does some of this and I did a bit more for my recently started and regrettably currently triaged/abandoned Thunderbird memory work. I even went so far as to push state on all XPConnect C++->JS crossings, although I also was interested in the execution and its timing as well as the memory costs.
FYI, Breakpad can only produce stack traces after-the-fact. (We don't ship the debug symbols with the app, which it needs to walk the stack.)
We could expose the symbols via a web service (I don't think the symbol server protocol is fine-grained enough for this) so that the client could request a bunch of symbol/frame factoids and do the walking, if we wanted to. We have the information, can make all sorts of tradeoffs about how to get it to the user, including just pulling them all down the first time the user wants to walk some stack.
Sure, we can do anything (it's just software, right?) I'm just pointing out that as currently implemented, Breakpad isn't really a good fit for the task.
(In reply to comment #29) For sure we can't do anything much interesting with the C++ heap without having symbols or unwind data. Given those two I think we could answer what's-in-the-heap and who-put-it-there questions, which would be a good start. For x86-{linux,darwin} with -fno-omit-frame-pointer (the default) we might be able to get away without unwind data, but pretty much all other platforms require it. I like the pre-canned factoid idea for a couple of reasons: * With careful design of the factoids, they can be platform independent. Almost all of the hassle about unwinding the stack and producing traces is in reading the debuginfo. So this scheme minimises the extra complexity compiled in to the browser, putting it instead in the factoid server. * At least for ELF/Dwarf3, the frame unwind info is stored in a form which is compact but not suitable for fast unwinding, so the Dwarf3->Factoid server could reformat it to be more suitable for fast unwinding.(*) There's lots of stuff on the web about Microsoft symbol servers, but I don't see anything for other platforms. Does there exist cross-platform versions of the technology? (*) partially evaluate the Dwarf3 CFI for each address range, producing, for each range, a canned summary of how to compute the integer registers of the calling frame from the values in this frame.
Didn't Vlad do some factoid-like work already? /be
I accidentally opened a dupe on this topic in order to point out http://www.eclipse.org/mat/ , which from my perspective offers a sort of high-point of the UI you might want. (Note in particular the pleasantly informative choice of showing dominator trees)
(In reply to comment #26) > For the C++ heap we might be reduced to guessing object > types by looking for pointers to vtables within the objects. I suspect you can get pretty far with this sort of thing. There are limits of course, but I think you can get a fair bit out of the existing XPCOM infrastructure. Could dig in an allocation looking for a pointer to <a href="https://developer.mozilla.org/en/Using_nsIClassInfo">nsIClassInfo</a>, and/or could tailor a small non-vtbl mixin class for decorating other non-XPCOM / non-virtual types with a pointer to a tiny amount of metadata (particularly: "your module and type name, an offset to a member pointer that's your owner, and/or offset to a member pointer to an object from which you inherit a sense of site-origin"). If you can infer an ownership-tree out of the graph dominators, a few unidentified intermediate nodes along a chain leading to, say, self-identifying image data, DOM nodes or JS objects probably isn't a big hazard to comprehension. I'd focus on this before worrying about the "who allocated you" side of the equation, and building a stack-crawling system. Just getting a characterization of the contents of a really bad heap would be a good start. The nice thing about bundling this in the standard build is we'll get a whole lot more sampling from the field, cases where sore spots are really standing out, not just "developer sandbox testcases".
(In reply to comment #35) I tried for a while to do a quick hack to get ELF symbol information into the process, so as to see if any info can be scraped out of heap blocks. But got nowhere; dlsym on Linux won't say anything about private data symbols, which most vtables are. Plus it's unportable and no use for field runs. I'm sure there's a benefit to be had from reading debug info, but it seems like it'll take serious integration with breakpad (in some way) to make it workable and cross-platform. > but I think you can get a fair bit out of the existing XPCOM > infrastructure. Could dig in an allocation looking for a pointer to <a > href="https://developer.mozilla.org/en/Using_nsIClassInfo">nsIClassInfo</a>, At profile time I am merely faced with a collection of blocks that malloc/new have noted. I don't know which of those I can safely run QueryInterface on. What would be nice is for the compiler to automatically pass a uniquely identifying text tag to each allocation. I can't find a way to do that in the general case. But for classes that use XPCOM, we're in luck. Every participating class has NS_DECL_ISUPPORTS, NS_DECL_CYCLE_COLLECTING_ISUPPORTS or NS_DECL_ISUPPORTS_INHERITED in its declaration. By adding to those macros the following public: \ static void* operator new (size_t size) { \ return moz_malloc_tagged(size, (char*)__PRETTY_FUNCTION__); \ } \ static void operator delete (void *p) { moz_free(p); } \ it is possible to make a class-specific override of new/delete, which passes __PRETTY_FUNCTION__ onwards, and that string contains the class name. Wiring all that together makes possible to generate stats like this (edited to make it less wide): objects/bytes: 118343/12905257 in CH, 29901/2965424 in CH-tagged [ 0] objects/bytes: 2000 / 480000 sStandardURL [ 1] objects/bytes: 4251 / 408096 CSSStyleRuleImpl [ 2] objects/bytes: 3165 / 278520 nsTextNode [ 3] objects/bytes: 2369 / 265328 XPCWrappedNative [ 4] objects/bytes: 876 / 147168 nsEventListenerManager [ 5] objects/bytes: 1533 / 147168 nsXULElement [ 6] objects/bytes: 2962 / 118480 AtomImpl [ 7] objects/bytes: 673 / 91528 nsHTMLAnchorElement [ 8] objects/bytes: 237 / 45504 nsLocalFile [ 9] objects/bytes: 156 / 41184 imgRequest (many lines deleted) This says: in the C++ heap there are currently 118343 blocks comprising 12905257 bytes. Of these, 29901 blocks (2965424 bytes) have a tag. Followed by a per-class breakdown for the tagged blocks, most voluminous first. The full profile has 447 lines, so at least 447 classes have been covered. What this also shows is that only about 1/4 of the blocks are identified. Since it covers (almost) all XPCOM classes, this gives a quick upper bound on what coverage we can get from XPCOM. It would be nice to expand the coverage. I am wondering if adding a single line to other allocation points would not be a bad tradeoff. It could be done incrementally.
Julian, I hate to suggest this, but I think the biggest benefit here would be to get this working under Win32 initially; there at least I know that the pdb files have all the info we need to resolve vtable pointers, and the dbghelp library makes doing that straightforward. I went off on a bit of a tangent yesterday and wrote a quick tool to walk the entire Win32 heap (win32, not jemalloc), dumping the first 4 bytes of each allocation for later vtable analysis. It returns all that to js, and then dbghelp or other tools (map files also come to mind -- we could make this completely cross platform if we standardized on one map file format) can be used via ctypes/etc. to translate to symbol names. I'll get that patch up and cc you.
(In reply to comment #37) > Julian, I hate to suggest this, but I think the biggest benefit here would be > to get this working under Win32 initially; there at least I know that the pdb > files have all the info we need to resolve vtable pointers, and the dbghelp > library makes doing that straightforward. I think this is a really bad idea. This should be a tool in *every build* of the program. Not "some developer builds, sometimes, when they happen to have set things up right". You should always be able to take a browser -- when it's up at 1gb resident after a week of browsing on a normal desktop and you don't know why -- and just ask it "why?" The problem we face here is that in "our testing", when we do it on talos or under controlled conditions, we have these lovely graphs that say we don't use much memory and we always release it and everything is lovely. And then users in the field keep posting back screenshots from week-long sessions with a few dozen tabs and a modest set of extensions saying "no, you don't, it's really out of control and my system is paging again and it's firefox's fault". Then we turn around and point at our pretty graphs and tell them they must be imaging things because it's well-behaved when we tested it. We need to close that gap, and that means gathering better *field data*.
I think we're saying the same thing -- I'm pointing out that 90% of our users are on win32, so getting that data would be a lot more useful than gathering linux data, and that there's a standardized way to obtain all the symbol data that we'd need via the symbol server, plus a simple library that we can use to query that data. All stuff that makes it possible to actually gathering actual field data :-) We could also look into teaching breakpad how to spit out text map files so that we have a standard "symbol data" format without having to write platform specific code to get it as well. But right now, as far as I know, win32 is the only platform where we have infrastructure to say "give me symbols for release/nightly build with signature abcd".
(In reply to comment #39) > I think we're saying the same thing -- I'm pointing out that 90% of our users > are on win32, so getting that data would be a lot more useful than gathering > linux data, I understand what you're saying, and I'm not disagreeing. However, I'm not trying to gather linux data. I'm trying to see how far it's possible to get without any external assistance, a la Graydon comments. I'm inclining more towards a source-oriented approach -- what can we do to get the compiler to add this info automatically? For one thing, vtable scraping is going to fail to tell us anything for straight malloc'd blocks, or for classes in which no vtable is required.
(In reply to comment #36) > What this also shows is that only about 1/4 of the blocks are > identified. Since it covers (almost) all XPCOM classes, this gives a > quick upper bound on what coverage we can get from XPCOM. 25% coverage of the heap is pretty useless. I wondered how many other allocation points would have to be hand annotated to improve that significantly. Problem is there are thousands of allocation points (constructors, mostly). On the assumption that allocation probably follows a 90/10 rule, I bootstrapped the process by hacking the profiler to record the top 5 frames of each allocation, commoned those up, and looked at the sources corresponding to the 5-frame groups that allocated the most. This tells me where to put hand annotations so as to maximise coverage of heap contents. An afternoon of doing this increased coverage to 80% of heap bytes, sometimes more, via 35 one-liner annotations, mostly like so MOZALLOC_PROF_TAG(this); in constructors, since MOZALLOC_EXPORT void moz_prof_tag(void* ptr, const char* tag); #define MOZALLOC_PROF_TAG(__ptr) \ moz_prof_tag((__ptr), __PRETTY_FUNCTION__) A sample profile is attached (see next comment), for Fx with 7 tabs open. It shows coverage of 74% of bytes and 87% of objects for a set of 177,885 live blocks that have been allocated through mozalloc. The lines marked "static void* C::operator new..." are allocations tagged automatically by the trick in Comment #36. All others are from hand-applied annotations.
Attached file heap profile as per Comment #41 (obsolete) —
I support the standalone from all side-files, works in product builds, source based approach. When in doubt, use brute force. If we have to mangle our source a bit, it could be worthwhile compared to the old manglings we put up with all over for dubious benefit. /be
Julian, how does your addition of operator new stuff work for classes that already have NS_DECL_AND_IMPL_ZEROING_OPERATOR_NEW?
This is cool, though I suspect people will immediately say "what's the stack trace for static void* nsCSSCompressedDataBlock::operator new(size_t, size_t)?" and you'll end up reinventing Massif's call tree structure. But that's ok, if it can be done within the browser so that anyone can use it that would be a big win. Minor comments about the profile output format: - Commas in big numbers (eg. 123,456) make them much easier to read; - Please mark each entry with the percentage of the total that it represents.
(In reply to comment #44) > Julian, how does your addition of operator new stuff work for classes that > already have NS_DECL_AND_IMPL_ZEROING_OPERATOR_NEW? Yes, an excellent question. Basically it doesn't. If a class overrides new and delete itself then adding a second override in NS_DECL_ISUPPORTS causes the compiler to complain. What I _really_ did was to rename NS_DECL_ISUPPORTS to NS_DECL_ISUPPORTS_NO_NEW_OVERRIDE (possibly a bad name :-) and to then do this #define NEW_OVERRIDES \ public: \ static void* operator new (size_t size) { \ return moz_malloc_tagged(size, (char*)__PRETTY_FUNCTION__); \ } \ static void operator delete (void *p) { moz_free(p); } \ #define NS_DECL_ISUPPORTS \ NS_DECL_ISUPPORTS_NO_NEW_OVERRIDE \ NEW_OVERRIDES (ditto with NS_DECL_CYCLE_COLLECTING_ISUPPORTS and NS_DECL_ISUPPORTS_INHERITED) Then, everywhere where g++ complained about duplicate overrides, changed NS_DECL_ISUPPORTS to NS_DECL_ISUPPORTS_NO_NEW_OVERRIDE (viz, back to what it was originally). From a bit of greppery, there are about 1700 uses of NS_DECL_ISUPPORTS_*, of which 23 had to be _NO_NEW_OVERIDE'd. So not a big hit. That said there was some complexity I didn't figure out yet. Even after the compiler accepted everything, in a few cases possibly to do with multiple inheritance, I was seeing uses of uninitialised values somehow to do with the automatic overriding, leading to segfaults, and I had to back off the the _NO_NEW_OVERRIDE versions for those classes. Possibly about 6 cases.
(Stuff I forgot to say in Comment #46) ... consequently, classes that are marked _NO_NEW_OVERIDE have to be hand-annotated using MOZALLOC_PROF_TAG as per Comment #41. MOZALLOC_PROF_TAG is a second way to get tag information into the profiler. It facilitates setting the tag on a block after the block has been allocated. The ideal of course is to set the tag at allocation time, a la moz_malloc_tagged in Comment #46.
(In reply to comment #42) > Created an attachment (id=449124) [details] > heap profile as per Comment #41 As a light diversion from the "what should we build?" question: in this profile, no less than 37% of the objects in the heap (66.6k out of a total of 177k objects) have to do with CSS. That seems a lot. Is that expected? HEAPPROF 16701: objects/bytes: 177885/18446274 in CH, 155018/13662436 in CH-tagged RANK OBJECTS/BYTES ALLOCATED BY 0 9131 / 1827388 static void* nsCSSCompressedDataBlock::operator new(size_t, size_t) 1 19084 / 1374048 nsCSSSelector::nsCSSSelector() 4 8525 / 818400 static void* CSSStyleRuleImpl::operator new(size_t) 9 8491 / 407568 nsCSSDeclaration::nsCSSDeclaration() 12 10306 / 247344 nsCSSSelectorList::nsCSSSelectorList() 17 6484 / 155616 nsCSSValueList::nsCSSValueList() 27 1747 / 69880 nsCSSValuePairList::nsCSSValuePairList() 29 1017 / 50568 nsCSSValue::URL::URL(nsIURI*, nsStringBuffer*, nsIURI*, nsIPrincipal*) 35 1156 / 36992 nsPseudoClassList::nsPseudoClassList(nsIAtom*, nsCSSPseudoClasses::Type) 59 354 / 16992 static void* nsDOMCSSAttributeDeclaration::operator new(size_t) 61 114 / 15504 static void* nsCSSStyleSheet::operator new(size_t) 103 57 / 4560 static void* CSSNameSpaceRuleImpl::operator new(size_t) 104 81 / 4536 static void* nsCSSRuleProcessor::operator new(size_t) 126 90 / 2880 static void* CSSImportantRule::operator new(size_t)
That seems like a large proportion, true; but you're also looking at a profile that (if I read correctly) only relates to ~18mb resident? So like .. a very small snapshot. Possibly quite un-representative of the program later on in its life. I assume this is just from a prefix of startup? Can you get a profile once it's been browsing for a while and, say, passes the 150mb resident mark? Might need to up the numbers in your static array.
> Is that expected? It could be, depending on what's being measured. You mentioned 7 tabs. What were they? Was gmail involved, say?
(In reply to comment #49) > That seems like a large proportion, true; but you're also looking at a profile > that (if I read correctly) only relates to ~18mb resident? Yes .. I was also suspicious about that. By running a profiled build on Massif, which intercepts all mallocs and frees, it's possible to see what proportion of heap allocations are routed through mozalloc and hence are intercepted. The news isn't good. For a startup with 4 tabs, intercepting mozalloc shows the heap volume peaking at around 29.4M. Massif puts that figure at a more credible 65.0M, meaning more than half the heap allocation bypasses mozalloc and so is invisible to the built-in profiler. I'm not sure what to do about this. It would be possible to force (eg) jsengine to route allocations through mozalloc, but that would create a new intermodule dependence. And there's a significant amount of heap allocated by libX11.so.6.3.0 and libxcb.so.1.1.0 which we can't reroute at the source level. One ungood kludge would be to write a shared object redefining malloc, free, etc and LD_PRELOAD it into the process. That of course only works on Linux and perhaps MacOSX. Anyway, some stats from the above run. I should add, all these figures exclude the overhead of the built-in profiler itself (about 18M), to make the comparison fair. ============ Built-in intercepts in mozalloc give a peak heap use of 29.4M. Massif says 65.0M. where did the rest go? Well, approximately: 12,737,468 0x704B744: moz_calloc (mozalloc.cpp:137) 8,484,176 0x704B81B: moz_malloc_tagged (mozalloc.cpp:105) 6,564,989 0x704B87D: moz_xmalloc (mozalloc.cpp:91) p 12737468 + 8484176 + 6564989 $1 = 27786633 So 27.8 MB was noted by Massif as going via mozalloc; that ties in at least approximately with the 29.4M figure above. For the rest: 11,023,595 in 735 places, all below massif's threshold (01.00%) PR_Malloc calls malloc directly: 4,754,242 PR_Malloc (prmem.c:467) js_malloc calls malloc directly: 3,832,065 js_NewScript (jsutil.h:193) 3,478,750 JS_ArenaAllocate (jsutil.h:193) 2,269,696 JS_DHashAllocTable (jsutil.h:193) 1,401,504 JSScope::create(JSContext*, JSObjectOps const*, JSClass*, JSObject*, unsigned int) (jsutil.h:193) 904,564 js_NewStringCopyN (jsutil.h:193) system libraries presumably call malloc directly: 3,142,400 XGetImage (in /usr/lib/libX11.so.6.3.0) 1,571,264 ??? (in /usr/lib/libxcb.so.1.1.0) extensions/spellcheck/hunspell/src/hashmgr.cpp calls malloc directly: 2,685,946 HashMgr::add_word(char const*, int, int, unsigned short*, int, char const*, bool) (hashmgr.cpp:188) ## Hmm, does this mean we're allocating 2.7MB merely to initialise ## a spell checker that didn't get used in this run? XPT_ArenaMalloc calls calloc directly: 1,126,648 XPT_ArenaMalloc (xpt_arena.c:221) sqlite3 calls malloc directly: 1,038,456 sqlite3MemMalloc (sqlite3.c:12975)
(In reply to comment #50) > > Is that expected? > > It could be, depending on what's being measured. You mentioned 7 tabs. What > were they? Was gmail involved, say? No gmail. I don't recall exactly, but I think the tabs were http://lwn.net/Articles http://www.youtube.com http://en.wikipedia.org/wiki/Main_Page and the other 4 were articles accessible from http://www.spiegel.de/international
bug 559263 has some changes to make us wrap malloc more thoroughly with mozalloc. I'm not sure if those apply on all platforms or just Android. cjones would be the one to talk to about mozalloc hijinks, in any case.
The other major one is the JS GC chunks, though that may already go through the right place on Linux. That's better now that Vlad has landed the hooks, though.
> No gmail. I don't recall exactly, but I think the tabs were OK. We'd have to take a look at their source to see whether the numbers make sense, but they're at least plausible...
(In reply to comment #51) > It would be possible to force (eg) jsengine to route > allocations through mozalloc, but that would create a new intermodule > dependence. And there's a significant amount of heap allocated by > libX11.so.6.3.0 and libxcb.so.1.1.0 which we can't reroute at the > source level. > > One ungood kludge would be to write a shared object redefining malloc, > free, etc and LD_PRELOAD it into the process. That of course only > works on Linux and perhaps MacOSX. > So, two notes --- first, mozalloc has no dependencies by design, so using it in js/src would be a relatively simple matter of getting jseng-folk consent and shuffling around sources. This would work on all platforms. Second, we already use linker tricks on linux and windows (!!) to replace system malloc with jemalloc, so we could re-use those hacks to point malloc at a version with your instrumentation. This would take a bit of work. Mac would be left out. (In reply to comment #53) > bug 559263 has some changes to make us wrap malloc more thoroughly with > mozalloc. I'm not sure if those apply on all platforms or just Android. cjones > would be the one to talk to about mozalloc hijinks, in any case. The android code uses GNU ld's --wrap, so linux only (for all intents and purposes).
(In reply to comment #56) > shuffling around sources. This would work on all platforms. Second, we > already use linker tricks on linux and windows (!!) to replace system malloc > with jemalloc, so we could re-use those hacks to point malloc at a version with > your instrumentation. I tried with --enable-wrap-malloc, but got into various build system/linkage difficulties, and gave up. Sticking with the source route for now.
(In reply to comment #51) > Built-in intercepts in mozalloc give a peak heap use of 29.4M. > Massif says 65.0M. I routed js_malloc, XPT_ArenaMalloc and the allocator in db/sqlite3 through mozalloc. This significantly improves coverage. For a startup of Fx with about 100ish tabs, Massif gives a peak C++ heap allocation size of 342.9MB, and the mozalloc interceptor sees 252.0MB of it (perhaps a bit more). That's 73.5%. Most of the rest escapes via PR_Malloc, and it might be possible to reroute a couple of its high-roller callers to mozalloc, so a figure of 80+% is possible, I'd say. That's sufficiently encouraging that I'd like to turn my collection of experimental hacks into a proper patch. However, I have one structural problem unsolved. The interceptor (which is in mozalloc) really needs a lock/unlock facility. Right now I'm kludging it by calling pthread_mutex_{lock,unlock} directly; obviously that won't work on Windows. I need to use the locking facilities in NSPR instead, but mozalloc doesn't depend on NSPR. I could I guess copy JSThinLock into mozalloc, so as to keep mozalloc standalone, but that's not ideal. Any suggestions?
jslock.{cpp,h} depend on NSPR -- a thin lock can fatten into something requiring a PRLock. If you just need a mutex, but also must stand alone, best to use raw Windows API and ifdef. Shouldn't be too bad. /be
Standalone C++ heap profiler, patch against m-c 46349:4aadb903f941 (Mon Jun 28 09:46:49 2010 -0700). Provides who-allocated-what-in-the-C++-heap analysis without external assistance. As per comment #51, covers about 80% of the C++ heap volume. Of that 80%, about 95% of the volume is tagged by allocation point, using the schemes in comment #41 and comment #36. In a quick experiment with 6 tabs, it is tracking about 42MB of blocks, whereas "about:memory" lists about 12.5MB at that point. Linux only for the time being. Working on extending it to MacOS and Windows. Should be jemalloc-agnostic, although that is untested. HOW TO USE: Profiling is enabled/disabled at Fx startup, by default disabled. To enable, start with env var MOZALLOC_HEAPPROF set to anything. When running, it monitors mallocs/frees (etc). Every 20000 such calls, it prints a summary line showing the current total volumes: 1456357/1183643/945M total allocs/frees/totalloc, 272714/40802k live blocks/bytes Periodically the (tracked part of the) heap is sampled and a census printed, one line per allocation point ("tag"), showing the number of blocks and bytes billed to each tag. A census is performed when the maximum volume has expanded by more than 4% (over 50MB) or by 2MB (under 50MB). This means that you are guaranteed to have a census within 4% of peak residency, and it will be the most recently displayed one. There is some space overhead (probably less than 10% of the allocated heap) when profiling is enabled, and some extra processing per allocation. Browser is still usable though. The extra space overhead is not shown in the generated profiles. When disabled, the overheads should be very close to zero (1 extra conditional branch per malloc and per free).
Attachment #445563 - Attachment is obsolete: true
For a startup/quit of the browser, with 6 tabs.
Attachment #449124 - Attachment is obsolete: true
Version that works on both Linux and MacOSX, at least for 10.5.8, 32-bit build with gcc-4.2. Also, rebase to m-c 46969:d9253109fd88 (Thu Jul 01 14:40:49 2010 +0200).
Attachment #454639 - Attachment is obsolete: true
Now with win32 support (at least, works on WinXP, MSVC9, debug and release builds). Patch relative to m-c 46969:d9253109fd88
Attachment #455470 - Attachment is obsolete: true
This now works on the biggest 3 targets. I'd like to move it towards land-ability. At this point, some level of feedback/review seems to me necessary. Patch provides functionality as described in comment #60. However, there are various kinds of rough edges: (from the user's aspect) * how should profiling be enabled? atm env var MOZALLOC_HEAPPROF is tested at startup; if set, profiling is enabled. This doesn't strike me as good. I thought about using a pref, but really we want the profiler to be enabled or disabled right from process start. * where should output go? currently it's sent to stderr (sending it to stdout breaks jsengine building). (from the Fx code base aspects) * Patch routes much more malloc/free traffic through mozalloc than previously. This means these modules acquire a new dependency on mozalloc: * js * db/sqlite3 I'm sure this totals standalone builds of js (haven't tried). Even with those routed through mozalloc, we only capture about 80% of heap allocation, so potentially more modules should be routed through mozalloc. [IMO, it no longer makes sense to have mozalloc separate from NSPR; they should be combined.] * When not profiling, there is an extra overhead of 1 memory reference (read) and 1 conditional branch per alloc and free. IMO this is insignificant in that (according to my own measurements) the behaviour of Fx in big runs is on the order of 1 malloc and 1 free per 50000 instructions. * As per 3rd para in comment #60, supposing we wanted to add more allocation tags to cover the 5% of the heap that is currently tagged as UNKNOWN (we know an allocation is happening, but we don't know is doing it.) Finding the point in the sources to annotated by hand requires some auxiliary mechanism to get a stack trace. I have been using a hacked valgrind. Given that the profiler is supposed to be standalone, I can't see how the dependence on some external tool to "bootstrap" it can be avoided. Just mentioning for completeness.
(from the Fx code base aspects in comment #64) += * the scheme described in comment #36 provides automatic tagging of allocations made by all nsCOMPtr classes. It does this by stuffing a per-class override of new and delete into the NS_DECL_*ISUPPORTS macros. Since there are a couple thousand of such classes, there will be some cost in terms of object code size. I'll measure.
* shows profiling results in "about:heap" * rebased to m-c 48706:a4d86c3a3494 (Mon Aug 02 01:28:10 2010 -0700) WIP. Not re-tested on OSX or Win. Appears to have stability problems (occasional heap block overwrites) that weren't there before.
Attachment #464408 - Attachment is obsolete: true
Not pretty, but possibly useful. Output is in two parts: * the most recent census (detailed breakdown of contents). Since a census is only made when the heap high-water mark rises, this is always a picture of the heap at or close to the max for the run so far. * periodic one-line summaries, to give some feel for changes over time.
Revised version of comment 67 patch, with about 180 allocation sites in js/src/*.{cpp,h} (that is, calls to js_malloc/_calloc/_realloc) hand-annotated with allocation tags. This gives much better resolution into jsengine's allocations. Prior to this change the profiler would say (eg) that lots of stuff was allocated by js_realloc, which isn't helpful -- see screenshot in comment #68. Now it can say who is making those calls. Still requires refinement to handle allocated-on-behalf-of scenarios. Eg, saying that 4.5% of the heap is allocated by PL_DHashAllocTable is pretty useless. Am thinking about tagging each block with 2 strings rather than one -- primary and secondary tags. So then it would be able to say, effectively: "X% of the heap was allocated by PL_DHashAllocTable, on behalf of nsFoo::bar" (where nsFoo::bar is using PL_DHashAllocTable to create/maintain a hash table). Still has stability problems. I believe this is the same problem as documented in bug 587610, and is unrelated to the profiler per se.
Attachment #464410 - Attachment is obsolete: true
jseward: given the presence of about:memory and DMD, I think we can safely close this bug. Please reopen if you disagree.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: