Last Comment Bug 625305 - (about:compartments) about:compartments
(about:compartments)
: about:compartments
Status: RESOLVED WONTFIX
[MemShrink:P2]
: footprint
Product: Core
Classification: Components
Component: XPConnect (show other bugs)
: Trunk
: x86 Windows XP
: -- normal with 3 votes (vote)
: ---
Assigned To: Nicholas Nethercote [:njn]
:
: Andrew Overholt [:overholt]
Mentors:
https://wiki.mozilla.org/Compartment_...
Depends on: 571249
Blocks: MemShrinkTools
  Show dependency treegraph
 
Reported: 2011-01-13 00:39 PST by Mike Shaver (:shaver -- probably not reading bugmail closely)
Modified: 2011-06-21 21:10 PDT (History)
34 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
about:compartments v1 (29.76 KB, patch)
2011-01-13 00:39 PST, Mike Shaver (:shaver -- probably not reading bugmail closely)
no flags Details | Diff | Splinter Review
screenshot (24.53 KB, image/png)
2011-01-13 00:42 PST, Mike Shaver (:shaver -- probably not reading bugmail closely)
no flags Details
updated: string-count, object-count, function-count, off-heap-object-data (3.65 KB, patch)
2011-02-04 21:46 PST, Mike Shaver (:shaver -- probably not reading bugmail closely)
no flags Details | Diff | Splinter Review
updated screenshot (57.58 KB, image/png)
2011-02-04 21:48 PST, Mike Shaver (:shaver -- probably not reading bugmail closely)
no flags Details
better atom accounting, gc-heap arena tracking (3.52 KB, patch)
2011-02-05 12:03 PST, Mike Shaver (:shaver -- probably not reading bugmail closely)
no flags Details | Diff | Splinter Review
updated patch: mjit-code-net, script-count, gc-heap, string-data, string-count, object-count, function-count, off-heap-object-data, atom-count, atom-data (36.70 KB, patch)
2011-02-05 14:57 PST, Mike Shaver (:shaver -- probably not reading bugmail closely)
no flags Details | Diff | Splinter Review
classify heap kinds, add tjit data, classify scripts (38.64 KB, patch)
2011-02-05 16:09 PST, Mike Shaver (:shaver -- probably not reading bugmail closely)
no flags Details | Diff | Splinter Review
patch for tracking cumulative memory usage (20.36 KB, patch)
2011-02-07 16:09 PST, Bill McCloskey (:billm)
no flags Details | Diff | Splinter Review

Description Mike Shaver (:shaver -- probably not reading bugmail closely) 2011-01-13 00:39:36 PST
Created attachment 503431 [details] [diff] [review]
about:compartments v1

I present a first cut of about:compartments, which tells you how much method JIT memory and how many gcBytes are being used by each current compartment.  Lots more to do (JS heap accounting, figuring out what to do with the DOM and layout and imglib memory, trace-jit stuff).  Dunno if it's interesting for FF4 -- new code doesn't run unless you go to about:compartments, by risk-minimization design.

I don't want to talk about how long it took to write those ~500 new lines of code.
Comment 1 Mike Shaver (:shaver -- probably not reading bugmail closely) 2011-01-13 00:42:39 PST
Created attachment 503432 [details]
screenshot

You don't have to tell me it's ugly.
Comment 2 Rob Campbell [:rc] (:robcee) 2011-01-13 05:56:58 PST
it's beautiful.
Comment 3 Rob Campbell [:rc] (:robcee) 2011-01-13 10:03:50 PST
did a quick skim through the code. This looks pretty sweet and could give us the basis for some real memory monitors(!).

There are some empty method stubs in here, and presumably a bit of work to do before it's ready for a blast through the try servers, but I'm pretty eager to get this kind of support into the platform so we can start creating memory monitors.

nice work!
Comment 4 Rob Campbell [:rc] (:robcee) 2011-01-13 10:04:24 PST
I'm also curious if there are any platform limitations on this. Will this run everywhere? Just Windows?
Comment 5 Mike Shaver (:shaver -- probably not reading bugmail closely) 2011-02-04 21:46:04 PST
Created attachment 509985 [details] [diff] [review]
updated: string-count, object-count, function-count, off-heap-object-data

Updated to include more stuff, and properly constrain to a compartment.
Comment 6 Mike Shaver (:shaver -- probably not reading bugmail closely) 2011-02-04 21:46:30 PST
Rob: it'll run everywhere, it's all platform-agnostic code.
Comment 7 Mike Shaver (:shaver -- probably not reading bugmail closely) 2011-02-04 21:48:06 PST
Created attachment 509986 [details]
updated screenshot

I took out gcLastBytes after I took the screenshot, and it's shrunk down a bit to show more interesting data -- 12MB of mjit code for facebook/plugins/recommendations!
Comment 8 Mike Shaver (:shaver -- probably not reading bugmail closely) 2011-02-05 12:03:30 PST
Created attachment 510040 [details] [diff] [review]
better atom accounting, gc-heap arena tracking

Next up: trace JIT memory, and then I move onto DOM structures!
Comment 9 Mike Shaver (:shaver -- probably not reading bugmail closely) 2011-02-05 14:57:50 PST
Created attachment 510054 [details] [diff] [review]
updated patch: mjit-code-net, script-count, gc-heap, string-data, string-count, object-count, function-count, off-heap-object-data, atom-count, atom-data

I'm still not very good at mq.
Comment 10 Mike Shaver (:shaver -- probably not reading bugmail closely) 2011-02-05 16:09:37 PST
Created attachment 510064 [details] [diff] [review]
classify heap kinds, add tjit data, classify scripts

tjit-code-net doesn't include everything I want it to yet, because some of the allocators don't expose their balance through public-enough interfaces.

http://webkit.org/perf/sunspider-0.9.1/sunspider-0.9.1/driver.html {
  "mjit-code-net": 1786695,
  "tjit-code-net": 52816,
  "script-count": 865,
  "script-count-mjit": 367,
  "script-count-tjit": 4,
  "string-count": 13,
  "string-data": 140,
  "object-count": 9602,
  "function-count": 8745,
  "off-heap-object-data": 210968,
  "gc-bytes": 39051264,
  "gc-bytes-objects": 18595840,
  "gc-bytes-functions": 6639616,
  "gc-bytes-xml": 0,
  "gc-bytes-strings": 13815808
}
Comment 11 Bill McCloskey (:billm) 2011-02-07 16:09:48 PST
Created attachment 510436 [details] [diff] [review]
patch for tracking cumulative memory usage

This patch overlaps with the ones in this bug. It tracks net and gross usage, as well as cumulative usage (discounting frees). Cumulative numbers seems to be somewhat more stable than the current total, so it might be useful for tuning.

Eventually I'd like to integrate this stuff with the other patches here. For now I'm posting it so that dvander can maybe use it.
Comment 12 Nicholas Nethercote [:njn] 2011-02-08 03:14:28 PST
I reckon ultimately about:memory and about:compartments should be merged, though this wouldn't have to happen immediately.  It'd also be nice if the measurements were less ad hoc -- for example, about:memory's "mapped"/"in use" measurements are just for the heap and exclude executable memory allocated by the JITs.  And it'd be nice if all the sub-listings added up to the totals in an understandable fashion.

As for this bug:  something I'd love to see in about:compartments would be a "Do a GC" button.  Actually, you could have one per compartment to force a compartment GC, plus one to force a global GC.  This would be a huge help for a bug like bug 630932 where I was trying to work out if there's a leak, but I could never quite tell if there was a GC pending that would get rid of a lot of junk.  Would that be difficult?  

(Hmm, "javascript: gc()" suffices?  Though that's presumably a global GC?  Can a compartment GC be done that way?  Also, can the cycle collector be manually triggered?)
Comment 13 Timothy Nikkel (:tnikkel) 2011-02-08 12:35:45 PST
There is garbageCollect in nsIDOMWindowUtils.
Comment 14 Mike Shaver (:shaver -- probably not reading bugmail closely) 2011-02-08 18:58:44 PST
the "includeInTotal" and units will combine to give us the totals you want.  I may come up with a prefix model too, to make some of the hierarchy more straightforward as well.  Probably not until this weekend, though, and I'm not sure this'll make it in 4.
Comment 15 Nicholas Nethercote [:njn] 2011-02-09 14:54:37 PST
(In reply to comment #12)
> I reckon ultimately about:memory and about:compartments should be merged,

I blogged about this and some follow-on thoughts:  http://blog.mozilla.com/nnethercote/2011/02/09/a-vision-for-better-memory-profiling-with-aboutmemory/
Comment 16 Nicholas Nethercote [:njn] 2011-02-10 10:45:03 PST
Comment on attachment 510064 [details] [diff] [review]
classify heap kinds, add tjit data, classify scripts

>+    size_t grossSize() {
>+        size_t gross = 0;
>+        for (ExecutablePool **ep = m_smallAllocationPools.begin();
>+             ep != m_smallAllocationPools.end();
>+             ++ep) {
>+            gross += (*ep)->grossAllocationSize();
>+        }
>+        return gross;
>+    }

Oh, this is insufficient.  m_smallAllocationPools holds up to four ExecutablePools;  it does this just to minimize fragmentation (see bug 616310).  If more than four pools have been created, the ExecutableAllocator will have lost track of them.  (References to the pools will be stored in other places like JITScripts and IC structures so they haven't been lost altogether.)

So ExecutableAllocator will need to have a vector added to it that hang onto all of the pools it has created.  I was planning to do this anyway, because I want to get rid of the reference counting of executable pools and instead just deallocate them all when the compartment is destroyed, and that requires exactly this vector.
Comment 17 Mike Shaver (:shaver -- probably not reading bugmail closely) 2011-02-10 11:15:50 PST
yeah, I know that's wrong.  I don't use it anywhere yet, IIRC.  I was thinking about fixing it to count correctly, and then I saw your cleanup bug, so I figured I'd just wait it out. :)
Comment 18 Nicholas Nethercote [:njn] 2011-03-27 17:47:38 PDT
Will this help identify leaks?  Eg. if I open foo.com and bar.com in two tabs, then close the bar.com tab, if the bar.com compartment stays around that means there's a leak of some kind?
Comment 19 Rick Johnson 2011-04-08 07:04:32 PDT
(In reply to comment #15)
Maybe not merge about:memory and about:compartment memory information?
The compartment resource presentation has significant user relevance and could potentially ascend to be a menu selection and therefore its display requirements may become different -- much more user friendly/interactive.

For now, would simple cross references suffice?  (see "about:compartments" ...)
Comment 20 Nicholas Nethercote [:njn] 2011-05-12 18:45:11 PDT
This patch does a GC-style trace of the live heap.  But that has a couple of problems:

- you can overflow the stack

- it only counts live objects

Bug 571249 will add code to do traversals of the GC arenas (ie. both live and dead objects).  That will be done on a global basis, but will provide a good bases for per-compartment traversals required for this bug.
Comment 21 Nicholas Nethercote [:njn] 2011-06-21 21:10:11 PDT
I'm going to WONTFIX this in favour of bug 661474, which is my preferred way of presenting this information.  Parts of the code from Shaver's patch will live on in bug 666075, however.

Note You need to log in before you can comment on or make changes to this bug.