Last Comment Bug 655642 - Remove "mapped" memory reporter on Windows
: Remove "mapped" memory reporter on Windows
Status: RESOLVED DUPLICATE of bug 657327
:
Product: Toolkit
Classification: Components
Component: General (show other bugs)
: unspecified
: All Windows 7
: -- normal (vote)
: ---
Assigned To: Nicholas Nethercote [:njn]
:
Mentors:
: 658806 (view as bug list)
Depends on:
Blocks: MemShrinkTools
  Show dependency treegraph
 
Reported: 2011-05-08 19:56 PDT by Nicholas Nethercote [:njn]
Modified: 2011-06-07 06:08 PDT (History)
4 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
patch (11.61 KB, patch)
2011-05-09 19:39 PDT, Nicholas Nethercote [:njn]
no flags Details | Diff | Review

Description Nicholas Nethercote [:njn] 2011-05-08 19:56:16 PDT
For bug 633653 I used the "privatebytes" measurement on Windows for the "mapped" entry in about:memory.  I assumed that the amount of shared memory used by a process would be insignificant compared to "privatebytes".  (I believe this is true on Linux.)

But attachment 530624 [details] on bug 655257 showed I was wrong -- the "win32/privatebytes" number is smaller than the "malloc/allocated" number.  (In the new terminology, the "mapped" number is smaller than "mapped/heap/used" + "mapped/heap/unused".)  A sizeable chunk of the heap is shared!

AFAICT there's no way on Windows to find out the total amount of address space being used by a process, ie. to find the sum of the private bytes and the shared bytes.  about:memory degrades gracefully in the case where "mapped" is unknown (indeed, we currently do exactly that for old Windows versions that don't provide an API for measuring privatebytes).  So I propose doing that.  The "mapped" number isn't that useful anyway, it's the heap that is most important.
Comment 1 Nicholas Nethercote [:njn] 2011-05-09 19:39:38 PDT
Created attachment 531230 [details] [diff] [review]
patch

This patch:
- Changes the "mapped" reporter on Windows to just return -1, which is
  interpreted by about:memory as "unknown".

- Adds a "private" reporter on Windows which reports the privatebytes number
  in the "Other Measurements" list in about:memory.

- Tweaks how "unknown" values are displayed by about memory.  On Windows the
  "Mapped Memory" tree now looks like this:

    Mapped Memory
    45.56 MB (100.0%) -- mapped [*]
    ├──45.00 MB (98.77%) -- heap
    │  ├──33.44 MB (73.39%) -- used
    │  └──11.56 MB (25.38%) -- unused
    └───0.56 MB (01.23%) -- js
        ├──0.44 MB (00.96%) -- mjit-code
        └──0.13 MB (00.27%) -- tjit-code

  Things to note:
  * There's a "[*]" next to the name.  If you hover the mouse over it you get
    a tool-tip that says "Warning: this memory reporter was unable to compute
    a useful value. The reported value is the sum of all entries
    below 'mapped', which is almost certainly less than the true value."

  * The existing tool-tip description for "mapped" is the usual one;  it now
    explains that the number isn't obtained on Windows.

  * The value used for "mapped" on Windows is just the sum of the children
    entries.

- Updates the chrome test accordingly.
Comment 2 Nicholas Nethercote [:njn] 2011-05-13 06:02:10 PDT
Comment on attachment 531230 [details] [diff] [review]
patch

I've been thinking about this some more.  I've decided I want to make a similar change on all platforms.  More specifically:

- Change "mapped" so that it's just the sum of all "mapped/*" reporters.  This gives it a meaning of "memory that Firefox explicitly asked to be mapped in".  To make it as accurate as possible, we should add reporters for all the places where we call mmap/VirtualAlloc.

- On Linux and Mac, change what's currently reported as "mapped" to "vsize", and put it in the "Other Measurements" list.  That way it's present (for completeness), but it's less prominent;  this is particularly good on Mac where the number is misleadingly huge (eg. 2GB+ at start-up) because it includes tons of shared memory.

Hmm.  Once that's done, the rationale for separating the "Mapped Memory" from the "Used Heap Memory" mostly disappears.  Perhaps they should be merged!  That would make Jesse happy :)
Comment 3 Bas Schouten (:bas.schouten) 2011-05-16 08:42:41 PDT
(In reply to comment #0)
> 
> AFAICT there's no way on Windows to find out the total amount of address
> space being used by a process, ie. to find the sum of the private bytes and
> the shared bytes.  about:memory degrades gracefully in the case where
> "mapped" is unknown (indeed, we currently do exactly that for old Windows
> versions that don't provide an API for measuring privatebytes).  So I
> propose doing that.  The "mapped" number isn't that useful anyway, it's the
> heap that is most important.

FWIW, we know there is such a way, since there's plenty of programs that do :-). (See VMMAP for example) But I agree that your new approach is fine.
Comment 4 Nicholas Nethercote [:njn] 2011-05-16 17:46:23 PDT
(In reply to comment #3)
> > 
> > AFAICT there's no way on Windows to find out the total amount of address
> > space being used by a process, ie. to find the sum of the private bytes and
> > the shared bytes.
> 
> FWIW, we know there is such a way, since there's plenty of programs that do
> :-). (See VMMAP for example)

I'd love to know how to do this;  it would be worth including in the "Other Measurements" list (see bug 657327) if it's not too hard to compuate.  AFAIK you can't do it with GetProcessMemoryInfo(), which is the only Windows memory measurement API that I'm aware of.
Comment 5 Nicholas Nethercote [:njn] 2011-05-22 01:48:19 PDT
*** Bug 658806 has been marked as a duplicate of this bug. ***
Comment 6 Nicholas Nethercote [:njn] 2011-05-22 22:22:30 PDT
Bug 657327 ended up subsuming this bug, as described in comment 2.

*** This bug has been marked as a duplicate of bug 657327 ***
Comment 7 James Ross 2011-06-07 06:08:06 PDT
I believe the total virtual address space used by a process is available via the "\Process(*)\Virtual Bytes" performance counter (and GetProcessMemoryInfo's PrivateUsage is "\Process(*)\Private Bytes"). You might also be interested in showing "\Process(*)\Working Set" (total working set, covering shared and private bytes).

The real problem with performance counters is finding yourself (current process) in them, which basically boils down to enumating the \Process instances (with PDH) and matching "\Process(*)\ID Process".

Note You need to log in before you can comment on or make changes to this bug.