655642 - Remove "mapped" memory reporter on Windows

Status: RESOLVED DUPLICATE of bug 657327

(Toolkit :: General, defect)

Description

[Nicholas Nethercote [inactive]]

For bug 633653 I used the "privatebytes" measurement on Windows for the "mapped" entry in about:memory.  I assumed that the amount of shared memory used by a process would be insignificant compared to "privatebytes".  (I believe this is true on Linux.)

But attachment 530624 [details] on bug 655257 showed I was wrong -- the "win32/privatebytes" number is smaller than the "malloc/allocated" number.  (In the new terminology, the "mapped" number is smaller than "mapped/heap/used" + "mapped/heap/unused".)  A sizeable chunk of the heap is shared!

AFAICT there's no way on Windows to find out the total amount of address space being used by a process, ie. to find the sum of the private bytes and the shared bytes.  about:memory degrades gracefully in the case where "mapped" is unknown (indeed, we currently do exactly that for old Windows versions that don't provide an API for measuring privatebytes).  So I propose doing that.  The "mapped" number isn't that useful anyway, it's the heap that is most important.

Comment 1

[Nicholas Nethercote [inactive]]

Attachment: patch

This patch:
- Changes the "mapped" reporter on Windows to just return -1, which is
  interpreted by about:memory as "unknown".

- Adds a "private" reporter on Windows which reports the privatebytes number
  in the "Other Measurements" list in about:memory.

- Tweaks how "unknown" values are displayed by about memory.  On Windows the
  "Mapped Memory" tree now looks like this:

    Mapped Memory
    45.56 MB (100.0%) -- mapped [*]
    ├──45.00 MB (98.77%) -- heap
    │  ├──33.44 MB (73.39%) -- used
    │  └──11.56 MB (25.38%) -- unused
    └───0.56 MB (01.23%) -- js
        ├──0.44 MB (00.96%) -- mjit-code
        └──0.13 MB (00.27%) -- tjit-code

  Things to note:
  * There's a "[*]" next to the name.  If you hover the mouse over it you get
    a tool-tip that says "Warning: this memory reporter was unable to compute
    a useful value. The reported value is the sum of all entries
    below 'mapped', which is almost certainly less than the true value."

  * The existing tool-tip description for "mapped" is the usual one;  it now
    explains that the number isn't obtained on Windows.

  * The value used for "mapped" on Windows is just the sum of the children
    entries.

- Updates the chrome test accordingly.

Comment 2

[Nicholas Nethercote [inactive]]

Comment on attachment 531230 [details] [diff] [review]
patch

I've been thinking about this some more.  I've decided I want to make a similar change on all platforms.  More specifically:

- Change "mapped" so that it's just the sum of all "mapped/*" reporters.  This gives it a meaning of "memory that Firefox explicitly asked to be mapped in".  To make it as accurate as possible, we should add reporters for all the places where we call mmap/VirtualAlloc.

- On Linux and Mac, change what's currently reported as "mapped" to "vsize", and put it in the "Other Measurements" list.  That way it's present (for completeness), but it's less prominent;  this is particularly good on Mac where the number is misleadingly huge (eg. 2GB+ at start-up) because it includes tons of shared memory.

Hmm.  Once that's done, the rationale for separating the "Mapped Memory" from the "Used Heap Memory" mostly disappears.  Perhaps they should be merged!  That would make Jesse happy :)

Comment 3

[Bas Schouten (:bas.schouten)]

(In reply to comment #0)
> 
> AFAICT there's no way on Windows to find out the total amount of address
> space being used by a process, ie. to find the sum of the private bytes and
> the shared bytes.  about:memory degrades gracefully in the case where
> "mapped" is unknown (indeed, we currently do exactly that for old Windows
> versions that don't provide an API for measuring privatebytes).  So I
> propose doing that.  The "mapped" number isn't that useful anyway, it's the
> heap that is most important.

FWIW, we know there is such a way, since there's plenty of programs that do :-). (See VMMAP for example) But I agree that your new approach is fine.

Comment 4

[Nicholas Nethercote [inactive]]

(In reply to comment #3)
> > 
> > AFAICT there's no way on Windows to find out the total amount of address
> > space being used by a process, ie. to find the sum of the private bytes and
> > the shared bytes.
> 
> FWIW, we know there is such a way, since there's plenty of programs that do
> :-). (See VMMAP for example)

I'd love to know how to do this;  it would be worth including in the "Other Measurements" list (see bug 657327) if it's not too hard to compuate.  AFAIK you can't do it with GetProcessMemoryInfo(), which is the only Windows memory measurement API that I'm aware of.

Comment 6

[Nicholas Nethercote [inactive]]

Bug 657327 ended up subsuming this bug, as described in comment 2.

Comment 7

[James Ross]

I believe the total virtual address space used by a process is available via the "\Process(*)\Virtual Bytes" performance counter (and GetProcessMemoryInfo's PrivateUsage is "\Process(*)\Private Bytes"). You might also be interested in showing "\Process(*)\Working Set" (total working set, covering shared and private bytes).

The real problem with performance counters is finding yourself (current process) in them, which basically boils down to enumating the \Process instances (with PDH) and matching "\Process(*)\ID Process".