Remove private byte counter collection from platforms where it doesn't make sense

RESOLVED FIXED

Status

Testing
Talos
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: jmaher, Assigned: jmaher)

Tracking

Trunk
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Assignee)

Description

3 years ago
in looking at a linux private bytes regression (https://bugzilla.mozilla.org/show_bug.cgi?id=1073662#c71), it raises a question of what we should and shouldn't be measuring.

On linux we collect:
Main RSS - from the browser: http://hg.mozilla.org/build/talos/file/2bdbc49134c4/talos/pageloader/chrome/memory.js
Private Bytes - http://hg.mozilla.org/build/talos/file/2bdbc49134c4/talos/cmanager_linux.py#l74
XRes - http://hg.mozilla.org/build/talos/file/2bdbc49134c4/talos/cmanager_linux.py#l124

It could be that one or more of those are not very useful for actually tracking regressions.


On OSX, we track:
Private Bytes - (process list + filter): http://hg.mozilla.org/build/talos/file/2bdbc49134c4/talos/cmanager_mac.py#l33
Main RSS - from the browser: http://hg.mozilla.org/build/talos/file/2bdbc49134c4/talos/pageloader/chrome/memory.js


I suspect some of this is subject to concern as well.


On Windows we collect:
Main_RSS - (from the browser, just like osx and linux)
Private Bytes - http://hg.mozilla.org/build/talos/file/2bdbc49134c4/talos/cmanager_win32.py
% Process Time - http://hg.mozilla.org/build/talos/file/2bdbc49134c4/talos/cmanager_win32.py
Modified Page List Bytes - http://hg.mozilla.org/build/talos/file/2bdbc49134c4/talos/cmanager_win32.py

Please weigh in here on what we should remove from the list, change, or add.
(In reply to Joel Maher (:jmaher) from comment #0)
> On linux we collect:
> Private Bytes -
> http://hg.mozilla.org/build/talos/file/2bdbc49134c4/talos/cmanager_linux.
> py#l74

This tallies up all the mappings flagged as writable. The problem with this is that it doesn't take into account pages that have been MADV_DONTNEED - these mappings technically still have the writable flag, but we know Linux drops the physical pages.

(In reply to Joel Maher (:jmaher) from comment #0)
> On OSX, we track:
> Private Bytes - (process list + filter):
> http://hg.mozilla.org/build/talos/file/2bdbc49134c4/talos/cmanager_mac.py#l33

From GetProcessData (a bit farther up), this seems to actually be the Virtual Memory Size. I don't know how much sense it makes to track this, since we seem to be pretty stable around 3.8 GiB.
(Assignee)

Comment 2

3 years ago
is there a way to adjust our private bytes collection for linux to ignore MADV_DONTNEED?


It sounds like there is a good argument for disabling private bytes for osx.
(In reply to Joel Maher (:jmaher) from comment #2)
> is there a way to adjust our private bytes collection for linux to ignore
> MADV_DONTNEED?

Parse /proc/pid/smaps instead of /proc/pid/maps, it has more info.
(Assignee)

Comment 4

3 years ago
it looks like we had a couple more regression emails (showed up this morning) related to private bytes on linux32, I should take care of this sooner than later.

;glandium, is the format of /proc/pic/smaps the same as maps?  Would the same code in http://hg.mozilla.org/build/talos/file/2bdbc49134c4/talos/cmanager_linux.py#l74 work on /proc/pids/smaps?

If so, I can change that easily and disable the collection for osx10.8!
(In reply to Joel Maher (:jmaher) from comment #4)
> is the format of /proc/pic/smaps the same as maps?  Would the
> same code in
> http://hg.mozilla.org/build/talos/file/2bdbc49134c4/talos/cmanager_linux.
> py#l74 work on /proc/pids/smaps?

From [1], it's quite different. I'm not really sure what the calculation could look at to get what we want - perhaps the 'Locked' size of each mapping? The example puts that number at 0, so I'm not sure if that would work in practice.

Also, note that it says "The /proc/[pid]/smaps file is present only if the CONFIG_PROC_PAGE_MONITOR kernel configuration option is enabled."

[1] http://man7.org/linux/man-pages/man5/proc.5.html
Locked is for mlock()ed pages. Rss is what you probably want. Which would then only differ to the RSS bytes we already capture by the fact that they only apply to writable mappings.
(Assignee)

Comment 7

3 years ago
hmm, this is sounding more and more like the rss we collect from the browser is going to be the most valuable for linux memory collection.  At the very least, lets turn off osx private bytes.
(Assignee)

Comment 8

3 years ago
Created attachment 8524591 [details] [diff] [review]
disable private bytes on osx (1.0)

anyone else can jump in here and veto this.
Assignee: nobody → jmaher
Status: NEW → ASSIGNED
Attachment #8524591 - Flags: review?(wlachance)
Comment on attachment 8524591 [details] [diff] [review]
disable private bytes on osx (1.0)

The code change looks fine, I don't know enough about the state of memory measurement on Linux/Mac to say whether this is the right thing though.
Attachment #8524591 - Flags: review?(wlachance) → review+
(Assignee)

Updated

3 years ago
Blocks: 1101789
(Assignee)

Updated

3 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.