The default bug view has changed. See this FAQ.

Make jemalloc_stats.committed meaningful on *nix (where MALLOC_DECOMMIT is not defined)

RESOLVED FIXED in mozilla10

Status

()

Core
Memory Allocator
RESOLVED FIXED
6 years ago
6 years ago

People

(Reporter: Justin Lebar (not reading bugmail), Assigned: Justin Lebar (not reading bugmail))

Tracking

unspecified
mozilla10
x86_64
Linux
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [MemShrink][inbound])

Attachments

(1 attachment)

(Assignee)

Description

6 years ago
With MALLOC_DECOMMIT defined, jemalloc will decommit memory it doesn't think it needs.  When DECOMMIT is not defined, jemalloc madvise(MADV_FREE / MADV_DONTNEED)'s memory it doesn't need.

AFAICT, a page is MADV_FREE'd iff it would be decommitted, were DECOMMIT defined.

For the purposes of jemalloc_stats.committed, a page is "committed" even after it's been MADV_FREE'd.  But OS issues aside (I understand this may not be the case on Mac), a MADV_FREE'd page should be quickly returned to the operating system.

So I propose modifying jemalloc to report a new field, call it jemalloc_stats.live, which is equal to jemalloc_stats.committed when DECOMMIT is defined, and, when DECOMMIT is not defined, is equal to what jemalloc_stats.committed would be if DECOMMIT were defined.  That is, it treats MADV_FREE'd memory as "not live".

I think this should be pretty simple to do, because the machinery is already there to support DECOMMIT.

jemalloc_stats.live would allow us to measure how much of the live heap is fragmented.  Right now, we can only do this meaningfully on Windows.

Bug 674290 is an alternate way to get a similar result, assuming that jemalloc makes the vast majority of anonymous mappings.  But I think it would be good to measure this directly in jemalloc.

This work would of course be overwritten if and when we update to the new version of jemalloc.
(Assignee)

Updated

6 years ago
Summary: Change jemalloc's heap-committed reporting not to include MADV_FREE'ed memory → Add jemalloc_stats.live, which reports how much memory jemalloc is *actually* using
(Assignee)

Comment 1

6 years ago
This is important because it lets us compute how much fragmentation there is in jemalloc on *nix.

Currently, the only way I know of to compute the amount of wasted memory in jemalloc is to compute

 resident/anonymous - (heap-allocated + js-gc-heap).

This gives you the amount of resident memory that isn't due to live heap allocations or the js gc heap.  This value includes all of the heap fragmentation, but it may include other anonymous mappings.  We can't really tell right now.
Whiteboard: [MemShrink]
(Assignee)

Updated

6 years ago
Assignee: nobody → justin.lebar+bug
(Assignee)

Updated

6 years ago
Depends on: 683597
(Assignee)

Updated

6 years ago
Summary: Add jemalloc_stats.live, which reports how much memory jemalloc is *actually* using → Make jemalloc_stats.committed meaningful on *nix (where MALLOC_DECOMMIT is not defined)
(Assignee)

Comment 2

6 years ago
Created attachment 565638 [details] [diff] [review]
Patch v1

Note the dependency on bug 683597.
Attachment #565638 - Flags: review?(khuey)
(Assignee)

Comment 3

6 years ago
I wonder if we can test this somehow, or if we need to resort to prayer.
(Assignee)

Updated

6 years ago
Blocks: 692963
Attachment #565638 - Flags: review?(khuey) → review+
(Assignee)

Comment 4

6 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/371619a15509
Whiteboard: [MemShrink] → [MemShrink][inbound]
https://hg.mozilla.org/mozilla-central/rev/371619a15509
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla10
You need to log in before you can comment on or make changes to this bug.