jemalloc malloc_stats seem weird

RESOLVED FIXED

Status

()

Core
Memory Allocator
RESOLVED FIXED
8 years ago
8 years ago

People

(Reporter: vlad, Assigned: Jason Evans)

Tracking

unspecified
Points:
---

Firefox Tracking Flags

(status1.9.2 beta4-fixed)

Details

Attachments

(1 attachment)

malloc stats are reporting some weird numbers.. e.g. with a bunch of big picture blog pages open, I see:

committed: ~117MB
allocated: ~20MB
mapped: ~12MB
dirty: ~2MB

calling stats over and over a few times I see the allocated number drop down to 1-2MB every now and then, sometimes fluctuating between 1MB and 40MB.
Blocks: 515354
I have some preliminary patches for this from Jason, butthey still give wrong results -- apparently the reserve code that we have is really screwing over the current stats calculations.  Given that we don't need it, should we just get rid of it and get back closer to upstream jemalloc?
OS: Windows NT → All
Hardware: x86 → All
(Assignee)

Comment 2

8 years ago
I'm not sure the stats are 100% correct, even with the patch in bug 515211, but it should be a lot easier to fix with that patch in place.  If there are still problems, I'll be happy to take another look.
My 10/29 minefield is now reporting these numbers in about:memory, wondering if it's related to this issue?

Memory mapped:	 56,623,104
Memory in use:	466,442,956

Can I really have more memory in use than mapped?  (And if so, by 9x?)

I'll try closing some tabs and re-measuring, etc.
I closed some tabs, loaded a few other pages, now have

Memory mapped:	1,655,701,504
Memory in use:	508,240,140

I think I need to spend some time understanding what these numbers mean before I bug you any more, though. :-)
I am seeing (on Windows 7):
Memory mapped:	3,271,557,120
Memory in use:	159,525,142

However checking with perfmon and taskmgr, I'm seeing:
Virtual Bytes: 491,655,168 (Peak was 506MB)
Working Set: 294,020K
Private Working Set: 187,860K
Commit Size: 202,708K
(Assignee)

Comment 6

8 years ago
Created attachment 411560 [details] [diff] [review]
Fix committed memory statistics

The major fix in this patch is that the per chunk 'ndirty' is now subtracted from the number of committed pages when an arena chunk is deallocated.  I didn't figure out that this was the main problem until I had improved the accuracy of committed memory tracking throughout the allocator; these are worthwhile (though minor) improvements, so I've left them in the patch.

I was unable to get about:memory working on Linux despite considerable effort, so all of my testing was performed with a test program Vlad gave me some time back.
Attachment #411560 - Flags: review?
Comment on attachment 411560 [details] [diff] [review]
Fix committed memory statistics

Great, thanks jason!  I'll give this a spin and check it in on the trunk.
Attachment #411560 - Flags: review? → review+
Comment on attachment 411560 [details] [diff] [review]
Fix committed memory statistics

This seems to work great -- landing it on trunk now, and asking approval for 1.9.2.  This fixes about:memory on jemalloc platforms.
Attachment #411560 - Flags: approval1.9.2?
http://hg.mozilla.org/mozilla-central/rev/e2b143e9609f
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED

Updated

8 years ago
Assignee: nobody → jasone
Attachment #411560 - Flags: approval1.9.2? → approval1.9.2+
http://hg.mozilla.org/releases/mozilla-1.9.2/rev/b378a1925d56
status1.9.2: --- → final-fixed
You need to log in before you can comment on or make changes to this bug.