515556 - jemalloc malloc_stats seem weird

Status: RESOLVED FIXED

(Core :: Memory Allocator, defect)

Description

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

malloc stats are reporting some weird numbers.. e.g. with a bunch of big picture blog pages open, I see:

committed: ~117MB
allocated: ~20MB
mapped: ~12MB
dirty: ~2MB

calling stats over and over a few times I see the allocated number drop down to 1-2MB every now and then, sometimes fluctuating between 1MB and 40MB.

Comment 1

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

I have some preliminary patches for this from Jason, butthey still give wrong results -- apparently the reserve code that we have is really screwing over the current stats calculations.  Given that we don't need it, should we just get rid of it and get back closer to upstream jemalloc?

Comment 2

[Jason Evans]

I'm not sure the stats are 100% correct, even with the patch in bug 515211, but it should be a lot easier to fix with that patch in place.  If there are still problems, I'll be happy to take another look.

Comment 3

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

My 10/29 minefield is now reporting these numbers in about:memory, wondering if it's related to this issue?

Memory mapped:	 56,623,104
Memory in use:	466,442,956

Can I really have more memory in use than mapped?  (And if so, by 9x?)

I'll try closing some tabs and re-measuring, etc.

Comment 4

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

I closed some tabs, loaded a few other pages, now have

Memory mapped:	1,655,701,504
Memory in use:	508,240,140

I think I need to spend some time understanding what these numbers mean before I bug you any more, though. :-)

Comment 5

[Rob Arnold [:robarnold]]

I am seeing (on Windows 7):
Memory mapped:	3,271,557,120
Memory in use:	159,525,142

However checking with perfmon and taskmgr, I'm seeing:
Virtual Bytes: 491,655,168 (Peak was 506MB)
Working Set: 294,020K
Private Working Set: 187,860K
Commit Size: 202,708K

Comment 6

[Jason Evans]

Attachment: Fix committed memory statistics

The major fix in this patch is that the per chunk 'ndirty' is now subtracted from the number of committed pages when an arena chunk is deallocated.  I didn't figure out that this was the main problem until I had improved the accuracy of committed memory tracking throughout the allocator; these are worthwhile (though minor) improvements, so I've left them in the patch.

I was unable to get about:memory working on Linux despite considerable effort, so all of my testing was performed with a test program Vlad gave me some time back.

Comment 7

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Comment on attachment 411560 [details] [diff] [review]
Fix committed memory statistics

Great, thanks jason!  I'll give this a spin and check it in on the trunk.

Comment 8

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Comment on attachment 411560 [details] [diff] [review]
Fix committed memory statistics

This seems to work great -- landing it on trunk now, and asking approval for 1.9.2.  This fixes about:memory on jemalloc platforms.

Comment 9

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

http://hg.mozilla.org/mozilla-central/rev/e2b143e9609f

Comment 10

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

http://hg.mozilla.org/releases/mozilla-1.9.2/rev/b378a1925d56