855151 - Automatic release of GCed memory back to system in OS X is less than optimal

Status: NEW

(Core :: Memory Allocator, defect)

Description

[u462496]

Attachment: Test procedure and results

This bug is related to issues found and discussed in Bug 852520, and is related to releasing of memory that has been GCed in Firefox to the system on Mac OS X.

In Bug 852520 it was found that memory GCed in Firefox isn't necessarily released to the system right away, and will only be released if the system is running out of memory.  In order to observe the effectiveness of the intended automatic method of releasing memory, I set up a test scenario where an instance of Firefox had a large amount of memory freed from GC, yet was still not released to the the system.  I then ran another app (also Firefox) on the system to use up available memory so I could monitor if and how well the memory freed from Firefox was being made available to the system.

I observed that the manner in which memory was released was less than optimal, in that the system began transferring memory to swap, while there was still significant amounts of memory not released to the system from Firefox.  Also, I observed that even after this, there was still a small amount of freed memory that never would get released.

The method I used to demonstrate this was thus:

(All memory monitoring was done using OS X Activity Monitor)

I opened an instance of Firefox with a session of 97 tabs, (call it Firefox 1) and loaded all the tabs.  After settle-down, I removed all tabs but 1.  After settle-down, recorded the amount of memory this instance of Firefox was using.

I then opened another instance/profile of Firefox (call it Firefox 2), which was for the sole purpose of using up available memory to bring the system into a state of running out of memory.  There were no measurements taken from this instance.  This was opened with a session of 376 tabs, which would begin loading on startup.

I then recorded the changes I observed in Activity Monitor for memory usage in Firefox 1, and the concurrent swap memory.  In some cases I observed that over 300 MB had already been transferred to swap, while there still remained over 300 MB of memory that had not yet been released from Firefox 1.

When I observed that memory usage in Firefox 1 would no longer decrease, I forced a memory release by indirectly calling jemalloc_purge_freed_pages() (see Bug 852520) through Javascript methods; in some cases by updating about:memory, in others, by running a command from a demo extension created just for this purpose.  (I felt the demo might lend to a more accurate measurment due to all the code that runs to update about:memory).  In every case the memory usage dropped after this action, on an average of 70 MB.  I felt this provided a figure indicating what the memory usage *should have* ended up at through the intended automatic methods of releasing memory back to the system.

Details of my findings are attached, along with test sessions and demo extension to force memory purge.

If using the demo extension, make sure addon bar is enabled and look for "Trigger Memory Release" menu button.  Click on "nsIMemoryReportManager.resident" to trigger a purge.

Comment 1

[u462496]

Attachment: Session - 97 tabs for Firefox test instance

Comment 2

[u462496]

Attachment: Extension to force memory release

Comment 3

[u462496]

The bigger session for the utility instance was too big to attach:

http://tech.kevinallasso.org/mem_release_session/

Comment 4

[Justin Lebar (not reading bugmail)]

What version of MacOS are you running, and what is your hardware?

Comment 5

[u462496]

OS X 10.7.5

Model Name:             Mac mini
Model Identifier:       Macmini5,2
Processor Name:         Intel Core i5
Processor Speed:        2.5 GHz
Number of Processors:   1
Total Number of Cores:  2
L2 Cache (per Core):    256 KB
L3 Cache:               3 MB
Memory:                 4 GB
Boot ROM Version:       MM51.0077.B10
SMC Version (system):   1.75f0
Serial Number (system): C07GQ3KHDJD1
Hardware UUID:          089E18A8-92DE-5EA4-90BC-5C62BFF40D5C

Comment 6

[Justin Lebar (not reading bugmail)]

Memory allocated by the JS engine is not affected by jemalloc_purge_freed_pages, so this is not a JS bug.

Comment 7

[Nicholas Nethercote [inactive]]

glandium, any wisdom you have to add would be well-received :)

Comment 8

[Eric Rahm [:erahm]]

(In reply to Nicholas Nethercote [:njn] from comment #7)
> glandium, any wisdom you have to add would be well-received :)

I'm inclined to wontfix this, but Mike IIRC you were pondering adding some better heuristics for double-purging on mac. Any thoughts?

Comment 9

[Mike Hommey [:glandium]]

Yeah, I think we need to find some way to force purge those at some opportune moment. Tying this to GC would seem sensible, but  we might want to use an API that can do the double purge a few pages at a time instead of wholesale or something, so as not to penalize GC too much (OTOH, if GC triggers it, maybe there would naturally be less pages to purge).

Related: we're going to have to support double purge on linux too, for bug 1406304.