Last Comment Bug 710501 - [meta] Handle memory pressure events more consistently and reliably
: [meta] Handle memory pressure events more consistently and reliably
Status: NEW
:
Product: Core
Classification: Components
Component: General (show other bugs)
: unspecified
: x86_64 Linux
: -- normal with 1 vote (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
:
Mentors:
Depends on: 925343 708075 718100
Blocks:
  Show dependency treegraph
 
Reported: 2011-12-13 18:39 PST by Nicholas Nethercote [:njn]
Modified: 2013-10-10 07:32 PDT (History)
11 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments

Description Nicholas Nethercote [:njn] 2011-12-13 18:39:56 PST
I started an email thread on dev-platform a while back about handling memory pressure events more consistently and reliably:
https://groups.google.com/d/topic/mozilla.dev.platform/RKDv7IozaGc/discussion		 	 		

The most useful messages in the thread were these:

https://groups.google.com/d/msg/mozilla.dev.platform/RKDv7IozaGc/jutUdRyl4lQJ
https://groups.google.com/d/msg/mozilla.dev.platform/RKDv7IozaGc/lRuQzDa8tKsJ
https://groups.google.com/d/msg/mozilla.dev.platform/RKDv7IozaGc/eOFV8FhBd90J

the rest were about low-level technical details.

When a memory pressure event is triggered, sometimes it's not that urgent -- it'd be nice to release memory soon.  But sometimes it is urgent, and we can't afford to allocate any memory while freeing other memory.  (For example, running GC and CC requires allocating some memory, possible several MB, and so isn't suitable for the urgent case.)  We should probably distinguish these two cases.
Comment 1 Justin Lebar (not reading bugmail) 2011-12-13 22:21:41 PST
I wonder whether, rather than having separate memory-pressure events, we should have one event and then check how much free space we have.
Comment 2 Justin Lebar (not reading bugmail) 2011-12-13 22:23:56 PST
On platforms where we have swap, this is just a virtual memory space check.  On platforms where we don't have swap, this is just an RSS check.
Comment 3 Justin Lebar (not reading bugmail) 2011-12-13 22:24:51 PST
> On platforms where we don't have swap, this is just an RSS check.

Or rather, it's a matter of calling the equivalent of $ free.
Comment 4 Emanuel Hoogeveen [:ehoogeveen] 2011-12-14 22:51:13 PST
(In reply to Justin Lebar [:jlebar] from comment #2)
> On platforms where we have swap, this is just a virtual memory space check. 
Well, min([available RAM + swap], [available address space]) to handle systems with either low RAM and swap or simply running 64-bit Firefox, I assume.
Comment 5 Justin Lebar (not reading bugmail) 2011-12-15 06:32:00 PST
You're right, although this is a bit complicated.  Can't Windows dynamically expand the swap file?  There's a limit to how big it can get, certainly...
Comment 6 Emanuel Hoogeveen [:ehoogeveen] 2011-12-15 11:37:10 PST
Yes, the limit on Windows 2000/XP/2003 for one paging file appears to be 4095 MiB ([1]), though it is possible (through the registry) to assign several separate paging files. On Windows 7 the limit is considerably higher, though I haven't been able to find what it is: my paging file is 64 GiB for the occasional long rendering task. Unfortunately I haven't found any ways to query the maximum paging file size programmatically.

If the user has customized his or her paging file, the default/minimum and maximum sizes can be found in the PagingFiles REG_MULTI_SZ value in HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\SessionManager\MemoryManagement - where the first column is the paging file location, the second column is the default/minimum size and the third column is the maximum size (there can be multiple rows for multiple paging files). If the paging file is set to a system managed size, the second and third columns will be set to 0 (unfortunately).

The main reason I brought it up is that the default paging file size is set to 1.5 times the system RAM or 2 GiB, whichever is smaller (at least on Windows 2000/XP/2003), which could be pretty small for some users. However I'm not sure how it sets the default maximum paging size - that might simply be the full 4 GiB, assuming enough hard drive space is left.

Alternatively, I suppose it could be argued that any paging is a bad thing, so it would make more sense to fire a memory pressure event as soon as available physical RAM reaches a critical level. That would avoid this whole mess, but wouldn't cover the strict out of memory case.
Comment 7 Emanuel Hoogeveen [:ehoogeveen] 2011-12-15 11:38:07 PST
Sorry, I forgot to link to [1] in my post above:

[1] http://support.microsoft.com/kb/237740
Comment 8 Justin Lebar (not reading bugmail) 2011-12-15 11:44:22 PST
Thanks for that investigation, Emanuel.

I am, in general, wary of the idea that we should aggressively drop memory when the system starts running out of available physical memory -- it's not necessarily Firefox at fault!  A user could be on a 16gb system with Firefox using 200mb of RAM; if the system starts to page, should Firefox really take drastic action?

But preventing OOMs by monitoring available virtual address space is an easy place to start.
Comment 9 Nicholas Nethercote [:njn] 2011-12-20 19:58:26 PST
Oh geez:

    +pref("memory.low_virtual_memory_threshold_mb", 128);
    +pref("memory.low_physical_mem_threshold_mb", 0);
    +pref("memory.low_physical_memory_notification_interval_ms", 10000);

Inconsistent use of "memory" vs. "mem"?  jlebar, can you please do a follow-up fix to make them all use "memory"?  Thanks.
Comment 10 Nicholas Nethercote [:njn] 2011-12-20 19:59:09 PST
Whoops, comment 9 was supposed to go into bug 670967.
Comment 11 Justin Lebar (not reading bugmail) 2012-01-17 15:05:10 PST
We talked about this bug in memshrink a bit today.

It looks like, unless we can reduce JS usage, we're not going to be able to make a big dent here.  And JS usage is hard to reduce without effectively unloading tabs, either by session restoring them, or by somehow serializing the compartment to disk.
Comment 12 Nicholas Nethercote [:njn] 2012-01-17 17:15:56 PST
JS is big, but it isn't everything.  In my current session layout is 15%, and when the style sheet reporter lands (bug 671299) that'll probably jump to over 20%.

And dropping bfcache will allow some JS compartments to be freed.  So I think we can make reasonable progress without having to do really big changes.
Comment 13 Justin Lebar (not reading bugmail) 2012-01-17 17:28:41 PST
We already drop bfcache on a timer and on memory pressure.  This bug would unify the mechanism with the other things we drop, which is virtuous, but that doesn't change the amount of memory which is eventually freed.

I guess dropping 30% of our memory (layout plus sqlite) is good.  To some degree, the question is: What's our goal?
Comment 14 Nicholas Nethercote [:njn] 2012-01-17 17:37:24 PST
My first goal is to drop stuff that's easy to drop, and see how far that takes us.  Doing that will require the unification of mechanisms, as you mentioned, which is a good thing.

Note You need to log in before you can comment on or make changes to this bug.