Closed Bug 276342 Opened 18 years ago Closed 8 years ago
ideas to counter memory fragmentation
A lot of bugs are being filed about the memory usage of Firefox. I'm not going to repeat all those complaints, most of which don't really understand that you can't really shrink your heap in most cases. The misleading display of the Windows Program Manager (minizing seems to shrink, but doesn't really do that) just makes it worse. What I'm wondering is this : shouldn't we try to reduce the memory fragmentation ? I was more or less successful with this in my daytime job, which suffered the same problem. It's about an application that consumes hundreds of megabytes of data during peak hours, but can drop to a few dozen of MB's during idle time. What I did was to flush as much as memory as I could (clean out all caches, empty pools and arena's, ...). This would merge many different memory fragments back together. Then I would reload the configuration data (using those same blocks), and free the old configuration data (which would be merged too). The result was that often huge blocks of memory on the end of the heap were being freed, and memory was being given back to the OS. Yes, it's really possible to do that, even under UNIX'es (it works under Tru64 and Solaris, possibly becuase their malloc seem to prefer to return blocks in the start of the heap, not the end). The amount of memory given back is ofcourse never the total empty heap, but I often saw reductions from 500MB to 100 MB or so. The trick was first to free as much as possible, and then to 'replace' your curent memory (the blocks you really can't free) with fresh ones. I understand that this technique isn't ususable for every application, but it's a start. ideas : - When a windows is closed, we try to clean up some memory by forcing the GC-collector (for JS) and a few other things. Can we do the same things when closing tabs ? Maybe not for every tab that is being closed, but at most once per minute or so. Or when 'close other tabs' is used. - Can't we use the memory-pressure observers for this ? I know that they exist in Minimo, but I wonder if they ever get called in Firefox. We might also call them if a certain memory-level is reached (100MB for example). - How about the various pools and arena's in Firefox. Are they all being cleaned by the memory-pressure observers ?
I found out that the memory observers are called from nsMemoryImpl.cpp, when IsLowMemory() returns true. But IsLowMemory is only defined for Windooze and Mac. So other platforms aren't using this code at all (there's not even a memeory flusher thread). I know why we can't implement IsLowMemory, but I don't think that's the right solution, even on Windows and Mac. On Windows it would fire when if less than 10% free memory was available, on Mac when less than 256K was available or less than 2MB temp-memory. The first one might be reasonable (if an average pagefile is 500MB, then it's 50MB), but the Mac-version is still for Mac OS 9 and earlier, where only RAM memory was counted. Mac OS X now uses virtual memory (1GB in my case), so IsLowMemory() will never return true. And temp-memory doesn't exist anymore. I can't remember seeing that warning since I've switched to X. Since all OS'es now use virtual memory or pagefiles, I don't think we can use this kind of heuristic anymore, except in case of an emergency. Maybe we should just flush some memory when a certain memory limit has been reached ; for example 100MB, and every extra 50MB after that. That's the way I implemented it in my apps. Or make sure it's called once per hour. Note that we still have the problem with UNIX-like malloc-libraries, that don't provide any way to count the memory allocated. I'm using a wrapper library for this purpose, so I can check memory leaks down to a single byte, but I don't think it's possible to use this in Gecko, unless we're sure that everyone calls NS_Alloc, NS_Realloc and NS_Free (new_handlers, bleh)
Note that nsCacheService.cpp or nsMemoryCacheDevice.cpp don't implement a memory-pressure observer at all.
I found a copy of dlmalloc in /xpcom/build/malloc.c. Are we using that to override the malloc-implementation of the OS ? It also contains the code for malloc_trim, but I can't find any call to it, either our copy or the one from the OS. Maybe we can try to call it in the memory flusher ? Malloc_trim() was the reason why I could combat memory fragmentation in my application (see comment 0), although in that case, it was called automatically by free() itself (Tru64Unix). On other OS'ses, you might have to do that manually.
I think I'm closing this bug, since there hasn't been any reaction. Comment 0 described a situation in my daytime-job, where the OS (Tru64Unix) was reclaiming memory in a particular way, which might not be possible here. We're now using Linux (although a specialised version with a different malloc-implementation), where we can have this effect by calling malloc_trim() directly (it's not called in free). I think (but I'm not sure) that regular Linux uses dlmalloc, which might implement malloc_trim() too. But I don't know if it's called in free().
closing my own bug
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → WONTFIX
Re-opening this bug (instead of filing a new one) since we've been doing some recent investigation of memory fragmentation. On windows it might be worth trying the low fragmentation heap: http://msdn2.microsoft.com/en-us/library/aa366750.aspx http://xania.org/200512/crt-heap-fragmentation-in-windows To see if that helps us. Pav's also been doing some investigation on paging out/in to see if that helps.
Status: RESOLVED → REOPENED
Resolution: WONTFIX → ---
W.r.t. the implementability of IsLowMemory, do the various OSes not provide APIs to get the amount of free physical (not virtual) memory?
On linux, /proc/vmstat and /proc/meminfo give some memory info. But I'd like an about:config pref on memory usage to be observed, pressure or no pressure.
On OS/2 we certainly also have APIs to query memory usage, I am not sure why IsLowMemory was never implemented. Doing it as well as Windows should only need a few lines of code. At least the IsLowMemory portion, not sure if there other code that needs to be added elsewhere. Intesting to see that the old Mac implementation removed but a new one for MacOSX was never added. I guess just not enough people who work on the platform specific code know about this function...
dmose: the windows api is fairly flexible: http://msdn.microsoft.com/en-us/library/aa366772(VS.85).aspx the maemo api is simplistic [ok | low on memory | very low on memory] on an n810, grep MemFree /proc/meminfo yields a number like: 12312 i assume vmstat was /proc/stat (at least it's the only match that man vmstat mentions in FILES) - although it isn't a very useful file... however, knowing how much memory is available isn't particularly related to memory fragmentation - so i think all of this belongs in some other bug.
Product: Firefox → Core
QA Contact: general → general
Version: unspecified → Trunk
Is this bug superceeded by current "memory fragmentation" efforts, and no longer needed?
This bug is not useful. A lot of the work Stuart did and jemalloc deals with heap-level fragmentation, and we're still looking at VM-level fragmentation but this bug won't help track it.
Status: REOPENED → RESOLVED
Closed: 15 years ago → 8 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.