276342 - ideas to counter memory fragmentation

Status: RESOLVED INCOMPLETE

(Core :: General, enhancement)

Description

[Jo Hermans]

A lot of bugs are being filed about the memory usage of Firefox. I'm not going
to repeat all those complaints, most of which don't really understand that you
can't really shrink your heap in most cases. The misleading display of the
Windows Program Manager (minizing seems to shrink, but doesn't really do that)
just makes it worse.

What I'm wondering is this : shouldn't we try to reduce the memory fragmentation
?  I was more or less successful with this in my daytime job, which suffered the
same problem. It's about an application that consumes hundreds of megabytes of
data during peak hours, but can drop to a few dozen of MB's during idle time.

What I did was to flush as much as memory as I could (clean out all caches,
empty pools and arena's, ...). This would merge many different memory fragments
back together. Then I would reload the configuration data (using those same
blocks), and free the old configuration data (which would be merged too). The
result was that often huge blocks of memory on the end of the heap were being
freed, and memory was being given back to the OS. Yes, it's really possible to
do that, even under UNIX'es (it works under Tru64 and Solaris, possibly becuase
their malloc seem to prefer to return blocks in the start of the heap, not the
end). The amount of memory given back is ofcourse never the total empty heap,
but I often saw reductions from 500MB to 100 MB or so. The trick was first to
free as much as possible, and then to 'replace' your curent memory (the blocks
you really can't free) with fresh ones. I understand that this technique isn't
ususable for every application, but it's a start.

ideas :

- When a windows is closed, we try to clean up some memory by forcing the
GC-collector (for JS) and a few other things. Can we do the same things when
closing tabs ? Maybe not for every tab that is being closed, but at most once
per minute or so. Or when 'close other tabs' is used.

- Can't we use the memory-pressure observers for this ? I know that they exist
in Minimo, but I wonder if they ever get called in Firefox. We might also call
them if a certain memory-level is reached (100MB for example).

- How about the various pools and arena's in Firefox. Are they all being cleaned
by the memory-pressure observers ?

Comment 1

[Jo Hermans]

I found out that the memory observers are called from nsMemoryImpl.cpp, when
IsLowMemory() returns true. But IsLowMemory is only defined for Windooze and
Mac. So other platforms aren't using this code at all (there's not even a
memeory flusher thread).

I know why we can't implement IsLowMemory, but I don't think that's the right
solution, even on Windows and Mac. On Windows it would fire when if less than
10% free memory was available, on Mac when less than 256K was available or less
than 2MB temp-memory. The first one might be reasonable (if an average pagefile
is 500MB, then it's 50MB), but the Mac-version is still for Mac OS 9 and
earlier, where only RAM memory was counted. Mac OS X now uses virtual memory
(1GB in my case), so IsLowMemory() will never return true. And temp-memory
doesn't exist anymore. I can't remember seeing that warning since I've switched
to X.

Since all OS'es now use virtual memory or pagefiles, I don't think we can use
this kind of heuristic anymore, except in case of an emergency. Maybe we should
just flush some memory when a certain memory limit has been reached ; for
example 100MB, and every extra 50MB after that. That's the way I implemented it
in my apps. Or make sure it's called once per hour.

Note that we still have the problem with UNIX-like malloc-libraries, that don't
provide any way to count the memory allocated. I'm using a wrapper library for
this purpose, so I can check memory leaks down to a single byte, but I don't
think it's possible to use this in Gecko, unless we're sure that everyone calls
NS_Alloc, NS_Realloc and NS_Free (new_handlers, bleh)

Comment 2

[Jo Hermans]

Note that nsCacheService.cpp or nsMemoryCacheDevice.cpp don't implement a
memory-pressure observer at all.

Comment 3

[Jo Hermans]

I found a copy of dlmalloc in /xpcom/build/malloc.c. Are we using that to
override the malloc-implementation of the OS ?

It also contains the code for malloc_trim, but I can't find any call to it,
either our copy or the one from the OS. Maybe we can try to call it in the
memory flusher ?

Malloc_trim() was the reason why I could combat memory fragmentation in my
application (see comment 0), although in that case, it was called automatically
by free() itself (Tru64Unix). On other OS'ses, you might have to do that manually.

Comment 4

[Jo Hermans]

I think I'm closing this bug, since there hasn't been any reaction.

Comment 0 described a situation in my daytime-job, where the OS (Tru64Unix) was reclaiming memory in a particular way, which might not be possible here. We're now using Linux (although a specialised version with a different malloc-implementation), where we can have this effect by calling malloc_trim() directly (it's not called in free). I think (but I'm not sure) that regular Linux uses dlmalloc, which might implement malloc_trim() too. But I don't know if it's called in free().

Comment 5

[Jo Hermans]

closing my own bug

Comment 6

[Mike Schroepfer]

Re-opening this bug (instead of filing a new one) since we've been doing some recent investigation of memory fragmentation.  On windows it might be worth trying the low fragmentation heap:

http://msdn2.microsoft.com/en-us/library/aa366750.aspx
http://xania.org/200512/crt-heap-fragmentation-in-windows

To see if that helps us.  Pav's also been doing some investigation on paging out/in to see if that helps.

Comment 7

[Dan Mosedale (:dmosedale, :dmose)]

W.r.t. the implementability of IsLowMemory, do the various OSes not provide APIs to get the amount of free physical (not virtual) memory?

Comment 8

[Gummi]

On linux, /proc/vmstat and /proc/meminfo give some memory info.

But I'd like an about:config pref on memory usage to be observed, pressure or no pressure.

Comment 9

[Peter Weilbacher]

On OS/2 we certainly also have APIs to query memory usage, I am not sure why IsLowMemory was never implemented. Doing it as well as Windows should only need a few lines of code. At least the IsLowMemory portion, not sure if there other code that needs to be added elsewhere.
Intesting to see that the old Mac implementation removed but a new one for MacOSX was never added. I guess just not enough people who work on the platform specific code know about this function...

Comment 10

[timeless]

dmose: the windows api is fairly flexible:
http://msdn.microsoft.com/en-us/library/aa366772(VS.85).aspx

the maemo api is simplistic [ok | low on memory | very low on memory]

on an n810, grep MemFree /proc/meminfo yields a number like:
12312

i assume vmstat was /proc/stat (at least it's the only match that man vmstat mentions in FILES) - although it isn't a very useful file...

however, knowing how much memory is available isn't particularly related to memory fragmentation - so i think all of this belongs in some other bug.

Comment 11

[Wayne Mery (:wsmwk)]

Is this bug superceeded by current "memory fragmentation" efforts, and no longer needed?

Comment 12

[Benjamin Smedberg]

This bug is not useful. A lot of the work Stuart did and jemalloc deals with heap-level fragmentation, and we're still looking at VM-level fragmentation but this bug won't help track it.