Closed Bug 447710 Opened 16 years ago Closed 15 years ago

jemalloc eats disk space

Categories

(Core :: Memory Allocator, defect)

x86
Linux
defect
Not set
major

Tracking

()

RESOLVED DUPLICATE of bug 465127

People

(Reporter: tuukka.tolvanen, Assigned: jasone)

References

Details

With my default profile (10--100 pages), starting fx (trunk 2008072320 linux) gobbles up 246MB of disk space within a minute of startup (if the disk fills up, it dies). du doesn't say where the space is going. lsof shows 186 lines of

firefox-b 22296 tt   DEL REG  8,2    2243767      /tmp/jemalloc.1PB2xU

and its friends, which either is related or isn't. On a fresh test profile, the usage is 20MB, and 18 /tmp/jemalloc.kLoqVk thingers.
2008-07-01-02-mozilla-central  1MB, no lsof -c firefox-bin | grep /jemalloc
2008-07-02-02-mozilla-central 28MB,  27 items

jasone Tue Jul 01 23:01:12 2008 +0000 446be0510148 Bug 427109: Add memory
reserve and xmalloc() API, r=benjamin This re-commmit adds locking that was
missing in changeset c14ab4f6cec6.

didn't test, but that's in the window and the MALLOC_PAGEFILE comments in the diff sound tasty. 427109 sounds like this may be intentional? but I'm amused to be dying by running out of disk space when I have ram left :)
Blocks: 427109
Huh. I thought that we were only going to mmap the reserve memory, not the main memory... but now that I'm thinking about it again, that would still mean that we could overcommit. Jason, what memory is supposed to be disk-backed?
(In reply to comment #2)
> Huh. I thought that we were only going to mmap the reserve memory, not the main
> memory... but now that I'm thinking about it again, that would still mean that
> we could overcommit. Jason, what memory is supposed to be disk-backed?

As you say, in order to prevent overcommit, *everything* has to be disk-backed.  I am increasingly concerned about the out-of-disk-space failure mode.  The patch in bug #414946 changes the default to overcommit (for all non-Windows platforms), though MALLOC_OPTIONS=o can still be used to disable overcommit.

Tuukka, in the meanwhile you can work around this by setting MALLOC_OPTIONS=O in your environment, or by setting MALLOC_TMPDIR to a directory that has adequate space available.
Assignee: nobody → jasone
bug 448840 mentioned a performance problems with the disk backend
Component: General → jemalloc
Product: Firefox → Core
QA Contact: general → jemalloc
Summary: running fx eats a bunch of disk space → jemalloc eats disk space
Flags: wanted1.9.1?
Flags: blocking1.9.1?
Filling the pagefile with zeros is not the best option to reserve disk space. Why is it done?

How about a fcntl(fs, F_RESVSP... style call instead?
Hmm.. F_RESVSP seems to be IRIX specific, but there's fallocate( on Linux and also , posix_fallocate(fd, offset, length)...
Okay, it seems Solaris has posix_fallocate too, and MacOS has fcntl(fd, F_PREALLOCATE...

FreeBSD seems to lack any way to do this, still these APIs should be used where available...
Would it be acceptable to put these as platform-specific ifdefs in jemalloc.c around line 2430?
I'd like jasone to comment, but yes, platform-specific ifdefs are acceptable, *provided* that these system calls are generally available. For instance, are they in the 2.4 kernel or do they require the 2.6 kernel, or some specific version of the 2.6 kernel? I'm not sure we should drastically change our system requirements drastically for the 3.1 release, so I'd like to get some data about the versions in which the fallocate/posix_fallocate calls are actually avaialable.
fallocate is only available since Linux 2.6.23 and on Solaris it's a relatively recent addition too, not in any official release yet

Plus the Linux implementation is only available for some filesystems. So a runtime FS type check would be needed...

So it doesn't look like such a good idea now to add these calls, and with overcommit being the default it probably doesn't matter.

Unrelated but I also noticed that on OpenSolaris /tmp is memory-backed tmpfs anyway, and I remember some Linux distro also doing this
I'm open to using better APIs, assuming that they 1) are widely available, and 2) do the right thing, hopefully more efficiently.  From comment #11 I gather that this is something worth looking at in the future, but that it may take a couple of years for the APIs to become common enough.
I'm not seeing this problem anymore in the the final 3.1b2.  Was it fixed, or did something on my system just make it go away?
It looks like this was fixed by bug 465127, which was accidentally fixed in bug 464995.
Depends on: 465127
I just started using 3.1bX, actually

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b3pre) Gecko/20081230 Shiretoko/3.1b3pre

built from the trunk. I use a lot of tabs, and the /tmp/jemalloc files ran out the filesystem. I don't understand the comments above about this being fixed.

For now I'm working around it with MALLOC_OPTIONS=O
Shiretoko/3.1b3pre is 191branch, not trunk; the fix was pushed to 191 only yesterday (bug 465127 comment 14)
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → DUPLICATE
Flags: wanted1.9.1?
Flags: blocking1.9.1?
You need to log in before you can comment on or make changes to this bug.