431739 - Limit dirty page purging

Status: RESOLVED FIXED

(Core :: Memory Allocator, defect)

Description

[Jason Evans]

Due to a mis-merge from the original source, jemalloc in Mozilla currently purges *all* unused dirty pages when the limit is reached, rather than terminating purging when the dirty page count drops to half the limit.

Algorithmically, the dirty page purging can cause allocation performance to drop exponentially as memory usage increases.  The constant factors are small, but this effect can become an issue if the application uses hundreds of megabytes of memory.

Comment 1

[Jason Evans]

Attachment: Limit dirty page purging.

Limit dirty page purging so that it terminates once the number of pages is no
more than half the limit.

Add a separate tree to track arena chunks that contain dirty pages.  This
substantially improves worst case allocation performance, since O(lg n) tree
search can be used instead of O(n) tree iteration.

Comment 2

[Jason Evans]

I am still conducting experiments to determine a reasonable threshold for dirty page purging.  The current patch restores the default, which is 512 pages (~2 chunks).

Comment 3

[Jason Evans]

Attachment: Limit dirty page purging (v2)

Do not disable debug code that uses red-black tree iterators, now that the patch in bug #431735 supports iteration on Windows.

Comment 4

[Jason Evans]

Attachment: tp RSS with various purge thresholds

The plot shows RSS for the tp benchmark (from StandAloneTalos 1.1), using five different purge threshold settings.  The plot shows that lower thresholds do decrease memory usage, but that the differences are incremental.

I continue to have a difficult time getting reliable numbers from the sunspider benchmark, which claims impossibly low variance in its results, despite high actual variance.  Judging from various benchmarks I have run, the speed difference for these five thresholds is lost in the noise.

My inclination is to drop the default purge threshold from 2048 pages to 1024 pages.  This is comparable to what we did before this bugfix, since we purged all dirty pages, whereas the fixed code purges half the dirty pages twice as often.

Comment 5

[Jason Evans]

Attachment: Limit dirty page purging (v3)

This patch revision sets the default purging threshold to 1024 pages.

Comment 6

[Jason Evans]

changeset:   15461:7983bd8cb164
user:        Jason Evans <[e-mail]>
date:        Fri Jun 20 10:34:42 2008 -0700
summary:     Bug 431739: Limit dirty page purging, r=benjamin