206109 - preference footprint issues

Status: RESOLVED INCOMPLETE

(Core :: Preferences: Backend, defect)

Description

[James Nord]

There is no documentation on how much memory a pref takes up.

[Note this is a placeholder for information]

from *very* simple testing it seems to be as follows,

stripped prefs
MEM USAGE 21,352K	VM 16,044K

==============================

extra prefs:
user_pref("james.test.pref.1", 1);
...
user_pref("james.test.pref.99", 1);

MEM USAGE 21,484K	VM 16,132K

diff = MEM = 132K Vm 88K = 220K
per pref = 2,2K /pref

==============================

extra prefs:
user_pref("james.test.pref.1", "This is a longer string");
...
user_pref("james.test.pref.99", "This is a longer string");

MEM USAGE 21,536K	VM 16,076K

diff = MEM = 184K Vm 100K = 216K
per pref = 2,2K /pref

Comment 1

[James Nord]

sorry, messed up assigning it to myself

Comment 2

[James Nord]

Adding CC dmose, sspitzer per request IRC request from dmose

Comment 3

[Dan Mosedale (:dmosedale, :dmose)]

If it's really this much, we need to hack it way down, not just document it.

Comment 4

[Christopher Aillon (sabbatical, not receiving bugmail)]

Does profile sharing affect these numbers at all?

Comment 5

[[e-mail]]

profilesharing won't affect the numbers on a per-pref basis. it doesn't add any
pref storage overhead, only transaction communication overhead.

Comment 6

[Alec Flett]

prefs is actually incredibly efficient (IMO) - PLDHash reduces the excess
hashtable allocations, the pref names are stored in an arena, and the string
pref values are malloc'ed.

While I agree that prefs needs more documentation, I think the evidence you are
giving is inconclusive. There are many factors at work here, and I don't think
adding 2 prefs really increased your VM by 88k

furthermore, you don't add VM size and Mem size - "MEM" is going to be a subset
of VM, and so I don't get how your MEM usage is actually higher than VM.

I don't see the bug here to be honest.

Comment 7

[James Nord]

alec, read again...
I added 99 prefs, each pref added *approx* 2k

yes the tests where simple, yes I will be doing more accurate testing when ;-)

The bug spawned from a discussion in another bug which I wrote a patch for and
Dan Mosedale reviewed.  In that I introduced a hidden pref that defaulted to "".
 Dan was arguing that I shouldn't have a default but should catch the error that
the pref doesn't exist for the default case.  But we had no evidence how much
memory a pref adds to mozilla. 

Comment 8

[Alec Flett]

ah! sorry.

in any case, you wrote "per pref = 2.2k/pref" and measuring total VM based on
adding data to one module is still an incredibly inaccurate measure of this
stuff.  For example, say I start up and I allocate 10,000 4k pages from the VM -
as a part of this the heap uses 1,000 4k pages, but on the last page it only
actually uses 3k of the 4k in the page.
Now, if I allocate 1.1k of data, the VM will jump by 4k, because you've just
required one more page. If I allocate 6k of data, the VM jumps by 8k, and so
forth. And this is working on the assumption that the heap allocates the
absolute minimum from the VM. Perhaps the heap allocates chunks of VM in 16k
increments, or perhaps it is really efficient and finds gaps in the available
heap so that allocating 6k of data fits nicely into the current heap, and no VM
is allocated.

Here are more details on how pref allocation works:
The hashtable structure for prefs is built such that it initially holds about
1024 PrefTableEntries, each of which are about 16 bytes. PLDHashtable then
doubles each time the pref table fills up. So if there are 2000 prefs, the pref
table holds space for 2048 prefs.. and when I allocate 49 more prefs, the table
doubles to hold 4096 prefs.

As for pref names, the arena starts out with 8192 bytes, from which we allocate
each pref name. When that chunk fills up, it allocates another chunk of 8192
bytes, and so forth. The allocations in these 8k chunks are 3-bit aligned, so
that a 9 character string actually takes up 16 bytes. However, as we are using
arenas, there is no malloc overhead (which is traditionally another 8 bytes)

As you can see, it is incredibly hard (and it would be wrong) to calculate the
"per-pref" allocation cost - it depends on how many prefs are already in the
table, how much space the existing pref names take up, and so forth. In fact,
adding a new pref very often costs exactly 0 bytes - the pref name goes into an
arena which has already been allocated, and the pref hash entry goes into the
hashtable which more than likely has more space.