677954 - Limit the cache size used by places on Thunderbird [memory]

Status: RESOLVED FIXED

(MailNews Core :: Database, defect)

Description

[Jonathan Protzenko [:protz]]

Attachment: Patch v1

Turns out the default pref (6% of the physical memory) will allow places to use up to 245M on my 4GB machine, and almost 1G on asuth's 16GB machine. Although sqlite will be smart enough not to allocate all the memory at once (which explains why we haven't seen over-the-roof memory usage patterns), I believe this is pretty useless and should be set to a more reasonable value.

This patch suggests a value of 1% which sounds like pretty high to me already, given how little places is used inside Thunderbird.

Comment 1

[:aceman]

How is it used in Thunderbird? Does it store browsing history if you open any web URLs in tabs? Is that history accessible in the UI?

Comment 2

[Wayne Mery (:wsmwk)]

(In reply to aceman from comment #1)
> How is it used in Thunderbird? Does it store browsing history if you open
> any web URLs in tabs? Is that history accessible in the UI?

sure it keeps history. but without https://addons.mozilla.org/en-US/thunderbird/addon/thunderbrowse/ you don't really have UI

Comment 3

[Wayne Mery (:wsmwk)]

is this accurate about the current impact ... (as noted in Bug 677805) given that places database is likely to be small (i.e. not much history, etc), the current setting is highly unlikely to account for unexplained high memory in the average user's environment?

Comment 4

[Andrew Sutherland [:asuth] (he/him)]

(In reply to Wayne Mery (:wsmwk) from comment #3)
> is this accurate about the current impact ... (as noted in Bug 677805) given
> that places database is likely to be small (i.e. not much history, etc), the
> current setting is highly unlikely to account for unexplained high memory in
> the average user's environment?

Yeah.

If you set your Thunderbird 'start page' to "about:memory?verbose" (if you want to see everything) or just "about:memory" (if you just want the big things), you can see how much memory is generally allocated to SQLite.  You'll want to be careful about suggesting use of that, however, since there are some crashers related to SQLite memory reporters for clients that don't close their database connections properly through at least Thunderbird 8.

Comment 5

[Andrew Sutherland [:asuth] (he/him)]

(or use a means of opening a content tab to get there... that should work even better...)

Comment 6

[Jonathan Protzenko [:protz]]

Especially since Thunderbird Conversations has failed to close its database connections properly until yesterday (ahem).

Comment 7

[Mark Banner (:standard8)]

(In reply to aceman from comment #1)
> How is it used in Thunderbird? Does it store browsing history if you open
> any web URLs in tabs? Is that history accessible in the UI?

We use some of the interfaces, at most, we could be storing favicon information. However, places is disabled by default, so it shouldn't be storing history.

(In reply to Jonathan Protzenko [:protz] from comment #0)
> Turns out the default pref (6% of the physical memory) will allow places to
> use up to 245M on my 4GB machine, and almost 1G on asuth's 16GB machine.
> Although sqlite will be smart enough not to allocate all the memory at once
> (which explains why we haven't seen over-the-roof memory usage patterns), I
> believe this is pretty useless and should be set to a more reasonable value.

It is slightly surprising that places is allocating memory and doing stuff with the db when the places history is disabled.

Comment 8

[Jonathan Protzenko [:protz]]

Well I haven't checked that the codepath is actually disabled when places history is disabled...

Comment 9

[Andrew Sutherland [:asuth] (he/him)]

Jonathan's and my comments were pretty much based on inspection only, but about:memory?verbose for me shows the following under Thunderbird 6 beta, implying the databases are indeed established.  Maybe they just can't grow because they don't add extra stuff?

│      ├─────373,576 B (00.23%) -- places.sqlite
│      │     ├──298,520 B (00.19%) -- cache-used
│      │     ├───39,992 B (00.03%) -- schema-used
│      │     └───35,064 B (00.02%) -- stmt-used
│      ├─────118,760 B (00.07%) -- urlclassifier3.sqlite
│      │     ├──100,944 B (00.06%) -- stmt-used
│      │     ├───13,576 B (00.01%) -- cache-used
│      │     └────4,240 B (00.00%) -- schema-used
│      ├──────46,744 B (00.03%) -- signons.sqlite
│      │      ├──23,928 B (00.02%) -- stmt-used
│      │      ├──18,336 B (00.01%) -- cache-used
│      │      └───4,480 B (00.00%) -- schema-used
│      ├──────21,256 B (00.01%) -- cookies.sqlite
│      │      ├──18,336 B (00.01%) -- cache-used
│      │      ├───2,920 B (00.00%) -- schema-used
│      │      └───────0 B (00.00%) -- stmt-used
│      └──────13,184 B (00.01%) -- permissions.sqlite
│             ├───6,776 B (00.00%) -- stmt-used
│             ├───4,352 B (00.00%) -- cache-used
│             └───2,056 B (00.00%) -- schema-used

If I use sqlite externally do dump the contents of places.sqlite, it's not clear why it would even allocate that much to cache at all unless that's the smallest allocation block size it uses for the cache given the very high limit on it.

Comment 10

[Mark Banner (:standard8)]

Comment on attachment 552115 [details] [diff] [review]
Patch v1

So I couldn't actually notice any significant difference in what places allocates regardless of the pref setting.

However, I think it is reasonable, just in case, to limit the size of the cache, extensions can always turn it back up again if they are using it.

You might want to add an explicit note as to why we're using a different value to FF though.

Comment 11

[Jonathan Protzenko [:protz]]

http://hg.mozilla.org/comm-central/rev/501c3f855732

Comment 12

[Jonathan Protzenko [:protz]]

We should just revert that change, it's useless now bug 674210 has landed.