398333 - is places.sqlite using a cache that is too large ?

Status: RESOLVED WONTFIX

(Firefox :: Bookmarks & History, defect)

Description

[Jo Hermans]

As far as I can see, we're using 6% of memory as the cache for the places.sqlite file (if history_cache_percentage isn't set). For 1GB of physical memory, this means 60MB ! That seems to be an awfully large cache to me.

As a proof, I typed in "pragma cache_size;" in Storage explorer, and it reported a size of 62875 (I have 1GB of RAM, and page_size seems to be 1024).

I know that we're currently using 180 days of history, and we will be seeing places.sqlite files that will be extremely large (dozens of MB). So the cache will really be large, possible even the maximum of 60MB. Also, the preload will actually try to load the full 60MB at startup, which will be very slow.

I understand why a cache needs to be large. If it's (much) smaller than the database, access will be slow ofcourse. That why we don't use the default cache 2000 pages, which results in 2MB with our hardcoded page size of 1024. See also bug 373256. But with those large places files ...

Maybe we need to reconsider the default cache-size. If there are performance problems, then we need to fix them, like in bug 373256 comment 29. Or this is just another reason to limit the default days in history ; other bugs have mentioned that 180 days seem to be excessive.

Comment 1

[Dietrich Ayala (:dietrich)]

now that we have incremental vacuum happening during idle time, the files themselves should be as large, so we should look at this again.

we set the cache size dynamically at startup. possibly we could set the cache at the places.sqlite file size + 10% (if less than 6% of memory) or something like that.

Comment 2

[Jo Hermans]

That would not be the right solution, in most cases it will allocate exactly the same amount of memory as the current solution. The point is that the cache-size is set to 60MB for a 1GB computer, but that space is not really allocated. The real cache-size that is used depends on the actual data that you touch, which can never be larger than the database itself in the worst case. Only when you grow more than 10%, then your solution will be using less. It will even be worse for places.sqlite files that are larger than 60MB. And it might be suboptimal for very small database, like the ones that are created in new profiles.

What I was asking here is to *cap* the cache size to a reasonable value, 12MB or whatever is appropriate. I now that it can be bad is case you need to touch a lot of data (full vacuum, clear history, etc ...), and it doesn't fit in the cache anymore. This might be a problem if we're really going to use 180-days histories (which is too much in my opinion). But if we're doing nothing, then we will be using very large amounts of memory as a side-effect of the large history.

Comment 3

[Olli Pettay [:smaug][[e-mail]]]

Is this something we want to optimize before ff3?

Comment 4

[Stuart Parmenter]

I would be pretty scared of this being much more than 10mb total -- if our startup memory usage starts up at 100mb with 60 of that being sqlite that is bad times.

Comment 5

[Benjamin Smedberg]

Stuart, why? If I have 1+G of RAM I'd expect FF to use it.

Comment 6

[Gervase Markham [:gerv]]

Bug 451915 - move Firefox/Places bugs to Firefox/Bookmarks and History. Remove all bugspam from this move by filtering for the string "places-to-b-and-h".

In Thunderbird 3.0b, you do that as follows:
Tools | Message Filters
Make sure the correct account is selected. Click "New"
Conditions: Body   contains   places-to-b-and-h
Change the action to "Delete Message".
Select "Manually Run" from the dropdown at the top.
Click OK.

Select the filter in the list, make sure "Inbox" is selected at the bottom, and click "Run Now". This should delete all the bugspam. You can then delete the filter.

Gerv

Comment 7

[Lucas Malor (mail: c6kfnkn2uc AT snkmail DOT c0m)]

I add the cache is high even with auto private browsing mode.

Since I use a personal profile at university I don't have much space. 10 mb for 16 bookmarks without favicons in auto private browsing mode is excessive.

Comment 8

[Shawn Wilsher :sdwilsh]

(In reply to comment #0)
> As far as I can see, we're using 6% of memory as the cache for the
> places.sqlite file (if history_cache_percentage isn't set). For 1GB of physical
> memory, this means 60MB ! That seems to be an awfully large cache to me.
We are using up to 6% of memory.  It will only use the number of pages actually used by the database.

> As a proof, I typed in "pragma cache_size;" in Storage explorer, and it
> reported a size of 62875 (I have 1GB of RAM, and page_size seems to be 1024).
That pragma just reports the "suggested maximum number of database disk pages that SQLite will hold in memory at once per open database file".  It is not how much is actually used.

> Maybe we need to reconsider the default cache-size. If there are performance
> problems, then we need to fix them, like in bug 373256 comment 29. Or this is
> just another reason to limit the default days in history ; other bugs have
> mentioned that 180 days seem to be excessive.
The disk cache lets you not have to hit the disk.  The disk I/O is largely unavoidable, and hitting the disk causes performance problems.  The solution in bug 373256 comment 29 is specific to one specific task (that is not longer applicable anyway).  We were still hitting the entire DB in that case anyway, just in smaller chunks.
Additionally, the amount of history that we keep is determined by the amount of memory a user has so we will always fit into the cache (no longer a set day cap).

(In reply to comment #2)
> What I was asking here is to *cap* the cache size to a reasonable value, 12MB
> or whatever is appropriate. I now that it can be bad is case you need to touch
> a lot of data (full vacuum, clear history, etc ...), and it doesn't fit in the
> cache anymore. This might be a problem if we're really going to use 180-days
> histories (which is too much in my opinion). But if we're doing nothing, then
> we will be using very large amounts of memory as a side-effect of the large
> history.
Even if we touch the entire DB, it will still fit into the entire cache given our current expiration policy.

This is WONTFIX per the reasoning in comment 5.

Comment 9

[Jo Hermans]

(In reply to comment #5)
> Stuart, why? If I have 1+G of RAM I'd expect FF to use it.
about:memory reports that I'm currently using 32MB of storage/sqlite/pagecache, which is much lower than my current 6% limit (120MB on a 2GB desktop). So we're not even close to limiting the memory-usage of sqlite. I don't mind, that's what memory is for, but we'll surely receive some reports when Firefox 4.0 is released, and people can see what sqlite is using in the pagecache ...

(In reply to comment #7)
> I add the cache is high even with auto private browsing mode.
> 
> Since I use a personal profile at university I don't have much space. 10 mb for
> 16 bookmarks without favicons in auto private browsing mode is excessive.

This is about the sqlite pagecache (in memory), not the disk cache.

Comment 10

[Marco Bonardo [:mak]]

(In reply to comment #9)
> but we'll surely receive some reports when Firefox 4.0 is
> released, and people can see what sqlite is using in the pagecache ...

This is a psychological problem, if the OS would fake show half of the memory as free, even if it's full, nobody would complain. If we have users with 4-8GB of memory complaining that we are using 100MB to speed up their browsing experience, we need to really start doing some good evangelization on technologies and what they are for.

Comment 11

[Shawn Wilsher :sdwilsh]

(In reply to comment #9)
> This is about the sqlite pagecache (in memory), not the disk cache.
You can replace the phrase "disk cache" with "page cache" in everything I said above.  The page cache prevents you from hitting the disk by caching the data (which is why I sometimes call it a disk cache, even though it is actually called the page cache).

Comment 12

[Jo Hermans]

(In reply to comment #10)
> This is a psychological problem, if the OS would fake show half of the memory
> as free, even if it's full, nobody would complain. If we have users with 4-8GB
> of memory complaining that we are using 100MB to speed up their browsing
> experience, we need to really start doing some good evangelization on
> technologies and what they are for.

I know. At work, people are designing something (can't tell you what) with a full 8TB cache (yes, that's 8 terabyte), and people are still complaining about memory usage :-)