331158 - Autocomplete is slow the first time

Status: VERIFIED FIXED

(Firefox :: Bookmarks & History, defect, P1)

Description

[Brett Wilson]

The new autocomplete implementation in bug 320181 can be slow the first time you use it. This is because the sqlite (the basis for the new history system) pages data in lazily. The old Mork system brought all history into memory at startup. The new system results in faster startup times when history is large, but the speedup isn't free. The data still needs to be loaded.

On fast local disks the delay isn't a big deal. I have a networked home directory on Linux and it can be quite slow. The first time I press a new letter in the URL bar, it freezes until the needed data can be brought over the network. This takes almost no time up to several seconds depending on how many matches there are and the mood of the network.

Perhaps we can set a timer on startup to bring some data into the cache after the window is displayed. This will freeze the browser during that time, however.  We may have to play with different strategies to see where the pause is the least noticable.

Comment 1

[Jo Hermans]

Shouldn't we also have to delete expired records (which aren't loaded in at startup) at startup ? Now we only do it at shutdown.

Rationale : in 1997 I once installed a Netscape browser (version 3), that was displaying some meeting shedule. It was for a very busy European Union governemental conference center, but they had no budget to buy a real announcement display (airport style), especially because none of them had a web-interface at that time. So I hacked up something for them. 4 displays were continuously refreshing their info every minute, but were completely unattended. After a few weeks they got in trouble with memory leaks and such, so I told the operators that they had to restart the browser once in a while (and they were running Windows 95 too, which had its own problems). But they did it with a remote reboot, which means that the browser was never shut down.

The memory leaks were solved, but those browsers were never shut down the nromal way (File->Exit). Since the history-cleanup was only done during a normal startup, this means that the history was growing forever, but never could be cleaned. Netscape 4 fixed this I think.

Anyway, I worked around the problem by cleaning history during startup (the displays were running until 2 years ago !!!), but we might be careful in the current implementation, just in case a user never shuts down cleanly. If we purge old records during startup, a good part of the index will be read in automatically. For most users, there will be almost nothing to clean.

PS : there's an anecdote about Bill Gates and those displays. I'll tell it tonight on IRC.

Comment 2

[Brett Wilson]

Jo: Your request has nothing to do with autocomplete. We may do expiration at runtime for 328598, so see that one.

Comment 3

[Jo Hermans]

(In reply to comment #2)
> Jo: Your request has nothing to do with autocomplete. We may do expiration at
> runtime for 328598, so see that one.
> 

That's only during shutdown. I filed bug 331244 for it.

Comment 4

[Brett Wilson]

Slowness is prohibitive in some cases, bumping priority.

Comment 5

[Doug Halamay]

This bug is hitting me everytime I startup FF.  I have my history set to 90 days...is that a possible cause of this?

Comment 6

[James Ross]

Could well be, my history is set to 365 days (!) and the delays for some characters (the delay seems to depend on how much matches the 1 character you've typed) are easily 30s (this is a local setup, btw).

It's also not just the first time either, it seems to "expire" from a search cache at some point or something, and the same letter is again prohibitively slow when requested some time later. I assume this is related to sqlite's paging of data.

Comment 7

[Brett Wilson]

I am curious: for you people with 90 or 365 days of history, how big is your bookmarks_history.sqlite file, and how much memory do you have?

The current cache size is 10MB. If you have a lot more than that you will get paging.

I have a patch that will help the common cases of a 10-20MB .sqlite file. I wonder if/what we need to do something for you edge cases.

Comment 8

[Doug Halamay]

My bookmarks_history.sqlite is 8.3MB, with 90 days of history, and I have 1GB of RAM.

(In reply to comment #7)
> I am curious: for you people with 90 or 365 days of history, how big is your
> bookmarks_history.sqlite file, and how much memory do you have?
> 
> The current cache size is 10MB. If you have a lot more than that you will get
> paging.
> 
> I have a patch that will help the common cases of a 10-20MB .sqlite file. I
> wonder if/what we need to do something for you edge cases.
> 

Comment 9

[James Ross]

I have 365 days of history, and the following profile files relating to SQLite:
  history.dat              of 33,770,675 bytes
  bookmarks.html           of     49,434 bytes
  bookmarks_history.sqlite of 71,377,920 bytes

  formhistory.dat          of    674,958 bytes
  formhistory.sqlite       of    443,392 bytes

I'm curious why formhistory shrank so much when history/bookmarks bloated itself so much, but that's not relevant here...

My system had 1GB RAM until a few days ago, when I stuck another 2GB in there, making it probably very much an edge-case in all respects. :)

Comment 10

[Brett Wilson]

James: Thanks for your statistics. The amount of cache we will use depends on available memory. My rule of thumb is that a typical user's browsing will be about 10MB, which we should load completely into memory on a 256MB machine. If we use the same maximum fraction of memory, you will actually be about the same, so it should be OK when my patch goes in.

The new history file is larger partially because we store more data (favicons and multiple visits to the same page), and because of indices. The old system would compute the indices on startup, meaning that history would get larger in memory. This doesn't happen to us, 10MB on disk is 10MB in memory. We also have more indices so many operations can be faster. Without indices, sqlite is much more space efficient than Mork.

Comment 11

[Brett Wilson]

Attachment: Patch

This patch has 3 components:
1. Modification to sqlite to provied a preload capability. This loads the first N pages of the database file into the cache, where N is the maximum size of the memory cach (or the size of the database if smaller).

2. Add a function to mozIStorageConnection to expose this capability.

3. Modification of the history service to use the new function. The maximum cache size is now computed as being 6% of machine memory rather than a hardcoded 10MB. I also upped the default page size to 4K for new databases since seeks look like they're the limiting factor when paging.

Comment 12

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Comment on attachment 217513 [details] [diff] [review]
Patch

r=me; I'd prefer to see the sqlite bits go upstream as well, so that we don't have to maintain a patch.  It seems like a useful addition to the sqlite API anyway, so hopefully drh will take it.

(Sorry it took so long to get to this!)

Comment 13

[Brett Wilson]

On 1.8 branch and trunk.

Comment 14

[Tracy Walker [:tracy]]

Verified with nightly builds from 01-07-08

Comment 15

[Gervase Markham [:gerv]]

Bug 451915 - move Firefox/Places bugs to Firefox/Bookmarks and History. Remove all bugspam from this move by filtering for the string "places-to-b-and-h".

In Thunderbird 3.0b, you do that as follows:
Tools | Message Filters
Make sure the correct account is selected. Click "New"
Conditions: Body   contains   places-to-b-and-h
Change the action to "Delete Message".
Select "Manually Run" from the dropdown at the top.
Click OK.

Select the filter in the list, make sure "Inbox" is selected at the bottom, and click "Run Now". This should delete all the bugspam. You can then delete the filter.

Gerv