193236 - History Window Excruciatingly Slow

Status: RESOLVED INVALID

(Core Graveyard :: History: Global, defect)

Description

[Bill McGonigle (not currently reading bugmail; please contact directly)]

1.3b, OS X 10.2.3, G4/400, 1GB RAM

Opening the history window can take 30 seconds sometimes, and never less than
15.  Beachball spins frequently.

While it's open the whole browser drags ass.  Closing History restores snappiness.

I saw bug 91044, but that seems to be mostly scrolling and this problem doesn't
show under linux (my other platform).

Comment 1

[Andrew Hagen]

This should block 1.4a, as we need to enhance performance.

Reporter, do you see this with a lot of entries in your history.dat? When you
see this problem, roughly how big is your history.dat file? 

Comment 2

[Bill McGonigle (not currently reading bugmail; please contact directly)]

I'm still seeing this on the 1.3 candidate build.

Right now my history.dat is 3.2M.  I have it set for 60 days.

Comment 3

[Tom Metro]

I've noticed this as well with Netscape 7.01 on Win32. My history.dat was
probably in excess of 5 MB when I first noticed the problem. (It's currently
about 11 MB.) I haven't yet confirmed that the problem still exists in 7.1 or in
other products (Mozilla, Firebird), but I expect it will.

Comment 4

[Adam Hauner]

Changing to All as I have this problem with 1.6 (and previous milestones too) on
Win2K.

Comment 5

[Doug Halamay]

I can confirm this on Firefox:
UA: Mozilla/5.0 (Windows; compatible; U; Windows NT 5.1; en-US; rv:1.8a)
Gecko/20040417 Firefox/0.8.0+

Although I'm currently using a trunk build, the history performance in Firefox
has been really terrible for a long time now (well before the 1.7 branch).

My history.dat file is approx. 8.36MB.  I wonder what the problem is here.  Is
it just slow reading the file, or is it a sort issue?  Here's hoping this gets
looked at soon.  I may take a peak and see if I can figure it out, but I think
someone with more experience needs to take a look.

Comment 6

[D. Rata]

The slowness of the History window in Mozilla (and in Firefox by extension) is a
vexing and long-standing inefficiency, and it is one of the few areas in which
Gecko browsers suffer greatly in comparison to IE. I have a large history.dat
file (over 18 MB) in my Moz 1.7.3/WinXP profile, so I expect a short wait when
opening my history. But 30+ seconds is way too long, particularly because I am
unable to access other parts of the browser in the meantime. Also, I notice a
sluggishness when deleting history entries. In particular, deleting ten or more
entries at a time results in an exponential increase in execution time. While
waiting, I notice that my CPU utilization goes up to 100% and stays there
continuously.

In contrast, my IE history folder is also rather sizeable (nearly 12 MB
including subfolders), but IE's history pane comes up instantaneously and
remains responsive while I search, browse, and delete entries.

IE uses a two-level tree structure to store its history. In the History folder
is a master index.dat file that presumably contains basic information about all
URLs browsed. Below this top level are subfolders for each week of entries. Each
subfolder contains its own index.dat file, which presumably contains more
detailed information about that week's entries. (I use "presumably" because I
don't know anything about the internal data structure of the binary index.dat
files, but this can probably be learned with some doing.) In my case, the master
index.dat file is about 2.5 MB and the combined size of the secondary index.dat
files is about 9.5 MB.

I'm not necessarily advocating that Mozilla emulate IE's method of storing the
browser history, but IE's advantage in quickness cannot be ignored. The Gecko
browsers badly need a more efficient data structure to store history entries,
because the existing one simply does not scale well.

Comment 7

[D. Rata]

Since my previous post I have read through the comments of bug 241438 (regarding
Moz's history implementation), bug 245745 (re history in Firefox), and bug
175600 (re the disk cache). I am encouraged to see that my concerns are already
being addressed with the intended conversion of history entries and other
profile data from the kludgy Mork file format to a speedier, SQLite-powered
database.

This step represents a major change to the inner workings of Mozilla, but it is
definitely worth the effort and time required to make it work. The open-source
SQLite engine appears robust enough to address the scalability issues that I
mentioned, yet compact enough to keep the Moz code footprint at a manageable
size. Relational databases in general are a mature technology that is highly
scalable and highly optimizable in practice. The poorly designed Mork format,
however, offers no such advantages and is destined to wither once it is pruned
from the Mozilla code base.

IMO the SQLite code should be fast-tracked, as it will likely result in a major
performance increase across the board. I hope that it will result in more
efficient memory usage as well, as Moz's enormous memory footprint is another
longtime bugaboo of mine.

Comment 8

[Michael Gabriel]

its actually worse for me.
my history.dat has grown to 16 megs, mozilla now takes a 3-4 sec break before it
renders the first page. if i delelte the files its fast again...

Comment 9

[Wayne Mery (:wsmwk)]

*** Bug 294332 has been marked as a duplicate of this bug. ***

Comment 10

[Brett Wilson]

Places replaces the current history system in current Firefox nightlies. It has different performance characteristics from the old system. The new places window being slow with large histories is bug 330906.

Comment 11

[Bill McGonigle (not currently reading bugmail; please contact directly)]

OK, I'm adjusting the depends-on field to reflect Brett's comment.

  depends on: 245745 -> 330906

at first I thought of dup'ing this to that, but that's not right either.  We dumped Mork for SQLLite because Mork was too slow to be usable for bugs like this, but now SQLLite is too slow to be usable. 

So, this bug probably needs to live in case we dump SQLLite for a BDB or something to get acceptable performance, then we can change the depends-on again.

Comment 12

[quartz12h]

I have a 10MB history (about a year).
I was trying to clean it up manually; I like to keep only wikipedia and bug reporting sites history. It ran for more than an hour and didn't even complete. I was asked numerous times to stop or continue a slow script! I cannot believe this was implemented in javascript!

It's definitely time for a real format with b-tree indexing. The comment above about using embeded DB opensource is a great idea. Hypersonic or Derby would work fine if you want to do that in java.

The most important flaw to me is the history view itself. It lacks flexibility.
Whoever decided to group everything older than 6 days together hasn't used the history on a long term, obviously.

Ultimately, I expect a simple ORDER BY and GROUP BY syntax, with a WHERE clause on each field. Clearly a SQL wrapper could be made to vastly improve the history browsing.

Meanwhile, all file i/o should be done in temporary RAM buffers (even 100MB ram isn't hard to find these days) and commit the final history file in 1 shot, then hot-swap the filename atomically.

Finally, I am wondering how the painting of visited link is slowed down by searching these huge history DB.

Comment 13

[quartz12h]

Also, we should introduce the notion of blacklisting/whitelisting URLs from the URL history DB. This way, users cans not only disable the history or only disable/enable it for some sites, for any privacy reasons without the current ALL-or-nothing approach relative to the history file itself.

Comment 14

[Brian Carpenter [:geeknik]]

History in the nightly builds of 3.1b3pre is painfully slow. When I click on History it takes 10+ seconds for the Menu to display, and then when I click on Show All History, it takes another 10+ seconds for the window to display. I only have about 60 days worth of history but even scrolling the history window is laggy.  I call this painfully slow because displaying the Bookmarks is instant, and to the best of my knowledge, History and Bookmarks share the places.sqlite file, and mine is only 62MB in size. It should hardly be an issue loading something like this on my Q9450 w/ 4GB RAM and fast SATA2 drive.

Comment 15

[scratch]

if you install the SQLite Manager extension and compact your places.sqlite file, it seems to help quite a bit.

Comment 16

[Brian Carpenter [:geeknik]]

#15, while I appreciate your suggestion, I'm just not interested in running an extension to fix a problem that shouldn't exist in the first place.  If there is no lag/slowness when opening the bookmarks, there should be no lag/slowness when viewing your history. It's all in the same file. :)

Comment 17

[Discovery]

Add oil!

Comment 18

[Shawn Wilsher :sdwilsh]

As originally filed, this bug is INVALID.  The back and front end for History have been completely re-written.  Any issue you are currently seeing is *not* this bug, and you should file a new one.

Comment 19

[Shawn Wilsher :sdwilsh]

Er, to clarify, I'm not saying we have a problem, but this bug is slowly morphing into different things that were not what it was really filed for.  This ends up making things significantly harder for developers to track what the real problem is.  Me marking this as INVALID is not saying we don't think this is a problem.  In fact, there are several bugs open on things that can make history slow right now:
bug 487777
bug 466421
bug 443328
bug 432706
bug 417262

Comment 20

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

(In reply to comment #19)
> Er, to clarify, I'm not saying we have a problem

He's really not saying that we _don't_ have a problem. :)

Comment 21

[Shawn Wilsher :sdwilsh]

(In reply to comment #20)
> He's really not saying that we _don't_ have a problem. :)
Hey look - typing fail!  Yes, I mean to say that we are not saying that we do not have a problem.