435642 - Too much I/O access to places.sqlite and places.sqlite-journal

Status: RESOLVED WORKSFORME

(Firefox :: Bookmarks & History, defect)

Description

[Philip]

User-Agent:       Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.9) Gecko/2008051206 Firefox/3.0

The files urlclassifier2 and places.sqlite-journal are access very often! 633x in 3 seconds per file. See this information via Filemon.exe.

Reproducible: Always

Steps to Reproduce:
1. start filemon.exe
2. set filter to urlclassifier2.sqlite or/and places.sqlite-journal
3.
Actual Results:  
start surving on the internet

Expected Results:  
see results in filemon.exe

Comment 1

[Jo Hermans]

That's normal, those are the databases for Places (bookmarks and history) and the Spyware/Malware checks. 633 accesses in 3 seconds seems quite a lot, it's probably due the urlclassifier2 file that is being downloaded. That happens the first few days that you use Firefox, until it reaches 55 MB or so. After that, it should stop.

Firefox 3.0 received a lot of optimizations in this area the last few months, more might be coming in the future.

Comment 2

[Jo Hermans]

See for instance bug 421482 for an improvement that is coming in a later version, although that ones hurts Unix/Linux much more than Windows.

Comment 3

[Hugh Allen]

For me it's urlclassifier3.sqlite and urlclassifier3.sqlite-journal. Better to set the filter to "Process Name is firefox.exe". (also I'm using Sysinternals' Process Monitor which supersedes Filemon)

I have a noisy HDD and *always* notice disk activity. If it's excessive / unexplained / unnecessary it concerns me.

How can I "confirm" a bug? I suppose I would need higher privileges on bugzilla?

Comment 4

[Philip]

Hello,

After a month the HD is still very busy if I run firefox. Its the same problem, access the two files very offten. I think its a bug in firefox.

Philip

Comment 6

[Lucas Malor (mail: c6kfnkn2uc AT snkmail DOT c0m)]

This bug seems to not have duplicates and could be confirmed. Anyway it describes the same generic problem, but for two different components.
I think it's better that someone will open two bugs: one for anti-phishing and the other for Places, and resolve this bug as invalid.

Comment 7

[Lucas Malor (mail: c6kfnkn2uc AT snkmail DOT c0m)]

Emh. I forgot Bug 441481, that I marked as duplicate of this bug by mistake. So only one bug must be opened: the Places one.

Comment 8

[VanillaMozilla]

There is good evidence on places here:
http://forums.mozillazine.org/viewtopic.php?f=38&t=838165 .  The number of writes in a short time greatly exceeds the number of entries.  It seems to be related to 3800+ entries in the moz_bookmarks table and 4100+ entries in the moz_places table, which doesn't seem excessive.

Comment 9

[VanillaMozilla]

I'm going to stick my neck out a little and confirm and morph this for "places.sqlite" on the basis of user-supplied information.  Per comment 8, user now reports hard drive thrashing with a new profile but old "places.sqlite", even though places file is only 3.5 MB.

Morphing the bug by removing "urlclassifier", rather than opening a new one.  Hope I'm not stepping on toes, but not much is happening yet, and this keeps all the info together with this bug.

For the record--original summary was "Too much I/O accesses to urlclassifier3.sqlite, places.sqlite-journal and urlclassifier2.sqlite, high hard disk activity".  New summary "Too much I/O access to places.sqlite and places.sqlite-journal".

Comment 10

[VanillaMozilla]

Forgot to mention, this was not the bug reporter, but a separate user--who supplied procmon logs.

Comment 11

[michael fox]

I came to this bug in a different way.  I'm running Firefox 3.0 and Firefox 3.1 Beta 1 on an Asus eee pc running Ubuntu 8.04 (hardy).

On this PC (which is becoming very popular) Firefox runs like ass.

I found out that it has slow write performance on it's solid-state hard drives.

The thing that is destroying firefox performance is that it is constantly writing to these various sqlite databases.  

And it's not just that it write to the databases, it's the way it writes to them.  Most processes politely send data to the Linux vm and then let it, in combination with the ext3 filesystem, decide when to write it to the disk.  But not the Firefox sqlite -- no -- this sqlite SOB forces the disk to sync every single write.  So, it doesn't let the OS cache the database, it makes sure that every write (of which there are dozens per page request) is accompanied by an IO-blocking sync which stops firefox and makes it appear to freeze and unfreeze and freeze and unfreeze constantly.

The ubuntu-eee community has come up with this workaround: disable history and checking for evil sites in Preferences -> Privacy.  Also, disable remembering how forms are filled in.  Then there's a lot less database access (though still some).

This issue is very serious.  If sqlite could just be normal and allow the operating system to do its job, then it would be fine if it made 100 write requests a minute.  But all this syncing is sinking Firefox performance on solid-state drives and (I see from this thread) ordinary hard drives alike.

Comment 12

[michael fox]

Please make this bug a priority for the 3.1 release.  Most people don't realize the exact nature of the bug, but they notice that firefox is doing weird things to their hard disk, runs stop-and-starty, and generally sucks ass.  Solving this one little problem will improve performance for affected users more than any other thing you could do.

Comment 13

[michael fox]

The reason that this bug has persisted until now is that most hard drives or hard-drive controllers have a hardware-based write caching system above and beyond what the operating system provides.

The solid-state drive on the asus eeepc does not have that.  I imagine that other people complaining about their hard drives griding have really cheap systems with no hardware write cache too.  The upshot, is that you will not see this performance problem on a fancy developer desktop or a macbook pro.  You need some **** hardware to experience it.

There is a way to see it, though (at least under linux), you do the following (as root):
Kill all log deaamons like klogd and syslogd.
Goto /proc/sys/vm
echo 1 > block_dump;  -- this will causes the kernel to dump information on low-level writes to the hard disk
echo 30 > dirty_background_ratio
echo 0 > dirty_expire_centisecs
echo 60 > dirty_ratio
echo 0 > dirty_writeback_centisecs

Then remount your ext3 filesystem with a long commit time.  This is the system that firefox is writing it's database on, /home or whatever:
mount /home -o remount,commit=60000

At this point the operating system is trying to cache almost everything.  It will not write anything to the disk unless it is desperate for memory or someone executes a low-level sync command.  You can do that from the command line by saying:
sync

Now, you want to watch this by saying:
cat /proc/kmsg

This is giving you the kernel debug messages that you enabled with block_dump.
Now if you run firefox with history and malicious websites and forms and cookies all disabled, then you'll mostly see messages like:

<7>[ 4462.147950] firefox(6918): dirtied inode 670643 (sessionstore-1.js) on sdb1

Which means that firefox wrote something to the disk but it is being cached for now.  You should only see messaages like that until the OS is desperate for RAM or someone does a low-level sync.  

If you turn on history in Preferences -> Privacy, you will start to see tons of messages like:

<7>[ 5089.705103] firefox(6918): dirtied inode 670648 (cookies.sqlite-journal) on sdb1
<7>[ 5091.798717] firefox(6918): dirtied inode 670648 (sessionstore-1.js) on sdb1
<7>[ 5093.219440] firefox(6918): dirtied inode 670644 (places.sqlite-journal) on sdb1
<7>[ 5093.230582] firefox(6918): WRITE block 10852792 on sdb1
<7>[ 5093.230683] firefox(6918): WRITE block 10852800 on sdb1
<7>[ 5093.230728] firefox(6918): WRITE block 10852808 on sdb1
<7>[ 5093.230763] firefox(6918): WRITE block 10852816 on sdb1
<7>[ 5093.230802] firefox(6918): WRITE block 10852824 on sdb1
<7>[ 5093.230837] firefox(6918): WRITE block 10852832 on sdb1
<7>[ 5093.230875] firefox(6918): WRITE block 10852840 on sdb1
<7>[ 5093.230909] firefox(6918): WRITE block 10852848 on sdb1
<7>[ 5093.230942] firefox(6918): WRITE block 10854000 on sdb1
<7>[ 5093.230994] firefox(6918): WRITE block 10854008 on sdb1
<7>[ 5093.231031] firefox(6918): WRITE block 10854016 on sdb1
<7>[ 5093.232510] kjournald(4639): WRITE block 10877024 on sdb1
<7>[ 5093.232576] kjournald(4639): WRITE block 10989688 on sdb1
<7>[ 5093.232611] kjournald(4639): WRITE block 11167712 on sdb1
<7>[ 5093.232648] kjournald(4639): WRITE block 7504240 on sdb1
<7>[ 5093.232689] kjournald(4639): WRITE block 10846632 on sdb1
<7>[ 5093.232724] kjournald(4639): WRITE block 7841632 on sdb1
<7>[ 5093.232758] kjournald(4639): WRITE block 7520784 on sdb1

Which means now your program is forcing every access to the sqlite database to sync with the hard drive and all hell has broken loose.

Comment 14

[Dietrich Ayala (:dietrich)]

Removing blocking request, since this should be fixed by the fsync-reduction work in bug 442967, which is already blocking. We can re-evaluate the blocking status of this bug after the fsync work lands.

Comment 15

[Dietrich Ayala (:dietrich)]

Firefox 3.5 drastically reduced the number of writes to the database. It now periodically flushes browsing history, defaulting to once every 2 minutes. The flush will be immediately triggered by critical writes such as adding a bookmark.

I'm going to close this bug. Please re-open if you're still experiencing an overly high frequency of writes to the database.

Comment 16

[Gervase Markham [:gerv]]

Bug 451915 - move Firefox/Places bugs to Firefox/Bookmarks and History. Remove all bugspam from this move by filtering for the string "places-to-b-and-h".

In Thunderbird 3.0b, you do that as follows:
Tools | Message Filters
Make sure the correct account is selected. Click "New"
Conditions: Body   contains   places-to-b-and-h
Change the action to "Delete Message".
Select "Manually Run" from the dropdown at the top.
Click OK.

Select the filter in the list, make sure "Inbox" is selected at the bottom, and click "Run Now". This should delete all the bugspam. You can then delete the filter.

Gerv