Closed Bug 515538 Opened 16 years ago Closed 15 years ago

places.sqlite keeps growing without limit

Categories

(Firefox :: Bookmarks & History, defect)

Product:
Firefox
Component:
Bookmarks & History
Platform:
x86
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED DUPLICATE of bug 512854

People

(Reporter: linuxhippy, Unassigned)

Details

User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 I've been using my profile all the time back to FireFox-3.0, and places.sqlite is now about 100mb large after vaccum. The large db size stresses my notebook disk quite a lot, leading to lags as long as places.sqlite is not in disk-cache. There are only two large tables: moz_historyvisits: 1554490 entries moz_places: 83357 entries I already lowered some history-related settings, however without any effect: browser.history_expire_days=90 browser.history_expire_days.mirror=90 browser.history_expire_days_min=45 browser.history_expire_sites=20000 > ls -la places.sqlite > -rw-r--r-- 1 ce ce 98921472 2009-09-03 17:27 places.sqlite When does firefox decide to execute "delete from moz_places, moz_historyvisits" statements? In my case it seems the entries stay forever :-/ Reproducible: Always Actual Results: places.sqlite is already 100mb large, even with lowered history settings nothing changed. Expected Results: at least after changing history-value + vacuum database size should decrease
Have you checked out your moz_bookmarks table as well? Places.sqlite holds both bookmarks & history - sorry if you already knew that and accounted for that.
do you use google toolbar or StumbleUpon extensions? some extension is known to add a bunch of stuff to our database, that we don't have control upon.
And no, I haven't installed one of the two extensions mention. These are all tables with more than 100 entries: bookmarks: 131 favicons: 2132 inputhistory: 477 places: 83357 history_vists: 154490 All the places.sqlite data seems to come from places/history_vists favicons. As it looks, firefox doesn't delete the data out of the database. Another strange thing seems to be places/history_visits tables haven't changed in sice since I first looked at them - both still have the same amount of entries. When does firefox execute its "delete from table" statements?
could be you have collected a bunch of visits in a short amount of time. But could also be your db is corrupt. Could you please execute a PRAGMA integrity_check; on it, the same way you did execute vacuum. expiration happens at idle and at shutdown. definately having reduced history prefs you should see those entries becoming less at every browser close.
Integrity check says the DB is ok. (after running for quite a while ;) ). I also haven't collected a lot of visits in a short time, I remember the db beeing 30mb one or two years ago, then 50mb and now its ~95mb. So either there are some wrong settings (which relevant one I haven't posted above do exist), or a bug in firefox leading to the behaviour of not deleting some rows in the db?
Any ideads how I could debug this, no that I've verified the db's integrity is ok?
I am curious what your computer's specs are? CPU type/ speed/ RAM? Your database is on the largish size for a places database, and your row counts confirm this. I don't think your db size is anything *way* out of the ordinary.
The question is why are there so many rows, not why the database file is that big. My specs are completly irrelevant, reading a 100mb file from disk takes *A LOT* of time, especially if it tends to fragment that heavily. However, because I did not get any useful feedback, I fired up "SQLLite Manager" and removed all rows I don't need. Now its fast and sound again.
(In reply to comment #8) > However, because I did not get any useful feedback, I fired up "SQLLite > Manager" and removed all rows I don't need. fwiw, this is an awesome method to corrupt your bookmarks. We can't tell why you have many entries, could just be due to your "way of browsing", looks like lot of pages. Still something is broken in expiration or you never let you browser go in "idle" state.
> fwiw, this is an awesome method to corrupt your bookmarks. Yes, they are gone now too. But since I received no help from here and firefox was already painful to use, what else should I've done? > Still something is broken in expiration or Its a bug somewhere in expiration for sure. However I never got any help howto really diagnose the problem, so I simply don't care anymore. The data is gone, as is the problem (hopefully). - Clemens
(In reply to comment #10) > > fwiw, this is an awesome method to corrupt your bookmarks. > Yes, they are gone now too. But since I received no help from here and firefox > was already painful to use, what else should I've done? removing places.sqlite would have generated a new clean one with only your bookmarks. Notice this is bugzilla, not supportzilla, to get support help you should go to support.mozilla.org. Here we only try to debug and fix issues but clearly we needs steps or files to reproduce the issue. And if we ask for informations you should not consider them irrilevant, we need them to try reproduce the issue. Btw, we plan to rewrite expiration for 3.7, and 3.6 has a lot of perf improvements, could even be upgrading to 3.6 would have been enough for your case.
Bug 451915 - move Firefox/Places bugs to Firefox/Bookmarks and History. Remove all bugspam from this move by filtering for the string "places-to-b-and-h". In Thunderbird 3.0b, you do that as follows: Tools | Message Filters Make sure the correct account is selected. Click "New" Conditions: Body contains places-to-b-and-h Change the action to "Delete Message". Select "Manually Run" from the dropdown at the top. Click OK. Select the filter in the list, make sure "Inbox" is selected at the bottom, and click "Run Now". This should delete all the bugspam. You can then delete the filter. Gerv
Component: Places → Bookmarks & History
QA Contact: places → bookmarks
Let me add a piece of evidence to this issue. I am running XP Pro with Firefox 3.5.x. I do have the Google Toolbar, and I had selected the toolbar option to show the thumbnails when a new tab is opened. For some time, I have been noticing that Firefox took a LONG time to start up, and Task Manager was typically showing it taking, maybe about 200 MB of memory usage. Recently, when I was defrag'ing my disk, I got a report that places.sqlite was over 500 MB and would not defrag. After some research on the file, I turned off the thumbnails option. I installed sqlite and did a compact file function on places.sqlite. The file was reduced to about 45MB. Firefox now starts up faster - but still slower than it should, IMHO, and task manager now shows it initially as about 100 MB of memory usage. I have no idea why places.sqlite grew so large or what my current file contains for data. However, I think, at least for performance and efficiency reasons, Firefox needs to include user tools or functions to compact and repair these databases. Maybe there is another approach, but letting these files grow so large without giving the enduser some means to diagnose and handle these files is not in the long run sustainable. john.
(In reply to comment #13) > Let me add a piece of evidence to this issue. I am running XP Pro with Firefox > 3.5.x. I do have the Google Toolbar, and I had selected the toolbar option to > show the thumbnails when a new tab is opened bug 489173, this is Google Toolbar fault, not ours. > Recently, when I was defrag'ing my disk, I got a report that places.sqlite was > over 500 MB and would not defrag. After some research on the file, I turned off > the thumbnails option. I installed sqlite and did a compact file function on > places.sqlite. The file was reduced to about 45MB. Indeed, Firefox 3.6 will compact for you, in future. > Firefox now starts up faster - but still slower than it should, IMHO, and task > manager now shows it initially as about 100 MB of memory usage. 100MB can be a good value, depending on your amount of available memory, browsers are quite complex nowadays.
Marco, Thanks for your comments. I made note of my Google Toolbar option because I knew that was a factor in the large size. However, I was trying to support the concept that that the .sqlite files need to be able to be compacted by Firefox no matter what add-on might balloon them. I will look forward to 3.6; any time frame when that might be? I agree, 100MB may be good value. 200 MB was not. The bug you referenced led to an add-on http://lifehacker.com/5347125/vacuum-places-improved-speeds-up-firefox-with-a-click-of-your-mouse Can you comment on it? Is it still necessary? Further, I noted that a later build of the Toolbar supposedly fixed the toolbar problem. I thought I was using the latest... Does it fix it, in your opinion? thanks, john.
(In reply to comment #15) > Marco, > > Thanks for your comments. I made note of my Google Toolbar option because I > knew that was a factor in the large size. However, I was trying to support the > concept that that the .sqlite files need to be able to be compacted by Firefox > no matter what add-on might balloon them. It depends, we cannot delete data at will, we can only compact unused space. So if the DB is 500MB of data, nothing can be done without touching user's data. > The bug you referenced led to an add-on > > http://lifehacker.com/5347125/vacuum-places-improved-speeds-up-firefox-with-a-click-of-your-mouse > > Can you comment on it? Is it still necessary? you can still use it if you want (i'd say not more than once a month), in Firefox3.6 we have a vacuum task that runs at last every month, at least every 2 months, so ideally it is not needed. It also has the possibility to force a vacuum using a small script in the error console, but as you can see that's not an easy task for a common user (we should link that to a button in about:support for 3.7). > Further, I noted that a later build of the Toolbar supposedly fixed the toolbar > problem. I thought I was using the latest... Does it fix it, in your opinion? I don't know their code, i think they fixed the issue, they also probably removed the entries, but the database won't be compacted by FX3.5 so if the toolbar made the db 500MB, it will stay 500MB even if most of the space is unused. 3.6 will compact though.
Marco, Thanks again. As I did the compact just after I turned off the Google Toolbar thumbnails, I cannot say that the "empty" space in places.sqlite was caused by turning off the option or was in the file originally. However, I do accept that the compact operation cannot remove actual data; only reclaim space no longer being used. I am not sure I understand what the Google fix is doing to places.sqlite. I can see why you say that it is a going forward fix; however, I do not understand what it is doing differently now. IE, if I were to turn the option on again, would the file just continue to grow, but more slowly? Or would it be doing a continuing garbage collection and removal? If the later, wouldn't that imply that the old data could/would get purged over time? BTW, I likely did not have the latest Google Toolbar; now I do. I will need to monitor its behavior now... As a "common user", when can I expect this FF3.6? Where is the error console? (or is this an upcoming feature?) thanks, john.
assuming they fixed the bug, they are probably now saving thumbnails in a separate database, so the problem should not reappear with their latest version. Firefox3.6 is expected during first weeks of next year, not so far.
Re comment #14: Will this include compressing all sqlite files or only places.sqlite?
(In reply to comment #19) > Re comment #14: Will this include compressing all sqlite files or only > places.sqlite? only places.sqlite
Isn't that bug resolved on trunk, possibly a dupe? Can it please be marked as such?
yep, thanks. Ideally there was still the google toolbar issue, but that's not our fault and latest version should work better.
Status: UNCONFIRMED → RESOLVED
Closed: 15 years ago
Resolution: --- → DUPLICATE
Still see this problem with the original profile with FireFix-4
(In reply to comment #23) > Still see this problem with the original profile with FireFix-4 sorry, but 99% the issue is caused by one of your add-ons, if you want to test them in a new profile and fiure out what it is, could be useful.
You need to log in before you can comment on or make changes to this bug.