Closed Bug 395299 Opened 17 years ago Closed 13 years ago

Clearing firefox's browser history doesn't change places.sqlite file size.

Categories

(Firefox :: Bookmarks & History, defect)

defect
Not set
normal

Tracking

()

RESOLVED WONTFIX

People

(Reporter: stevee, Unassigned)

Details

Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9a8pre) Gecko/2007090614 Minefield/3.0a8pre ID:2007090614

1. New profile, start firefox
2. Notice your places.sqlite filesize (~44KB)
3. Surf around a bunch of sites. (As many as possible).
4. Close firefox

5. Notice your places.sqlite filesize. (Let's say 77KB)

6. Start firefox
7. Tools > Clear Private Data
8. Tick 'Browser History
9. Click 'Clear Private Data Now'.
10. Close firefox

11. Notice your places.sqlite filesize. (Still 77KB even tho you've cleared your history and closed firefox)

12. Start firefox back up (in case this initiates some vacuuming or something)
13. Close firefox back down (in case this initiates some vacuuming or something)

14. Notice your places.sqlite filesize. (Still 77KB even tho you've cleared your history and closed firefox, and started it back up, and closed it again.)

Expected:
- After you clean your browsing history, your places.sqlite file should have decreased in size.

Actual:
- After you clean your broswing history, your places.sqlite file stays the same size.

What should happen? Is this a bug, or expected behaviour?
Flags: blocking-firefox3?
steve thanks for logging this bug.

since you are using a new profile, and a version of minefield with the incremental vacuuming change, can you check the places.sqlite file size after 15 minutes of idle?

after 15 minutes of idle (with a new profile created with a new version of minefield), we'll execute a "PRAGMA auto_vacuum")

http://lxr.mozilla.org/seamonkey/source/toolkit/components/places/src/nsNavHistory.cpp#3334
Is this a privacy bug or just a filesize bug?
> Is this a privacy bug or just a filesize bug?

file size.  

for privacy, we already zero out on delete by using the "SQLITE_SECURE_DELETE" compiler flag, that will cause SQLITE to 0-fill delete data.
(In reply to comment #1)
> since you are using a new profile, and a version of minefield with the
> incremental vacuuming change, can you check the places.sqlite file size after
> 15 minutes of idle?
> 
> after 15 minutes of idle (with a new profile created with a new version of
> minefield), we'll execute a "PRAGMA auto_vacuum")

Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9a8pre) Gecko/2007090522 Minefield/3.0a8pre (Contained landing of bug 390244)

I made a new profile, built up a small history (places.sqlite ~140KB), then cleared the browser history and waited >15 minutes, and indeed the places.sqlite dropped to 51KB! Thanks for clearing this up.
Marking as invalid based on comments.
Status: NEW → RESOLVED
Closed: 17 years ago
Flags: blocking-firefox3?
Resolution: --- → INVALID
Are we still doing vacuuming? And if not, is this bug still INVALID or not?
> Are we still doing vacuuming? 

No, we are not doing incremental vacuum anymore and we currently aren't doing vacuum.

> And if not, is this bug still INVALID or not?

The bug is valid (as it's a legitimate feature request to reclaim disk space on clearning history).

we don't currently have plans to do a full vacuum (because of how long it can take if you have lots of data). 

but thinking out loud:

I wonder how slow a full vacuum would be after clearing private data, as clearing private data should remove all of moz_historyvisits and leave us with just the moz_places, favicons, and annotations for bookmarks, right?

In this case, given how vacuum works, doing a full vacuum after clear private data might not take as long, unless you have an insane amount of bookmarks.

re-opening to see what dietrich and mak77 think, but this might get re-resolved to WONTFIX.
Status: RESOLVED → REOPENED
Resolution: INVALID → ---
This may be the wrong place to think about this, but what about all our other .sqlite files, eg: downloads, cookies, formhistory, etc. Are such files also going to grow and grow?
vacuum is not slow due to history, it's slow due to moz_places table and its indexes (because moz_places contains a lot of text), to moz_favicons (blobs) and to moz_bookmarks (if many bookmarks, but this should be a minor problem).. 

we could clean up history, delete all favicons, do an hard expiry of places, then the vacuum could take less time, but will still be large enough to require a progress bar... There should be a way to do a vacuum of single tables, but that is not supported atm.

imho other files will not suffer from long vacuum times since they contain fewer data or fewer tables, places.sqlite is the worst case containing a lot of tables, data and indexes.
(In reply to comment #8)
> This may be the wrong place to think about this, but what about all our other
> .sqlite files, eg: downloads, cookies, formhistory, etc. Are such files also
> going to grow and grow?
> 

see bug 394379 - we just need a good place to stick the vacuum, so that it doesn't hurt too much
I think that especially internet cafes and other places where Firefox is in use, and hundreds of users sharing a profile could raise the history size dramatically. It also implies the places.sqlite file size.

Normally users will clear their private data when logging of and they want to be sure that everything (e.g. history) is completely removed. So why not running a vacuum action at this time? We could bring-up a progress window if the action will take a bit longer (depending on file size).

Personally I don't like the idea in doing that while an upgrade is running because upgrades will be released with at least (normally) 2 month interval. Within this period the file size of places.sqlite could increase dramatically.
OS: Windows 2000 → All
Hardware: PC → All
I have the same problem. Clearing Private Data with all options selected in preferences doesn't delete the drop down list entries of website names and addresses under the location bar. I just find it very annoying. I like to keep my data in folders and organized. I don't like my data floating around my file system in places I didn't specifically put it. This is keeping me from using Firefox. I experienced this problem with Firefox 3 beta 4 on Mac OS X 10.4.11 PPC 1.33 GHz G4 with 1 GB RAM and 13 GB free space.
Benjamin, that sounds like bug 423266, not this bug.
(In reply to comment #11)
> I think that especially internet cafes and other places where Firefox is in
> use, and hundreds of users sharing a profile could raise the history size
> dramatically. It also implies the places.sqlite file size.

1. we cap history at 180 days by default, so there is a limit.
2. there are easy workarounds for public computers, such as keeping even less history, or disabling history altogether.
 
> Normally users will clear their private data when logging of and they want to
> be sure that everything (e.g. history) is completely removed. So why not
> running a vacuum action at this time? We could bring-up a progress window if
> the action will take a bit longer (depending on file size).

this bug is not about privacy, it's about file size. regarding privacy, read comment #3.
 
> Personally I don't like the idea in doing that while an upgrade is running
> because upgrades will be released with at least (normally) 2 month interval.
> Within this period the file size of places.sqlite could increase dramatically.

i agree. if it is beneficial to vacuum, then we should do it on a regular basis, instead of leaving it until a release comes along.
Status: REOPENED → NEW
Whiteboard: wontfix?
Bug 451915 - move Firefox/Places bugs to Firefox/Bookmarks and History. Remove all bugspam from this move by filtering for the string "places-to-b-and-h".

In Thunderbird 3.0b, you do that as follows:
Tools | Message Filters
Make sure the correct account is selected. Click "New"
Conditions: Body   contains   places-to-b-and-h
Change the action to "Delete Message".
Select "Manually Run" from the dropdown at the top.
Click OK.

Select the filter in the list, make sure "Inbox" is selected at the bottom, and click "Run Now". This should delete all the bugspam. You can then delete the filter.

Gerv
Component: Places → Bookmarks & History
QA Contact: places → bookmarks
Some points:
1- It is not common for people to let Firefox idle for 15 minutes especially because many people dont't know about this. As people rarely stay 15 minutes with an idle Firefox window, their files will be FOREVER HUGE. 

2- This affects Firefox performance as the time passes, because most users do not even know where the profile folder is in the computer and they think YOU (or WE = programmers, contributors) should care about doing this cleaning in the background, WITHOUT their intervention.

3- I think Firefox has to solve this problem, for example, when the user clicks the menu to clear something, Firefox could show the hourglass cursor, or even display a dialog explaining that the cleaning would take a little... 

I have seen some complements (extensions) promissing to shrink places sqlite and asking for donation. That's horrible: to correct a problem the browser HAS the users need to install a third party extension and are suggested to DONATE money.
And the people that NEVER install a complement ? If they never wait 15 minutes with an idle window of Firefox ?
 
In my PC, places.sqlite file had almost 10 MB, even after clearing history. Doing vacuum command with 2 command lines (sqlite3 executable and a bat file, as windows user) I shrank all sqlite files. Places.sqlite became only 200KB.

here are the lines of my bat file:

for %%i in (*.sqlite) do sqlite3 %%i VACUUM
for %%i in (*.sqlite) do sqlite3 %%i REINDEX

I just placed a bat file inside my profile folder and copyed sqlite3 executable to it and run the bat file. Magically all the sqlite files shrank drastically.
Conclusions of my previous post:
If you correct this bug, implementing VACUUM when the user clear any information, ex. history, Firefox will keep the fame of a FAST browser. It will probably also SAVE RAM MEMORY!
BUT...
If you DON'T correct this bug, Firefox will gain a fame of being a HEAVY AND SLOW browser after some time of use because people visit thousands of urls everyday.

When we clear the history WE EXPECT an improvement in Firefox performance.
I believe that VACUUMing all databases can help Firefox to recover its SPEED and LIGHTNESS, and its FAME of a fast and efficient browser.
(In reply to comment #16)

Firefox 3.6 vacuum after 5 minutes of idle and once a month (or every 2 months if the db is pretty sane). Making clear history take minutes to do its work is not a solution.
I won't comment much on extensions to vacuum, those are useless as of Firefox 3.6, but people is free to make whatever they want, including donations if they wish, nobody is forcing people to pay something, donation is just that.

> And the people that NEVER install a complement ? If they never wait 15 minutes
> with an idle window of Firefox ?

I'd say there is no way a user won't have 5 minutes of idle in a couple months, and if that happens, that just means that user is not an heavy user of the browser, thus fragmentation issues are pointless for him, he won't notice any difference.

> In my PC, places.sqlite file had almost 10 MB, even after clearing history.

And that is not an issue, 10 MB is a more than acceptable size for today's computers.

> Doing vacuum command with 2 command lines (sqlite3 executable and a bat file,
> as windows user) I shrank all sqlite files.

I hope you don't do that too often, vacuuming a database too often will reduce performances instead of what you could expect. Once a month is more than enough.

(In reply to comment #17)
> It will probably also SAVE RAM MEMORY!

why and how? Sqlite puts in memory only what it needs, so that is not going to help much. We also take a memory partition for our sqlite needs that is based on the memory the computer has available. Vacuum is not going to make a noticeable change.

> If you DON'T correct this bug, Firefox will gain a fame of being a HEAVY AND
> SLOW browser after some time of use because people visit thousands of urls
> everyday.

Yes, we all do, performance is always one of the first priorities, there are currently better targets than vacuum though.

> When we clear the history WE EXPECT an improvement in Firefox performance.

No, that command reads as "clear history", it is not "make my browser faster", if you clear history to speed up the browser, you're spending your hopes the wrong way, I would not think to remove sits from my car to make it lighter and thus faster.
The only thing you'll speed up is history sidebar, most likely.

> I believe that VACUUMing all databases can help Firefox to recover its SPEED
> and LIGHTNESS, and its FAME of a fast and efficient browser.

I'm not sure I've ever seen benchmarks of history implementations, current benchmarks are mostly based on JS or DOM. Vacuuming all databases (bug 541373) would be nice, sure, done once a month to fight fragmentation (the only issues is that fragmentations causes some query to be a bit slower, talking about 1-2% probably), but not more than that.
whatever we choose to do, it needs to be completely silent and transparent.  Otherwise we will definitely violate ux-implementation-level.
I think current solution is fine: we vacuum on idle-daily (5 minutes idle), once a month, we also inserted changes that go in the opposite direction (like fchunk size) to avoid fragmentation. And all of this is transparent to the user.
So, doing vacuum on any removal seems useless overhead even on today's fast computers, the size of the database is not directly connected to memory usage nor to performances, if not in a minimal part hardly measurable by the user. Doing vacuum that often would hurt perfornances more badly than what it'd help.
Thus, wontfixing.
Status: NEW → RESOLVED
Closed: 17 years ago13 years ago
Resolution: --- → WONTFIX
Whiteboard: wontfix?
This feature seems to be broken.  I have data for over a year in an 81mb places.sqlite file, 2 other systems are showing the same failure to clear the file (unique profiles).  One issue that was not discussed is the legal one.  Giving the impression to users that you are clearing the data off the system when in fact it's remaining hidden could open them up to liability.
You need to log in before you can comment on or make changes to this bug.