425219 - Places' history expiration algorithm seems overly complicated

Status: RESOLVED WONTFIX

(Firefox :: Bookmarks & History, enhancement)

Description

[Steve England [:stevee]]

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b5pre) Gecko/2008032523 Minefield/3.0b5pre ID:2008032523

The Places history expiration algorithm seems overly complicated.

The three relevant prefs are:
 - browser.history_expire_days_min = 90
 - browser.history_expire_sites = 40,000
 - browser.history_expire_days = 180
 
Firefox will:
 (1) Remember _all_ places visited for the first 90 days, with no exceptions (even if it's something like 200,000 history entries)
 (2) If after 90 days of browsing there are less then 40,000 history entries, additional history entries will be remembered, up to the limit of 40,000; with the proviso that:
 (3) If by 180 days the user still hasn't hit the 40,000 history entries limit, sites older than the 180 days limit will be removed from history.

For point (1) the user can set the number of days of history to remember in the Firefox prefs. Firefox is guaranteed to remember all URLs visited in this period, and this is what users expect. In this case, the size of the user's places.sqlite file is unbound and can be any size, depending on how many URLs they visit in this 90 day period.

For point (2) browser.history_expire_sites will make Firefox remember more history than the user expects, providing that in the last 90 days they haven't visited more than 40,000 URLs. Why this arbitrary limit of 40,000? For the first 90 days the user's history size is unbound and could reach into the 100,000s and beyond anyway. All this pref seems to do is potentially make firefox remember more of the user's history than they expect.

Then for point (3), if after 180 days the user has visited less than 40,000 sites, we start removing their history for no real purpose. Either the 40,000 limit is good enough, or it's not, so I don't see the reason after unexpectedly keeping more history than the user expected, to then unexpectedly start removing history items after an arbitrary length of time has elapsed.

My main point is, since (1) is true, why bother with the complication of (2) and (3) both in terms of code complexity, but also with respect to doing what the user expects. Randomly keeping more history than is expected with the possibility of then randomly loosing the history that was randomly kept in the first place seems silly to me.

Whether the user wants to keep 10 days of history or 200 days of history, why can't we just do exactly what they want (point (1), which we do anyway) and not bother going to the effort of doing unexpected things by enforcing arbitrary limits (points (2) and (3))

Comment 1

[Thomas Bertels]

This was done to prevent slow downs while maximizing the history retention.
See also bug 332748.

Comment 2

[Marco Bonardo [:mak]]

reference bug for future thoughts on expiration algo

Comment 3

[Radek 'sysKin' Czyz]

This comes up on support channels every now and then.

It would appear that there's a large group of users who don't put too much value on their history, particularly older history.

These people often see too big history as clutter in awesomebar, or a privacy risk, or unnecessary load on slow computers / slow (networked) roaming profiles. 

An obvious solution these people implement is to limit their history - usually to less than two weeks, sometimes to two days, depending on what they like.

There's completely no setting in prefs that would help them. Despite Prefs setting being relatively clear what it does when you think about it enough, their understanding is different, and rightly so given what it used to mean in FF2 as well as other browsers.

It is my humble opinion that simply expiring older items on schedule was a better solution, and "maximising history retention" is just not something most people want/need/expect. Not useful enough, simply put.

Comment 4

[mirage]

hi i've ever report this problem here :https://bugzilla.mozilla.org/show_bug.cgi?id=458849

Yes I think this must change. 

This should be :
If this option is checked => History is saved
the period backup history limits backup duration of history.

This must be keep my history a last x days !

Comment 6

[Mike Connor [:mconnor]]

Our original goal was to have 180 days of history.  We added the 40k visit limit as a way of limiting overhead.  But for very heavy users, this shortened the set of available history dramatically, so we added the minimum cap.  I don't think a straight 90 day cap is ideal, and I don't really get why "algorithim complexity" is something end users care about.  If it made expiring history more painful, sure, we should look at that, but as far as I understand, that's not the case.  The original bugs have lots of debate, but as far as I'm concerned, this is working well and as intended, so marking WONTFIX.  Users who really want to expire after N days can set history_expire_days to the same as history_expire_days_min and be content.

Comment 7

[Marco Bonardo [:mak]]

well, actually we have places stats, so we could guess a better number of pages limit (the biggest reported db had 150000) and bring the limit to an hard 120 or 150 days (we demonstrated to handle 180 days quite well). That would simplify the code a lot and most likely make expiration faster.
From a users point of view, i must admit very few of them actually notice the "LEAST" word in preferences, while having a "Remember a maximum of X(150000) pages from the last Y(120) days" would be really simpler for most of them.

Comment 8

[Gervase Markham [:gerv]]

Bug 451915 - move Firefox/Places bugs to Firefox/Bookmarks and History. Remove all bugspam from this move by filtering for the string "places-to-b-and-h".

In Thunderbird 3.0b, you do that as follows:
Tools | Message Filters
Make sure the correct account is selected. Click "New"
Conditions: Body   contains   places-to-b-and-h
Change the action to "Delete Message".
Select "Manually Run" from the dropdown at the top.
Click OK.

Select the filter in the list, make sure "Inbox" is selected at the bottom, and click "Run Now". This should delete all the bugspam. You can then delete the filter.

Gerv