403572 - Honor a user's preferences for retaining history entries

Status: VERIFIED FIXED

(Firefox :: Bookmarks & History, defect, P1)

Description

[Mike Beltzner [:beltzner, not reading bugmail]]

My history is set to retain 180 days, and in recent (early Nov 2007) nightlies I've noticed that pages which I *know* I've visited in the past 6 months are no longer appearing in my location bar drop down or history sidebar.

I brought this up with Dietrich in IRC, and he diagnosed:

23:07 < dietrich> hm, i think the problem is that we expire moz_places records w/ zero visits
23:07 < dietrich> that are not 180 days old
23:07 <@beltzner> don't understand the last bit there ... "that are not 180 days old"?
23:08 < dietrich> right now we don't cap moz_places records (unique urls)
23:08 < dietrich> we only expire visits
23:08 <@beltzner> oh
23:09 < dietrich> but the net effect is that your urls go away when they have no visits
23:09 <@beltzner> and you search the visit table?
23:10 < dietrich> well, the query joins against it
23:10 <@beltzner> right
23:10 < dietrich> but i think that we're overpruning (not just b/c of the 20k)
23:10 < dietrich> i think that moz_places records w/ a visit count > 0 shouldn't be removed until they hit the date limit

I would agree with that assertion; if the user has set a date cap, it's going to be strange for them to see history go missing.

The next bit of our discussion centered on the cap, and how it's implemented; this might result in some additional bugs, just recording here:

23:11 < dietrich> that's the problem w/ a deterministic cap: penalizes high-volume users
23:12 <@beltzner> so what are we pruning, the visit table to 20k entries?
23:13 < dietrich> yeah. but any URLs that end up w/ zero records due to the pruning well then get removed at shutdown, and that's really unecessary
23:13 < dietrich> well, not true
23:14 < dietrich> moz_places is the biggest percentage of places.sqlite, so it's great for performance, just not great for high-volume users

Comment 1

[Mike Connor [:mconnor]]

So, I think 20k visits is almost certainly too low, but I think we need to have some sort of upper bound and morph the date cap UI to be more clear about it being a maximum history. A full 180 days of history for heavy users is going to sting a lot when it comes to perf.  I'm also not sure how we're going to prune the moz_places URLs that are for embedded links, or if those just become cruft again.

That said, to mitigate the negative bits I think that we need to:

1) raise the limit to more like 40k visits (200+ real vists per day for 180 months)
2) lower the expiry time for embedded link visits to one day
3) when we get to within 500 visits of the cap, make sure we aggressively clear the embedded visits before any "real" visits get lost.
4) continue to purge moz_places records with zero visits.
5) figure out how to present these caps sanely for users in the UI...

Comment 2

[Mike Beltzner [:beltzner, not reading bugmail]]

> So, I think 20k visits is almost certainly too low, but I think we need to have
> some sort of upper bound and morph the date cap UI to be more clear about it

I can't imagine a way in which we'd express a cap based on visits ("Remember my last [80,000] visits"); I think it would be clearer if we could instead cap on unique URLs ("Remember the last [80,000] sites I've been to").

But that's taking this bug totally off topic; we should file a follow up, by which of course I mean you. I await your CC ;)

In the meantime, let's not do something where the user says "I want to keep 180 days of history" and instead we go delete a bunch of it.

Comment 3

[Marco Bonardo [:mak]]

Probably the cap would be hit less if places did not count refreshes as new visits... if you have a lot of refreshes your history will be really limited... probably that should be discussed and eventually changed before touching up the cap that has direct influence on the performance of ALL sql queries selecting from history and places.

Comment 4

[Marco Bonardo [:mak]]

Another possibility could be:
- Remember the last [2,000] sites I've visited
- Remember a maximum of [20] visits per address

on every insert you cap on visits to a single url, this could give better closing times since the expiration happens on insert and not on browser close.

Comment 5

[Mike Beltzner [:beltzner, not reading bugmail]]

Morphing the title slightly to reflect the true intent of this bug, which is: We should not delete history entries which the user has indicated (through the provided preferences UI) they wish to retain.

In current (2007-11-16) builds that means not expiring entires that are within the range of the value specified in browser.history_expire_days.

Bug 402880 is about changing the history retention preferences UI to reflect the new pruning mechanisms implemented by Places in order to preserve acceptable history/search performance (which means that this actually depends on, not blocks :)

Comment 6

[Dietrich Ayala (:dietrich)]

(In reply to comment #5)
> Morphing the title slightly to reflect the true intent of this bug, which is:
> We should not delete history entries which the user has indicated (through the
> provided preferences UI) they wish to retain.
> 
> In current (2007-11-16) builds that means not expiring entires that are within
> the range of the value specified in browser.history_expire_days.
> 

This occurred due to bug 332748. The final decision, arrived at a Tuesday Fx meeting IIRC, was "lets cap visits so that history is not painfully slow, and figure out the UI in beta 2". This clearly has had a negative effect, in that we're keeping less history than our 180-day users have become accustomed to, and in some cases, less than the Fx2 9 day default.

To respect the true intent of this bug, the fix is to disable the visit cap altogether, until bug 402880 is fixed.

FWIW, the current implementation does respect a user-specified date range. We only use the visit cap if the value of the history_expire_days pref is the default. However, if the user is at all aware of the 180 day limit, then this bug occurs, so the "default pref" approach is flawed.

> Bug 402880 is about changing the history retention preferences UI to reflect
> the new pruning mechanisms implemented by Places in order to preserve
> acceptable history/search performance (which means that this actually depends
> on, not blocks :)
> 

Ok.

Comment 7

[bomfog]

So far it seems this bug has mostly been concerned with keeping no less than x days of history. Is there a way for/need to specify keeping no more than x days? 

I want my visited links to expire once my memory of the page is likely to have faded sufficiently. I won't go into how few days that is, nor how fast the number's dropping.

Can I do that?

Comment 8

[Mike Connor [:mconnor]]

Resolving fixed, since 402880 landed last night, if this is wrong, please reopen...

Comment 9

[Tracy Walker [:tracy]]

verified with Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9b2pre) Gecko/2007120504 Minefield/3.0b2pre

Comment 10

[Gervase Markham [:gerv]]

Bug 451915 - move Firefox/Places bugs to Firefox/Bookmarks and History. Remove all bugspam from this move by filtering for the string "places-to-b-and-h".

In Thunderbird 3.0b, you do that as follows:
Tools | Message Filters
Make sure the correct account is selected. Click "New"
Conditions: Body   contains   places-to-b-and-h
Change the action to "Delete Message".
Select "Manually Run" from the dropdown at the top.
Click OK.

Select the filter in the list, make sure "Inbox" is selected at the bottom, and click "Run Now". This should delete all the bugspam. You can then delete the filter.

Gerv