1324960 - QUERY_FIND_EXOTIC_VISITS_TO_EXPIRE has unintended side effect to the "Remember history" setting

Status: RESOLVED WONTFIX

(Firefox :: Bookmarks & History, defect, P3)

Description

[beepboopbugzilla]

I'm losing older download history records despite setting "Remember history" and being nowhere close to the places.history.expiration.transient_current_max_pages pref value.

Steps to Reproduce: 
1) Set "Firefox will: Remember history" in about:preferences#privacy.
2) Leave browser alone for a while.

Actual Results:
Lost download (and some other history) records older than 60 days.

Expected Results:
Keep all records because of "Remember history" setting.


Looking at Bug 1262887, I understand the reasoning behind the creation of QUERY_FIND_EXOTIC_VISITS_TO_EXPIRE.  Unfortunately, it has the unintended consequence of bypassing what "Remember history" is stated to do.  I know there's precedent for ignoring the setting:  older pages expire based on an automatically calculated threshold -- but at least that's not completely hidden from a user and it can be controlled with an about:config pref.  In contrast, QUERY_FIND_EXOTIC_VISITS_TO_EXPIRE runs counter to expected behavior from "Remember history", with no way to control it.

Is there a pref to prevent QUERY_FIND_EXOTIC_VISITS_TO_EXPIRE from running?

Comment 1

[Marco Bonardo [:mak]]

this is working as intended, there's no setting to tweak the value here, and I'm not sure it makes sense to introduce one, when also the other settings are pretty much automatic (true, you can set current_max_pages, but that's a really advanced setting that is also an hidden pref).

We evaluated that downloads are not that useful after 60 days, and as such we set the limit there. It's possible we are wrong, but to revert that decision we need user cases and facts that benefit most users.
What's your use case for looking at so old downloads?

Comment 2

[beepboopbugzilla]

On a personal level, I've kept old download history to find original sources I've downloaded files from but either didn't bookmark or forgot the URL (and only have filename to go by).  I've used it in the opposite direction also: to quickly find out what directory I've saved a file to (I don't always use the same directory).

I set "Remember history" because it states "Firefox will remember your browsing, download, form and search history, and keep cookies from websites you visit."  It's true that the oldest pages are expired when current_max_pages is exceeded, but at least a way to avoid that is easily googled and set via pref for users who desire to keep them even longer and understand the consequences.  On the other hand, expiring old downloads cannot be avoided, even if one wants to remember all history.

I'm not doubting the usefulness of QUERY_FIND_EXOTIC_VISITS_TO_EXPIRE.  Not asking for its elimination.  My only issue is that it's even more hidden from advanced users and uncontrollable. QUERY_FIND_VISITS_TO_EXPIRE is also inconsistent with the "Remember history" setting, but at least users have the option to control it with a pref.

Comment 4

[Masatoshi Kimura [:emk]]

(In reply to beepboopbugzilla from comment #2)
> On a personal level, I've kept old download history to find original sources
> I've downloaded files from but either didn't bookmark or forgot the URL (and
> only have filename to go by).

I'm using the download history for exactly the same purpose.

Comment 5

[Masatoshi Kimura [:emk]]

Moreover, this cleanup will turn the visited download link back to non-visited after 60 days. Now I have to remember if I downloaded the file. It should be user-agent's job. Even if it is reasonable to cleanup old downloads, it should not affect the visited link state.

Comment 6

[Masatoshi Kimura [:emk]]

(In reply to Marco Bonardo [::mak] from comment #1)
> We evaluated that downloads are not that useful after 60 days, and as such
> we set the limit there.

Why? Where is the discussion? Bug 1262887 is about long URLs and has no explanations at all why downloads are suddenly added to cleanup candidates.

Comment 7

[jscher2000]

(In reply to Marco Bonardo [::mak] from comment #1)
> We evaluated that downloads are not that useful after 60 days, and as such
> we set the limit there. It's possible we are wrong, but to revert that
> decision we need user cases and facts that benefit most users.
> What's your use case for looking at so old downloads?

60 days is not old. My work involves projects spanning 12-36 months, and I save files in innumerable folders. If I want to find some research or filings I downloaded, it is far easier to search in Firefox's download history than to do a global search of my computer. Firefox's download history also shows the download date and URL in case I need that information for an affidavit; that metadata is not stored on disk with the files.

I also lost track of a set of eBooks I downloaded on Sept. 21, 2016 (for free). On December 27th, they went on year end sale. Since they looked familiar, I tried to find them in Downloads and they weren't listed, so I bought them. Yes, it's stupid to be so disorganized, but when Firefox always took care of me before, it's understandable.

Comment 8

[Marco Bonardo [:mak]]

You know, someone will come here and state he needs his history forever. And he's not wrong, just that it's not the most common thing for users, and has pretty bad performance implications. So we will never do that, even if the request would make sense.

The reason behind expiring downloads earlier, is that history can't grow forever, we must constantly expire stuff. So the choice is between expiring an old download, or an old visited page. The download looks far less interesting to keep, also because the visit to the page from which the download started remains, and it's more likely for the downloaded file to have been moved/removed.

That said, we can evaluate a compromise, which limit do you think it would work before you are unlikely to care about a downloaded file? 4 months?

Comment 9

[beepboopbugzilla]

How about an option to enable/disable it through the about:preferences#privacy page (also should be a warning that despite setting "Remember history" entries will still be deleted)?

Alternatively, how about a boolean pref in about:config to enable/disable the function?

There should be an option whichever way to control if QUERY_FIND_EXOTIC_VISITS_TO_EXPIRE runs or not.  Let the users make the decision of facing potential performance issues.

Expiring pages past a maximum automatically calculated value is fine because users still have the option to control it with places.history.expiration.max_pages.  By contrast, there is no way to control losing EXOTIC_VISITS.

Comment 10

[Marco Bonardo [:mak]]

(In reply to beepboopbugzilla from comment #9)
> Let the users make the
> decision of facing potential performance issues.

Very often the user thinks he doesn't care about perf, and then some time later he files a bug about performance being horrible. Then we are left with a new option to maintain and a new bug to deal with.

Comment 11

[beepboopbugzilla]

Users going into about:config know the potential consequences of making changes, especially if they go so far to thoroughly research specific prefs.  Setting places.history.expiration.max_pages comes with the understanding that it can impact performance.  A boolean pref to control expiring exotic_visits should also come with the same awareness.

Comment 12

[jscher2000]

As a temporary workaround to save the data, I've been using this monthly:

SQLite Manager extension: https://addons.mozilla.org/firefox/addon/sqlite-manager/

+

SELECT content AS Filename, datetime(dateAdded/1000000,'unixepoch','localtime') AS dateAdded, url AS Location, moz_anno_attributes.name AS Annotation
FROM moz_places, moz_annos, moz_anno_attributes
WHERE (moz_places.id = moz_annos.place_id) AND (moz_annos.anno_attribute_id = moz_anno_attributes.id) AND (moz_anno_attributes.name LIKE 'downloads/destinationFileURI')
ORDER BY dateAdded DESC

+

Results: Save As CSV

Comment 13

[mirh]

(In reply to Marco Bonardo [:mak] from comment #10)

Very often the user thinks he doesn't care about perf, and then some time
later he files a bug about performance being horrible. Then we are left with
a new option to maintain and a new bug to deal with.

I never really thought about that..
But anyway, even then, it seems clear what the right thing should be between "some little annoyance on the bug tracker every odd year" and "some use cases not having any way to be carried out whatsoever".

Comment 14

[Marco Bonardo [:mak]]

we have increase the timer to 90 days, that is actually the limit for ALL of history in other commonly used browsers... That should be sufficient for downloads, also considered the file manager keeps its own history by file added / last modified.