1476152 - Never Remember History doesn't clear history when activated

Status: RESOLVED WONTFIX

(Toolkit :: Data Sanitization, enhancement)

Description

[Nick Levinson]

Never Remember History is on, history includes the cache, and no exceptions are present. After quitting and restarting FF without going to the Internet and again after warm-rebooting the machine without going to the Internet, FF kept the cache. Mine had 4.5 MB each time. Never Remember History, since history includes the cache, means the cache should be automatically emptied on a browser quit. The browser should be forced to re-retrieve pages whenever FF is started and I enter a URL.

I'm using FF 61.0.1 (64-bit) fedora - 1.0 on a Fedora 28 Linux platform kept virtually evergreen.

Comment 1

[Honza Bambas (:mayhemer)]

(In reply to Nick Levinson from comment #0)
> Never Remember History is on, history includes the cache, and no exceptions
> are present. After quitting and restarting FF without going to the Internet
> and again after warm-rebooting the machine without going to the Internet, FF
> kept the cache. Mine had 4.5 MB each time. Never Remember History, since
> history includes the cache, means the cache should be automatically emptied
> on a browser quit. The browser should be forced to re-retrieve pages
> whenever FF is started and I enter a URL.
> 
> I'm using FF 61.0.1 (64-bit) fedora - 1.0 on a Fedora 28 Linux platform kept
> virtually evergreen.

If you leave Firefox running and shutdown or restart the system, the Firefox process will not have a chance to clean the cache up - it takes time, and the system will kill it.

There is probably no way to tell Fedora to wait until it's done.  Windows has a feature like that when you shutting the system down.

The purpose of clearing the cache at shutdown is mainly a privacy thing - to delete traces of what you have been doing, not to force reload on the next browser start.  Hence, even though the idea of clearing the cache at the start again when the pref is set sound interesting for your case, I don't think it correlates with the original purpose.

We know that deleting the cache at the shutdown is somewhat pain and are trying to find a better approach to it.

Comment 2

[Honza Bambas (:mayhemer)]

I think we can confirm this problem exists, but fixing it may be part of a larger project.

Comment 3

[Johann Hofmann [:johannh]]

How are we clearing cache in case of "Never remember history" aka permanent private mode? For the "Clear History when Firefox closes" feature (Sanitizer.jsm) we have a concept of pending sanitizations which should take care of clearing the cache on startup if we failed to clear on shutdown...

Comment 4

[Honza Bambas (:mayhemer)]

(In reply to Johann Hofmann [:johannh] from comment #3)
> How are we clearing cache in case of "Never remember history" aka permanent
> private mode? 

permanent PB = only store in memory.  The cache is smaller then.  We special case the option for clear on shutdown only (with a custom history settings) to go to disk but clear.  This is a legacy approach and I honestly tend to put this in memory as one possible solution to all issues we have with cache clearing at shutdown.

> For the "Clear History when Firefox closes" feature
> (Sanitizer.jsm) we have a concept of pending sanitizations which should take
> care of clearing the cache on startup if we failed to clear on shutdown...

I'm not aware of any code like that, can you point me to that please?  I would be really interested in how you detect that "we failed to clear on shutdown" case, specifically for the cache.  Thanks.

Comment 5

[Nick Levinson]

But, as noted, the problem also occurs when quitting FF without doing anything to leave the Linux OS.

Interesting point about leaving the OS. Perhaps someone should propose that the OS add a hook for such purposes, usable by any app, unless the user is in a rush to reboot/shutdown.

The idea of forcing a reload next time FF starts was not for FF to remember a list of pages previously seen, but only to say that the cache should be empty and therefore that if I start FF and thereafter enter a URL then the browser should have no choice but to go out to the Internet to find the URL's content.

Clearing the cache on start is a good second choice if clearing on reboot/shutdown is too complicated absent an OS hook.

I corrected the Version menu; my fault.

Thanks.

Comment 6

[Michal Novotny [:michal]]

(In reply to Honza Bambas (:mayhemer) from comment #4)
> I'm not aware of any code like that, can you point me to that please?  I
> would be really interested in how you detect that "we failed to clear on
> shutdown" case, specifically for the cache.  Thanks.

We could simply create appropriate eviction info file on disk at the beginning of CacheFileIOManager::SyncRemoveAllCacheFiles() so that CacheFileContextEvictor starts evicting the cache on next startup.

Comment 7

[Michal Novotny [:michal]]

(In reply to Nick Levinson from comment #5)
> But, as noted, the problem also occurs when quitting FF without doing
> anything to leave the Linux OS.

Can you list the files you found in the cache after quitting FF?

Comment 8

[Nick Levinson]

Seeing inside the cache: How would I do that? I don't see an option to examine the cache. All FF tells me is "[y]our stored cookies, site data and cache are currently using 205 KB of disk space" and, if I click the Clear Data button, "Cached Web Content (205 KB)". The size seems consistent with the kind of Web pages I'd been visiting in my latest session/s each time and clearing results in zero with a reasonably sloped ascent thereafter.

Comment 9

[Michal Novotny [:michal]]

(In reply to Nick Levinson from comment #8)
> Seeing inside the cache: How would I do that? I don't see an option to
> examine the cache. All FF tells me is "[y]our stored cookies, site data and
> cache are currently using 205 KB of disk space" and, if I click the Clear
> Data button, "Cached Web Content (205 KB)". The size seems consistent with
> the kind of Web pages I'd been visiting in my latest session/s each time and
> clearing results in zero with a reasonably sloped ascent thereafter.

I thought you found the files on the disk in a cache directory, but if I understand correctly you see 4.5MB usage in about:preferences, right? Then I'm not sure how you know it's data in the cache and not other site data.

In any case, Johann made a good point in comment #3. Honza and I talked about "Clear history when Firefox closes" option. "Never remember history" is different and once activated Firefox restarts to permanent private browsing mode. It doesn't remember the history from that point because nothing is stored to a disk, but it doesn't clear any data when switching to this mode. If this is what you're seeing then this bug is invalid.

Comment 10

[Nick Levinson]

I found the cache content with experiments and it's on disk, not just in RAM, and it contains URLs and maybe more, but not the source code.

The first experiment was with a Wikipedia page I had not seen at least since installing the OS on a reformatted hard drive (if I ever saw it). No cold or warm reboot intervened. The cache was 3.3 MB or, per the Clear Data dialog, 3.5 MB. With the Wikipedia page open, I switched to Never Remember History and the cache went to 5.0 or 5.0 MB, respectively, and showed no Web page and Undo Close Tab was grayed out. I switched to Remember History; the cache was still at 5.0/5.0 MB; and the Wikipedia page immediately redisplayed unsolicited.

In my next experiment, FF was set to Never Remember History, this bug page was open, I shut down, and I kept the computer off for well over 15 seconds to be sure RAM was empty. After starting the computer again, opening FF, and changing to Remember History, this bug page redisplayed automatically, without my entering a URL anew.

In my third experiment, FF was set to Never Remember History and I turned networking (Wi-Fi) off. The only networking I have is Wi-Fi, no external adapter was plugged in, and the internal "Wi-Fi [was] Off", according to the desktop environment menu at top right (the desktop environment is Gnome 3.28.2). I shut down and I kept the computer off for well over 15 seconds, probably over 30 seconds. After starting the computer again, opening FF, and changing to Remember History, this bug page failed to redisplay but did show a trace of the URL in the tab itself when the tab was not selected. When I selected that tab, FF could not find the address on the Internet although the URL was still in the address bar. Then I turned Wi-Fi back on, switched to Never Remember History, forcing an FF restart, and then switched to Remember History, forcing another FF restart. The bug page redisplayed automatically, without my supplying the URL.

So now it doesn't matter just where on the hard drive the cache is stored. Before I found that it didn't matter, I had looked at dozens of folders in response to Comment 9, but I think I'd have to use a file editor and maybe a decompressor. Doing a file search on my entire system for files modified within the last day got me 26,000+ files and counting before I cancelled the search in progress. But, whatever the folder, the cache is being stored on disk through reboots and lack of networking and that contradicts the UI.

Cookies and site data totalled 0 bytes whenever I looked at the Clear Data dialog. That the cache is of Web pages and not other site data is stated by FF at about:preferences and the Clear Data dialog. Cookies cannot occupy a negative storage amount; therefore, site data must be zero; therefore, the cache is of Web page URLs.

The cache seems too large to contain only URLs but I don't know what else is in the cache. Two URLs (including one that's local within FF) apparently need 10 KB and it doesn't take much more to get to half a meg. Going from 461 KB to 499 KB took 7 or fewer addresses and an average of 5KB per address seems inexplicably large. If sector size is in question, the hard drive is 152.5 GB. I don't know if it's relevant that the cache size is sometimes different in about:preferences#privacy and in the Clear Data dialog, sometimes one being a little larger, sometimes the other being a little larger, sometimes the two being equal (but rounding is possible).

Clear History When Firefox Closes (Comment 3) is an option within Use Custom Settings For History. Since the option to clear is available within the custom settings, then the UI is implying that Clear History When Firefox Closes is part of Never Remember History.

That FF is remembering the history even while Never Remember History is in effect and afterwards goes against users' expectation of privacy. The expectation comes from FF's UI wording. Therefore, the solution is either to unrecoverably eradicate the history as soon as Never Remember History takes effect or to rephrase the option and the dialogs to tell users that FF does not erase history but hides it for recovery later.

I'm not a patcher at this level, but I wonder if the way to eradicate the history when Never Remember History is invoked in the menu is simply to give an automatic click to the Clear Data button, which the user wouldn't think of doing manually given the UI's promise.

If by the bug being invalid (if so) you mean a new bug has to be filed, I can do that. But, whatever the bug number, the contradiction between behavior and promise is large enough that the basic nature of the bug is valid. It may take a while to fix, but that doesn't invalidate it.

Comment 11

[Michal Novotny [:michal]]

All the use cases you described work as expected because history isn't cleared when "Never Remember History" mode is activated. I've updated bug description and component, because this is not a cache bug.

Comment 12

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

This bug has nothing specifically to do with bookmarks nor browsing history.

It's about a design decision that was taken time ago, where enabling permanent PB doesn't clear existing data.
I think it's a wontfix or invalid, but anyway it's far more general than just browsing history.

Comment 13

[Nick Levinson]

Sounds like a good refocusing toward the solution. While the cache was relevant due to what the UI said, I hadn't meant to critique the cache itself as defectively programmed but had only chosen the component that seemed closest, because the problem was somewhere with FF, and I'm glad that where is now being identified.

If a new thought is to not fix it, then at least rephrase "Never Remember History" to "Never Remember New History (Keep Old History)", but better than that is to clear the cache when Never Remember History is invoked, so privacy is protected and the promise is kept.

Bug 513421 (not by me) covered this with a response that Invalid applies because the behavior is what was programmed but ignoring the disconnect from users' expectations derived from the UI. While people might be busy with other bugs, the UI should communicate what FF will do. Sooner or later, if the programming cannot conform to the UI, the UI needs to conform to the programming.

I have not seen an explanation about why old history should be retained, just that it is. I don't know a purpose, unless it's forensic. Absent a purpose, it should go.

This likely is similar to bug 1208693. I forgot that I had seen this problem 3 years ago. I keep believing the UI.

Comment 14

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

(In reply to Nick Levinson from comment #13)
> I have not seen an explanation about why old history should be retained,
> just that it is. I don't know a purpose, unless it's forensic.

For Browsing History, at least, it would be a dataloss, and a serious one since the address bar relies a lot on history to weight things. Since a separate option to also remove old history exists, this gives the user full control over what happens.

For more volatile data like the cache, I could agree with you that there's no real dataloss and could probably be re-evaluated.

Comment 15

[Nick Levinson]

Okay, if we go ahead with getting rid of it.

The removal step as an option for users exists but is contradicted by the UI's promise and therefore usually would not be exercised because the user believes it's already done. It's like if I delete a file in a file manager then I don't have to open another file manager to be sure it's deleted, since I should be able to depend on the first file manager to have done what it promised it did. Besides, exercising the option is a bit complicated. I'd have to worry that tabs that are open when clearing history would themselves be remembered, because they'd be present after the hiistory was cleared and their URLs might be stored, so I'd have to remember to close all the tabs except an innocuous one (like about:preferences or example.com) before clearing history (closing all tabs closes FF so one tab has to stay open). You might explain that I don't have to worry about that but if I'm clearing history because I distrust the UI and you're not hanging out with me so I rely only on the UI, I'd have to do extra steps just to be sure. If the procedural information is in Help or this bug report or somewhere online, we'd use those only if the UI was obviously wrong or incomplete and even then not if we're already offline. Thus, the UI's importance.

On use of data for address bar weights, various places tell me that if I disable something then I'll lose some functionality I might want. Telling users that Never Remember History might disable another feature is a solution to that problem (if that problem applies to this context). Then, there's no reason for the UI to mislead and the user can choose knowledgeably.

Also relevant: bug 463607.

Comment 16

[Nick Levinson]

The time an OS takes to shutdown or reboot being too short for FF to clean itself up (see comment 1), in my case the OS being Fedora 28 Linux, was the subject of my question at a Fedora help forum (https://ask.fedoraproject.org/en/question/124255/rebootshutdown-with-app-running-and-how-f28-affects-apps-cleanup-at-that-moment/) for which no answer forthcame. So I submitted a feature request at https://bugzilla.redhat.com/show_bug.cgi?id=1611040 and anyone should feel free to comment there. Possibly, the Fedora or Red Hat people will say that it is an upstream issue, probably in the kernel, but I don't know enough and so decided to address F28 at this time.

Separate point on main theme: FF is gaining wider distribution, and among users it's likely that nongeeks already outnumber geeks and, thus, the UI has to conform to nongeeks' expectations, perceptions, abilities, and interests.

Comment 17

[Johann Hofmann [:johannh]]

(In reply to Honza Bambas (:mayhemer) from comment #4)
> I'm not aware of any code like that, can you point me to that please?  I
> would be really interested in how you detect that "we failed to clear on
> shutdown" case, specifically for the cache.  Thanks.

https://searchfox.org/mozilla-central/rev/6201a9e0067cf6af118c6a99ae9314b8ceb2c4d5/browser/modules/Sanitizer.jsm#48

(In reply to Marco Bonardo [::mak] from comment #12)
> This bug has nothing specifically to do with bookmarks nor browsing history.
> 
> It's about a design decision that was taken time ago, where enabling
> permanent PB doesn't clear existing data.
> I think it's a wontfix or invalid, but anyway it's far more general than
> just browsing history.

I agree with this. WONTFIX and we might want to look at bug 513421 whenever we revisit/improve clear history.

Comment 18

[Nick Levinson]

Is there a reason not to correct the UI (per, e.g., comment 15)? The menu item should be reworded to conform to the programmed preservation of old history on storage media after shutdown (e.g., comment 10), unless there's a compelling reason otherwise.

Comment 19

[Johann Hofmann [:johannh]]

(In reply to Nick Levinson from comment #18)
> Is there a reason not to correct the UI (per, e.g., comment 15)? The menu
> item should be reworded to conform to the programmed preservation of old
> history on storage media after shutdown (e.g., comment 10), unless there's a
> compelling reason otherwise.

Isn't that bug 513421? I was thinking of duping it but it wasn't exactly the same...