Closed Bug 1522611 Opened 5 years ago Closed 5 years ago

Behavior involving the "bookmarkbackups" folder and deleting bookmarks is unexpected for a user wanting to delete data permanently

Categories

(Toolkit :: Places, defect)

Product:
Toolkit
Component:
Places
Version:
64 Branch
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED DUPLICATE of bug 1475582

People

(Reporter: mapimuz, Unassigned)

Details

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:64.0) Gecko/20100101 Firefox/64.0

Steps to reproduce:

Both examples can be reproduced using a new profile.
Note that clearing recent history in these example steps includes all of the checkboxes, and the "everything" time period.
Version 64.0.2 (64-bit) on Windows, but I imagine the behavior would probably be the same on Linux and BSD.

#Example 1 - Automated bookmark backups

Use a fresh browser profile because this will delete data.

  1. Go to "flightradar24.com", let the page load completely then bookmark it.

  2. Open up the bookmarks library using "Show All Bookmarks" or Ctrl + Shift + B

    Make a backup using Import and Backup -> Backup.

    Save it in "C:\Users\<username>\AppData\Roaming\Mozilla\Firefox\Profiles\<profile>\bookmarkbackups", to mimic the automatic backups that happen daily.

  3. Close the tab. Clear your recent history and delete the bookmark in that specific order.

    Close Firefox completely.

    Delete "places.sqlite" from "C:\Users\<username>\AppData\Roaming\Mozilla\Firefox\Profiles\<profile>".

  4. When you reopen Firefox, wait a few seconds, and type "f", "flightradar24.com" will come up.

#Example 2 - Clearing history and deleting a bookmark

  1. Go to "flightradar24.com", let the page load completely, bookmark it.

  2. Close the tab, then clear your recent history and delete the bookmark in that specific order. Close Firefox.

  3. When you reopen Firefox and type "f", "flightradar24.com" will come up.

Actual results:

#Example 1 - Automated bookmark backups

When running, Firefox will automatically make backups of your bookmarks daily in

C:\Users\<username>\AppData\Roaming\Mozilla\Firefox\Profiles\<profile>\bookmarkbackups

with .jsonlz4 files.

When you delete places.sqlite then reopen Firefox, places.sqlite is repopulated from the automated backup.

The domain names from the deleted bookmarks will pop up in the address bar when typing in letters, because of the "Options-> Privacy & Security -> Address Bar" default settings and the bookmarks being readded to places.sqlite.

#Example 2 - Clearing history and deleting a bookmark

A deleted bookmark is only completely deleted if you clear recent history for the correct time period afterwards.

This is because traces of the domain name are still left in places.sqlite.

Expected results:

#Example 1 - Automated bookmark backups

Not all users are aware that Firefox automatically makes bookmark backups. Nor are all users aware that these backups will persistently come back after "places.sqlite" is forcefully deleted. A user attempting to clean their profile completely may be surprised to see that they can't get rid of this data, without knowledge of the "bookmarkbackups" folder.

Deleting a bookmark should completely purge traces of that unwanted data.

#Example 2 - Clearing history and deleting a bookmark

When a user clears their history and deletes a bookmark, seeing that Firefox doesn't use a Trash folder, the expectation would be that the bookmark is completely deleted.

Seeing that the history was cleared and the bookmark deleted, the browser should forget about that url completely.

The user does not expect that the order of deleting things matters, or that a deleted bookmark would pop up in the address bar.

#Conclusion

To fix both examples, you have to delete the bookmark first then clear the recent history.
If done in that order initially, it wouldn't be a problem. This is unexpected behavior and an unpleasant surprise.

Deleting bookmarks that are no longer in history should remove traces of the domain name from all possible locations:

bookmarkbackups folder

cookies.sqlite

favicons.sqlite

permissions.sqlite

places.sqlite

sessionstore.jsonlz4

And any other files that could contain the domain names.

Also your browser extensions:

browser-extension-data folder

Any "storage\default\moz-extension\*" folders, for example Ublock Origin will keep metadata in its "idb" folder.

I understand the configuration and behavior of some extensions may be out of scope or out of your control, but it's worth mentioning.

Any temporary files Firefox uses while running (assuming they were not removed due to some error):

cookies.sqlite-shm

cookies.sqlite-wal

favicons.sqlite-shm

favicons.sqlite-wal

places.sqlite-shm

places.sqlite-wal

Whatever it is that "clear recent history" does to flush out places.sqlite is what should be done when deleting a bookmark no longer in history, in addition to removing it from the above locations.

Further, when a user deletes bookmarks and clears recent history, a string search from the command line inside of the profile folder should turn up clean:

Windows

findstr /msi "flightradar24" *

Unix-like (Linux and BSD)

grep -ri "flightradar24" *
Component: Untriaged → Bookmarks & History

I should expand upon the expected results and conclusion:

Deleting a bookmark should completely purge traces of that unwanted data.

If it's not in history, places.sqlite should not have it upon deletion. And deleting places.sqlite then starting Firefox should not automatically repopulate the file from the backup. This allows the deleted data not to pop up unexpectedly in the address bar while typing.

Of course a deleted bookmark still in history would remain in history until cleared by the user, so it wouldn't be gone from places.sqlite yet.

I realize that for many users automated backups are useful and could be a lifesaver if all their bookmarks are deleted. Those users can always manually restore their bookmarks, and a web search for "how to restore Firefox bookmarks" will give helpful resources. But the automatic restore feature should not trample upon the users attempting to delete data permanently by deleting places.sqlite. This can quickly become frustrating in that scenario, when the user can't figure out why the data they want to get rid of keeps coming back from the dead and popping up in the address bar as the first suggestion for a specific letter.

Deleting files in the profile is completely unsupported and that the backup will be restored is the expected behavior !
The automatic backup and restoring helps many users in case something happens with their data on the file system level.

People deleting files in the profile are the absolute minority, maybe 0,1% or less and people like you can always google and figure that out yourself.
Why should all the users (99,9%+) who will benefit from the automatic backup start to google around to find a solution ?
I bet that 50% of those users are not even smart enough to find the profile folder.

Why don't you use the private browsing feature that is made exactly for that what you want ?

Clearing history is designed to clear the browsing history. It is assumed that if you've bookmarked something, you did want to remember it - at least at the time.

As has already been explained, the bookmark backup system is there in case of something happening at the file system level, but also so that they can recover if they did accidentally delete something. If they wish, they can set the preference "browser.bookmarks.max_backups" to 0 which will disable backups completely.

The places database is regularly vacuumed, but we're not going to do that on every change - that would be a too expensive operation. Likewise removing or clearing the temporary files would mean forcing a flush or closing and reopening the database - also too expensive.

I can understand that you may wish absolutely all data to be removed, however that is hard to do when we're trying to support the use cases for the majority of users. As Matthias mentioned, the private browsing feature sounds more like what you want.

You may want to think again from the start about detailing why you want absolutely all traces removed - and then start by figuring out what would meet your requirements, e.g. can you secure the computer/data? if not, why? if you still can't, then rather than just removing one file, why not the whole profile? etc.

Component: Bookmarks & History → Data Sanitization
Product: Firefox → Toolkit

(In reply to Matthias Versen [:Matti] from comment #2)

Deleting files in the profile is completely unsupported and that the backup will be restored is the expected behavior !
The automatic backup and restoring helps many users in case something happens with their data on the file system level.

What I was saying is that automatic backups should not be automatically restored. I understand automatic backups have to happen to help the rest of the users. But somebody trying to clean a profile will be thrown off by the same url popping back in the address bar over and over, without knowledge of the "bookmarkbackups" behavior.

People deleting files in the profile are the absolute minority, maybe 0,1% or less and people like you can always google and figure that out yourself.

Clearing your history and deleting your bookmarks should be intuitive and should remove that url from places.sqlite, or at the very least not pop up in the address bar. The fact that it's still popping up unless the actions are done in a specific order is unexpected.

Why should all the users (99,9%+) who will benefit from the automatic backup start to google around to find a solution ? I bet that 50% of those users are not even smart enough to find the profile folder.

I'm referring to the act of "Import & Backup -> Restore", which contains a list of known backups.

Why don't you use the private browsing feature that is made exactly for that what you want ?

You're misunderstanding, the case is having a profile that you use normally and bookmarking items, then needing to clean that profile for another purpose (presentation, letting someone else use your laptop, etc.). Having to create a new profile would be unreasonable, the clear history and delete bookmark expected behavior should work in any order.

(In reply to Mark Banner (:standard8) from comment #3)

Clearing history is designed to clear the browsing history. It is assumed that if you've bookmarked something, you did want to remember it - at least at the time.

Yes, but when a user clears all history and deletes a bookmark and closes the browser, moz_bookmarks_deleted and any relevant table inside places.sqlite should no longer contain the url. I shouldn't have to do it in a specific order because this is unexpected and unintuitive.

As has already been explained, the bookmark backup system is there in case of something happening at the file system level, but also so that they can recover if they did accidentally delete something. If they wish, they can set the preference "browser.bookmarks.max_backups" to 0 which will disable backups completely.

Yes, I clarified my understanding of the need for the automatic backups in my second comment. The first comment was a little extreme. The automatic backups should not restore automatically, but manually by "Import & Backup -> Restore", which contains a list of known backups. I think expecting somebody wanting to delete data to know about setting an about:config variable is unreasonable. The deleted bookmark should not pop up in the address bar when it's no longer in history.

The places database is regularly vacuumed, but we're not going to do that on every change - that would be a too expensive operation. Likewise removing or clearing the temporary files would mean forcing a flush or closing and reopening the database - also too expensive.

Once the browser closes and the place is no longer in bookmarks or history, it should be flushed. The temporary files was part of my first comment which included too many files, the second comment was the clarification.

I can understand that you may wish absolutely all data to be removed, however that is hard to do when we're trying to support the use cases for the majority of users. As Matthias mentioned, the private browsing feature sounds more like what you want.

Thank you for understanding the perspective, but I don't think flushing places.sqlite on closing or having a user willingly restore a backup by clicking two buttons is unreasonable. The private browsing feature is not what I want, what I want is a reasonable way of clearing data from a profile, where if I clear all history for everything and delete bookmarks in any order, the urls will not pop back up in the address bar.

You may want to think again from the start about detailing why you want absolutely all traces removed - and then start by figuring out what would meet your requirements, e.g. can you secure the computer/data? if not, why? if you still can't, then rather than just removing one file, why not the whole profile? etc.

Upon further clarification from my other comments I no longer want absolutely all traces removed, the first comment was a little extreme. My second, third, and this comment clarify my requirements. The whole point is a user should be able to easily delete data from their profile without it coming back to haunt them through the address bar when using their computer for other situations or when somebody else is using their computer.

It should be able to be done from the browser itself, without having to open up the profile folder or start Firefox with the -P switch and delete the profile. New profiles would be unreasonable because a user should be able to keep all their plugins, themes, and settings and still be able to delete data. And not all would know about or remember the -P switch.

And again, if places.sqlite is deleted it shouldn't rebuild things you tried to delete automatically. For most users, pressing "Import and Backup -> Restore -> X" is trivial. However, the deleting places.sqlite part would never have been necessary if clearing recent history then deleting a bookmark and closing the browser worked as expected.

(In reply to mapimuz from comment #5)

Yes, but when a user clears all history and deletes a bookmark and closes the browser, moz_bookmarks_deleted and any relevant table inside places.sqlite should no longer contain the url. I shouldn't have to do it in a specific order because this is unexpected and unintuitive.

Slight correction, the url is still in the moz_origins and moz_places tables after clearing history and deleting the bookmark in that order, and closing the browser. Is there any good reason for this? Can the necessary operations on places.sqlite happen upon closing the browser?

Hey, I don't understand how this is a data sanitization issue. As you said, this component doesn't really concern itself with bookmarks and I don't see how clear history etc. could do something different here. To be honest I haven't followed this discussion very thoroughly but if there's something that Places can improve we should probably move the bug back to that component, otherwise I'll need to resolve it as WONTFIX or INVALID on my end.

Thanks!

Flags: needinfo?(standard8)

Hi Johann, I thought initially it was with sanitization, but it does look a little more like places now. In any case, I'll discuss it with Marco soon and figure out the way forward.

Component: Data Sanitization → Places
Flags: needinfo?(standard8)

(In reply to Johann Hofmann [:johannh] from comment #7)

Hey, I don't understand how this is a data sanitization issue. As you said, this component doesn't really concern itself with bookmarks

Where did I say that (unless you mean someone else)? The behavior of deleting bookmarks, automatic restore, and remanence of deleted data in places.sqlite is a crucial part and has to do with both.

and I don't see how clear history etc. could do something different here. To be honest I haven't followed this discussion very thoroughly but if there's something that Places can improve we should probably move the bug back to that component, otherwise I'll need to resolve it as WONTFIX or INVALID on my end.

Thanks!

  1. The url should be removed from the moz_origins and moz_places tables in places.sqlite after clearing history and deleting the bookmark in that order. Whatever it is that clear history does to properly flush the urls out of places.sqlite, is an operation that should take place once the user closes the browser if a place is no longer in bookmarks or history.

  2. places.sqlite should not rebuild itself from backups automatically. For a user trying to delete data this is unexpected behavior as information they thought they were getting rid of comes back to haunt them by popping up in the address bar once they start typing a letter. For most other users, doing "Import and Backup -> Restore" is trivial and contains a list of backups to choose from. However, I did mention deleting files would never have been necessary if clearing history and deleting bookmarks worked in any order.

My main priority would be for clearing history and deleting bookmarks to work in any order, because if you delete bookmarks last, the data is still in the tables. Deleting first then clearing history is what removes it from places.sqlite. But it's unreasonable to expect a user to have to do this in a specific order for unwanted urls not to pop up in the address bar while typing.

If running an operation on places.sqlite to flush the tables of places that are not in bookmarks or history upon closing the browser is possible, this might be the right place.

(In reply to mapimuz from comment #9)

Where did I say that (unless you mean someone else)? The behavior of deleting bookmarks, automatic restore, and remanence of deleted data in places.sqlite is a crucial part and has to do with both.

He didn't meant you because that was a discussion between 2 developers about the right component.

I'm a user like you but let me respond to your 2 points:

  1. The browser shutdown should not be delayed. A vacuum of the db on shutdown would be really, really bad.
  2. places.sqlite should of course be rebuild from backups automatically if the file just disappears. This is one of the best features added in the past to solve lost bookmark events from users who don't even find the manual bookmark backup feature.

(In reply to Matthias Versen [:Matti] from comment #10)

(In reply to mapimuz from comment #9)

Where did I say that (unless you mean someone else)? The behavior of deleting bookmarks, automatic restore, and remanence of deleted data in places.sqlite is a crucial part and has to do with both.

He didn't meant you because that was a discussion between 2 developers about the right component.

I'm a user like you but let me respond to your 2 points:

  1. The browser shutdown should not be delayed. A vacuum of the db on shutdown would be really, really bad.
  2. places.sqlite should of course be rebuild from backups automatically if the file just disappears. This is one of the best features added in the past to solve lost bookmark events from users who don't even find the manual bookmark backup feature.

Well for #1, there must be some way to do the necessary operation. Browsers like Chrome and Opera get it right. If you clear history and delete bookmark in any order, it behaves as expected.

For #2, I don't think having a user click "Import and Backup -> Restore -> Monday, January 28, 2019" is an undue burden. There would be many resources online to help them do this, and it's only clicking two buttons. I understand you disagreed with that, but it is a dead simple manual operation. I think "of course be rebuild from backups" is too strong of a conclusion, the user's consent should be factored in, Firefox shouldn't assume that's what I want do.

Maybe there could be a dialogue box saying that "Would you like to restore from backup?" whenever places.sqlite disappears?
Or as another idea, a troubleshooting button to "hard reset" a profile while keeping plugins, themes, and settings. Although that could be abused by obnoxious people trying to destroy someone else's data.
Or maybe, a trash folder, and once deleted from trash it performs the necessary operation?

I really don't quite understand why keeping track of certain urls (not in history, not in bookmarks), then removing those from places.sqlite is an expensive operation. Or why other browsers can do it, but somehow this is a stumbling block. I know that sounds harsh, but an explanation of the limitations specific to Firefox would be appreciated.

There are a few things to clarify and investigate.

  1. related to bookmark backups, we will never clean them up. We think the benefit of having them LARGELY overweights the possibility for someone to bookmark an unwanted page and not figure that out before the backup is created. Doing the opposite would be quite expensive, we'd have to repackaged every backup. On the other side there is a simple option in about:config named browser.bookmarks.max_backups that you can set to 0, and the problem is solved. Your privacy concern is not common among our users, even if most of our users are quite privacy aware, bookmarks are not usually considered part of the sensitive data.

  2. moz_origin should only contain origins if they are present in moz_places. If this is not the case, then it's a bug, unless you used a non supported external tool like CrapCleaner (or similar) to remove Firefox data. Those tools are likely to break data coherence and make privacy worse instead of better.

  3. Removing bookmarks doesn't immediately remove the moz_places entries, instead those stay there as orphans for a few minutes (usually less then 10 minutes). This is due to architectural problems from the past where everything was on the main-thread and blocking the UI. Now we solved most of those architectural problems and we can revert to a more user-understandable behavior of removing a page when a bookmark is removed, without waiting minutes. This is something I was already planning to do in bug 1475582.

  4. there are other cases where origins are not removed, for example I think we don't clean them from SiteSecurityServiceState.txt. Again there it's matter of balancing 2 weights: strict-privacy VS security. Those decisions are not trivial and may not always pend towards data removal.

Please let me know if you found other cases I didn't talk about here, so far the plan is to resolve point 3, while the others are wontfix. If there aren't other cases to investigate, I'd suggest duping this to bug 1475582.

(In reply to Marco Bonardo [::mak] from comment #12)

There are a few things to clarify and investigate.

  1. related to bookmark backups, we will never clean them up. We think the benefit of having them LARGELY overweights the possibility for someone to bookmark an unwanted page and not figure that out before the backup is created. Doing the opposite would be quite expensive, we'd have to repackaged every backup. On the other side there is a simple option in about:config named browser.bookmarks.max_backups that you can set to 0, and the problem is solved. Your privacy concern is not common among our users, even if most of our users are quite privacy aware, bookmarks are not usually considered part of the sensitive data.

The main issue was the automatic restoration without the user's consent. And the scenario was wanted bookmarks and then a need to "clean" the profile without losing plugins, themes, or settings. Like tidying up your browser in the case where you're using it for something else (presentation, someone using your computer, so on).

It's unreasonable to expect a user to adjust about:config variables when the deleting bookmarks and clearing history behavior should work in any order. I imagine there's many other people like me who are surprised to find bookmarks they thought were deleted popping back up in the address bar when typing the first letter.

  1. moz_origin should only contain origins if they are present in moz_places. If this is not the case, then it's a bug, unless you used a non supported external tool like CrapCleaner (or similar) to remove Firefox data. Those tools are likely to break data coherence and make privacy worse instead of better.

  2. Removing bookmarks doesn't immediately remove the moz_places entries, instead those stay there as orphans for a few minutes (usually less then 10 minutes). This is due to architectural problems from the past where everything was on the main-thread and blocking the UI. Now we solved most of those architectural problems and we can revert to a more user-understandable behavior of removing a page when a bookmark is removed, without waiting minutes. This is something I was already planning to do in bug 1475582.

I didn't use any external tools. If it's no longer in history, it shouldn't be in moz_places or moz_origins. At the very least, it shouldn't pop up in the address bar when typing as a suggestion. If orphans can't be removed from tables upon closing the browser, the very minimal solution would be filtering it out of the suggestions for the address bar.

Based on what you describe and granted that the other bug is fixed, deleting a bookmark would remove it from moz_places and moz_origins, all by the time the user closes the browser? If that's the case then it would solve the bug: as deleting and clearing history would work in any order, and deleted bookmarks would no longer pop up in the address bar while typing.

The whole issue with automatic restoration and deleting places.sqlite was secondary to the main issue of places.sqlite behavior with orphans. Meaning it's unnecessary to do that if the regular deleting bookmarks behavior works as expected. Although I still disagree with the automatic restoration itself, there should be a dialogue box or some way of getting the user's consent "Yes, I want you to restore places.sqlite from backup"

I just read the other bug 1475582

We currently run 2 kind of expiration, one runs always and removes orphans, another one runs only when the database is over the given limit.

I'd like to remove the former and make the latter run less often (every 15 minutes or so).

This says you're going to remove the one that "runs always and removes orphans" and make the "given limit" one run less often (every 15 minutes).
Unless I'm misunderstanding this kind of contradicts your previous statement in this thread:

... those stay there as orphans for a few minutes (usually less then 10 minutes)... we can revert to a more user-understandable behavior of removing a page when a bookmark is removed, without waiting minutes.

In order to do that you would need something that runs when a bookmark is deleted, not many minutes later.

Flags: needinfo?(mak77)

(In reply to mapimuz from comment #13)

The main issue was the automatic restoration without the user's consent.

This is something we WANT, so it is not going to change.

It's unreasonable to expect a user to adjust about:config variables when the deleting bookmarks and clearing history behavior should work in any order. I imagine there's many other people like me who are surprised to find bookmarks they thought were deleted popping back up in the address bar when typing the first letter.

I don't see the relation between removals and the about:config variable I suggested for backups, you're mixing up unrelated issues.
There won't be any changes to bookmark backups. We are happy with the status quo, we think it satisfies the needs for the majority of users. No doubt you can be unhappy about it, we can't satisfy everyone.

Based on what you describe and granted that the other bug is fixed, deleting a bookmark would remove it from moz_places and moz_origins, all by the time the user closes the browser? If that's the case then it would solve the bug: as deleting and clearing history would work in any order, and deleted bookmarks would no longer pop up in the address bar while typing.

Yes, removing a bookmark would immediately cleanup any orphan entry.

(In reply to mapimuz from comment #14)

Unless I'm misunderstanding this kind of contradicts your previous statement in this thread:

It's possible I didn't express well in the other bug, the scope there is to remove orphans immediately. I'll see if it's worth adding a clarification there.

Status: UNCONFIRMED → RESOLVED
Closed: 5 years ago
Flags: needinfo?(mak77)
Resolution: --- → DUPLICATE
You need to log in before you can comment on or make changes to this bug.