1467131 - Clean up bookmark annotation data once annotations have been removed

Status: NEW

(Toolkit :: Places, enhancement, P3)

Description

[Mark Banner (:standard8)]

Once annotations have been removed, we should clean up the moz_items_annos table in the places database.

Currently, some old annotations are being left there:

- Description (to allow users to export/save data, UI was removed in 62)
- Mobile folder (removed in 61).

When we do this depends on bug 1460577 being complete and we should consider the amount of time/releases (inc ESR) from when description field was stopped being supported.

Comment 1

[Mark Banner (:standard8)]

To be clear: we should also remove any remaining code that is exporting/backing up the remaining annotations.

Comment 2

[Dave]

Why not just rename it to Notes and bring back the edit box for it on the bookmarks properties dialog?
That way everyone can continue to use it like they have been for years, no data is lost, no useful feature is removed and you no longer have to auto-fill it with the web page description.

Comment 3

[Marco Bonardo [:mak]]

because it's a huge performance problem and we don't have resources to properly implement the feature you request, though it can be done properly by a WebExtension.

Comment 4

[Dave]

I thought maybe no longer auto filling it would speed things up.
What if you didn't integrate the notes field in api/search or anything and just made it show up when you right click the bookmark? Private Notes. i can't see how that would make anything slower. It has been a great feature for a very long time. I hate to see it go since it has increased my productivity a great deal and is something nice that Firefox had over Chrome. Now people will have to manually migrate all their notes to an extension that probably won't be as simple as a right click on the bookmark.

Comment 5

[Marco Bonardo [:mak]]

Everything you suggest has a development cost. As part of the Great or Dead effort, if we decide to keep a feature we must make it great for the user. And we just don't have the resources to do that in bookmarks at this time.
Anyway, if I should restart Places from scratch this is not something that I'd make part of bookmarks, I heard of people using this field to unsafely keep passwords, or more generic notes. There are (and can be) better solutions for both of those than reusing a description field.

A few alternatives to keep notes:
https://testpilot.firefox.com/experiments/notes/
https://addons.mozilla.org/firefox/search/?q=notes

Comment 6

[Jerry]

(In reply to Marco Bonardo [::mak] from comment #5)
> Everything you suggest has a development cost. As part of the Great or Dead
> effort, if we decide to keep a feature we must make it great for the user.
> And we just don't have the resources to do that in bookmarks at this time.
> Anyway, if I should restart Places from scratch this is not something that
> I'd make part of bookmarks, I heard of people using this field to unsafely
> keep passwords, or more generic notes. There are (and can be) better
> solutions for both of those than reusing a description field.
> 
> A few alternatives to keep notes:
> https://testpilot.firefox.com/experiments/notes/
> https://addons.mozilla.org/firefox/search/?q=notes

Marco, I agree with you "Everything you suggest has a development cost.", Just about Everything in life does have some type of cost associated with it.

However, as to "As part of the Great or Dead effort, if we decide to keep a feature we must make it great for the user." Here is where I respectfully don't share that philosophy. All good ideas evolve.

"Had I waited to build the perfect automobile, we would all still be on horses."
 -- Henry Ford

I might be wrong (it would not be the first time, certainly wont the last) I believe removing an end user editable data field without knowing what additional data is being captured by end users without consent much less warning may not be a good idea. Currently description for bookmarks is not great, but neither was the Henry Ford's Model T when it was first released, but look at how it has evolved over time from carriage to horseless-carriage and maybe within a decade driverless-carriage.

Respectfully yours, J

Comment 7

[Marco Bonardo [:mak]]

You made a good example, cars evolved, while this feature stayed broken and problematic and unused (by most) for years without nobody putting resources into it. Plus, this feature was used by a few, while causing issues to the most. The architecture wasn't built to affect only people using it, it was hurting *every* Firefox user.
I sympathize with your ideals, but I'm more than convinced there was no other way out.
Notifying earlier wouldn't have solved anything, but it would have caused double the dissatisfaction and annoyance on users (I have plenty of examples where users were annoyed when we announced a change, then after 6 weeks got furious when we actually made the change, even if they knew about it).
Additionally we should have notified every user, even those that never cared about the feature, that would have been annoyed. This is because the architecture (again) doesn't distinguish who cares from who doesn't.
Srsly, we evaluated every single possibility for this.

And fwiw, the data is preserved yet.

Comment 8

[Jerry]

Marco, thank you for your response, I was not expecting one.

You made excellent points, I'm just over protective of my data and don't like it when others fail to respect my property and/or work -sorry, I can't help how I feel. 

Nevertheless, I understand and respect your logic, I'm just struggling with the execution, but that's just me.

Regardless, I do appreciate the time you took to give me an reasonable explanation which is more than most people would have done, the mark of a true gentlemen and for that I thank you and all those who took part in the development of your character.


Respectfully yours, J

Comment 9

[Andreas M. Kirchwitz]

Shouldn't there at least some warning for the people? I've used annotations a lot, and all people should have a chance to save their data right in time before it's gone forever (although it's less useful now without a proper tool to work with that data).

By the way, auto-fill of annotations never has been a good idea and made the annotations less useful in fact. I never kept that when I added my own comments. Anyway, doesn't matter no more.

With "some warning" I mean some real warning, in fact. Something users cannot miss. Like a message when Firefox starts which must be dismissed manually. It's their data. Possibly a lot of data. Mozilla is going to - dramatically speaking - destroy it. That shouldn't be done silently or hidden deep down in some change log. People trusted Mozilla to keep their data safe.

Some users may be confused ("Hey, what's annotations? Never noticed it before. Still hate you for removing that feature even if I never used it!") but that's the way it is. Please think of all those people that actually used annotations. They will be thankful (well, not for the feature being removed but at least for not losing all their data).

Comment 10

[Marco Bonardo [:mak]]

(In reply to Andreas M. Kirchwitz from comment #9)
> Shouldn't there at least some warning for the people? I've used annotations
> a lot, and all people should have a chance to save their data right in time
> before it's gone forever (although it's less useful now without a proper
> tool to work with that data).

The data will be preserved a long enough time for people to notice misses and export.
An anyway I suppose you mean "descriptions", not "annotations". Annotations is a wider dataset that includes the description data.

> By the way, auto-fill of annotations never has been a good idea and made the
> annotations less useful in fact.

I agree, it's also one of the reasons for the removal.
> With "some warning" I mean some real warning, in fact. Something users
> cannot miss. Like a message when Firefox starts which must be dismissed
> manually. 

Unfortunately we can't due that, due to the fact descriptions were pre-filled with page data, we don't know which users modified them, so we should notify every user, but most of the users don't care and would be annoyed.

Comment 11

[Jeson]

(In reply to Dave from comment #2)
> Why not just rename it to Notes and bring back the edit box for it on the
> bookmarks properties dialog?
> That way everyone can continue to use it like they have been for years, no
> data is lost, no useful feature is removed and you no longer have to
> auto-fill it with the web page description.

Completely agree with this. Just rename Description to Notes, bring back edit box, and dont autofill it. Dont run search on it either. Existing Firefox users (like me) rely on this feature A LOT. Having Notes tied to ubiquitous bookmark is one GREAT simple feature. Making it DEAD is a big mistake.

Your motivation behind improving the performance is to keep existing users to use Firefox and attract new users. However, removing this heavily used feature gives existing users reason to switch to other browser. It causes Firefox to lose users. Isnt it a backward thinking ?

An idea to improve bookmark search performance WITHOUT removing the Description field:
In local memory, clone the bookmark without the description field. Run search on this clone instead.

BRING BACK THE DESCRIPTION FIELD

(Yes, I had to signup for new Bugzilla account just to voice this objection)

Comment 12

[[:philipp]]

will we take into account the 60esr -> 68esr migration for this bug in order to not surprise users on this channel with a dataloss situation after the next major esr upgrade?

Comment 13

[abpcsaregreat]

Please reconsider this change. The cost may be much higher than you think. You have seriously damaged our trust in you. We put our descriptions in with our bookmarks, where they can save us considerable time, and now the data won't be easily accessible (using another program wastes a lot of time, and is not a viable solution). This feature was saving us time, and now is going to waste a lot of our time. Our data (descriptions) will take so much longer to access, that it's pretty much the same thing as you have destroyed our data, and just recovering them will waste a lot of time, too.

You are damaging the Mozilla brand. Dumbing down Firefox to make it another Chrome will make Firefox redundant. It's a great program. Bookmark descriptions is a power user feature. If we wanted a simpleton program, we would just use Chrome.

Instead of removing this feature, why don't you fix it? I know there is a development cost, but there is a very large cost to screwing your users (which is what this is doing). This is how Microsoft operates. This is why we started avoiding Microsoft products in the first place.

On searching the notes, we don't need to search the notes (Firefox Bookmark search is weak anyway, because it doesn't show which local folder the found items are in). We just need a brief description attached to the link, so it is handy.

And we don't store passwords in there (anyone stupid enough to that, you won't be able to protect from themself anyway). We use the description field to store brief notes and conclusions about what we found at that website.

> if we decide to keep a feature we must make it great for the user.

This feature was fine. We don't need a Ferrari. We just need a place to jot down a few notes with the link, rather than trying to put them in the link title. This is one of the biggest problems with computers: you can't just jot down a little note on things, like you can on paper. And you guys fixed this for Firefox! That was great! And now you're taking it out again? This is a very useful feature.

Please reconsider this change, and leave us a little place to put our notes.

Comment 14

[Marco Bonardo [:mak]]

(In reply to [:philipp] from comment #12)
> will we take into account the 60esr -> 68esr migration for this bug in order
> to not surprise users on this channel with a dataloss situation after the
> next major esr upgrade?

This is a valid concern, and I think for that we'll have to keep the data around at least until Firefox 69. It's a bit unfortunate to have data duplication but I don't see a way out.

Comment 15

[Alex North-Keys]

I hate the change that lead to this "bug".  I used the annotations extensively - including through programs that autofiltered out more sensitive bookmarks using the annotations when making a publicly sharable version - and to the best of my knowledge, this is the only field in which full text could be maintained BY THE USER.  It might not be fair at this point, so I'm apologizing now, but:  What lead to the annotations being removed in the first place, this was a seriously useful feature with nothing else to move to if removed.  Was there so much development cost with keeping a couple of UI elements and a database field?

Comment 16

[Marco Bonardo [:mak]]

What you are describing is the Description field, thus bug 1402890, annotations is an internal API that was used to imeplement that, and it was a terrible API with huge performance and architectural problems.
We'd not remove a feature if it wouldn't cause problems to most users.
There are add-ons providing similar functionality, that is a win because then we can be more modular and not have a problematic feature affect every user: https://addons.mozilla.org/it/firefox/search/?platform=mac&q=notes

Comment 17

[Marco Bonardo [:mak]]

Sorry, fixed link without locale: https://addons.mozilla.org/firefox/search/?platform=mac&q=notes

Comment 18

[Zach_H]

Hi Marco - thanks for being so descriptive with reasonings as to why the "descriptions" field is being "dead" rather than "great" - I didn't realize it had such performance issue. I fully came here pitchfork in hand, but I didn't understand that this field is part of the multi-decade code cruft that had built up.

The reason I've enjoyed "descriptions" so much over the years is precisely because it's better than a regular "note" add-on in the sense that I get one "descriptions" field per bookmark, and therefore can put in relevant information about the website. Not passwords of course; that's what the passwords manager is for.

My question is; is there currently any talk in the core Mozilla developer's group as for a better-coded replacement for "descriptions" that can fulfill its same function in the core of the application? Is there an API available for developers to be able to extend each bookmark with a user-editable field? I understand that the descriptions field as we know it HAS to go away, and I've dutifully exported the bookmarks.html file as referenced so I haven't lost my data... but as of now, is there any thought for replicating its great functionality in the future, or if not by the core team, making it extensible to other developers? (The current "Bookmark Notes" extension, for example - I'm worried that the JSON its using to store its code might also be deprecated soon, causing us to be back at square 1 again). Thank you.

Comment 19

[Marco Bonardo [:mak]]

Until we're done refactoring the old code, it's unlikely we'll add new features to bookmarks, but then everything is possible.

Comment 20

[Patrick Seiter]

Please, please, please, don't delete the Description field internally!

I have a folder named "Citations" where I would bookmark every single page I found to be helpful in solving a problem, and I would blank the Description field and then use it to write in a synopsis of what the problem was, and how the page helped me find a solution. Essentially, I used the Description as a sort of diary for years before I started using Stack Overflow in 2013, but those Descriptions still represent years of my life and I would be heartbroken if they were all deleted. I still have a few other descriptions that I set from around 2016-2018, but those were for discrete use cases.

I just tried to write a diary description of a bookmark from Q1 2018 this past weekend, and when I realized the description field was disabled, I was brought here. I'm begging you, please don't delete this field internally or find a way to migrate it to the Places format.

due to the fact descriptions were pre-filled with page data, we don't know which users modified them, so we should notify every user

There should be a warning box in the bookmarks interface itself that says the Description field has been removed and why in the same place the Description field was. That way, the users like me who care about the description field will know that it's missing, and the ones who don't care will just ignore it. I've been very busy since Q1 2018 and haven't had the time to take a day to organize my bookmarks where I would have filled in their descriptions and sorted them into folders. I actually did panic upon seeing that I've been asleep on this for 10 months and might have already lost my data.

Comment 21

[Marco Bonardo [:mak]]

(In reply to Patrick Seiter from comment #20)

Please, please, please, don't delete the Description field internally!

We waited a year, and in the meanwhile all the Firefox versions kept that information and exported it in automatic bookmark backups, so anyone with a decent backup system will have this data forever. At this point in time I expect that almost everyone in need of that data already exported it somewhere else, because it was pretty obvious that it disappeared from the UI.

I'll start with the removal now, because we are past Firefox ESR 68, the description field was removed with a release note in Firefox 63. Firefox 69 will be released in 3 months, that is even additional time.

Comment 22

[Dave]

OK still really upset to see this feature go. It was hands down the best interface feature of firefox.
I hope you add a notes text area back to bookmark properties one day. It would be shameful if you waited for chrome to add it first.

Comment 23

[abpcsaregreat]

OK still really upset to see this feature go. It was hands down the best interface feature of firefox.

I agree completely. I have all the data backed up, but it is pretty much useless (as if it was destroyed) because it needs to be associated with the bookmarks when I'm using them in Firefox (I've gone back to the IE way of using really long link names). This feature was a great time saver, but it's wrecked now.

The removal of this feature has destroyed a lot of loyalty to Firefox. If Firefox wasn't still the default in Linux, we would have switched to a different browser because of the removal of this great feature, it is that annoying it's removal has wasted so much of our time.

In the future, I hope the devs will consider the efficiency (time saved or lost) to the user, before making changes. This kind of change, change, change mentality, that wastes the user's time, has destroyed much great software in the past.

Comment 24

[Andreas M. Kirchwitz]

(In reply to Marco Bonardo [::mak] from comment #21)

We waited a year, and in the meanwhile all the Firefox versions kept that information and exported it in automatic bookmark backups, so anyone with a decent backup system will have this data forever. At this point in time I expect that almost everyone in need of that data already exported it somewhere else, because it was pretty obvious that it disappeared from the UI.

Forever? Normal backup systems for end-users (and even in most companies) do not work that way, sooner or later old backups expire or get overwritten. Most Firefox users might even think of a bug in the bookmark editor and are still waiting for their data to come back eventually. (It's not that unusual for some bugs needing years to be fixed.)

I'll start with the removal now, because we are past Firefox ESR 68, the description field was removed with a release note in Firefox 63. Firefox 69 will be released in 3 months, that is even additional time.

Those extended bookmarks descriptions were a cool and unique feature of Firefox. Anyway, you're free to remove any feature you want, that's totally your decision, it's your software, and users don't not have any rights here.

However, silently removing user-entered data (user property) is fundamentally wrong. No trustworthy software should ever do this. This is an absolute no-go.

In Germany, that might even violate laws (e.g., StGB § 303a, 303b).

Users must be warned and explicitly agree when their data is going to be deleted. This data is their property, not the property of Firefox/Mozilla. It is no valid argument that this might confuse some users and could put additional workload on support teams. It is not sufficient to hide that in release notes that do not pop up during Firefox upgrades and also are pretty hard to find on Mozilla's web site even if someone wanted to read them.

Firefox is intentionally destroying property of their users without their consent. That is wrong.

Comment 25

[Patrick Seiter]

(In reply to Marco Bonardo [::mak] from comment #21)

We waited a year, and in the meanwhile all the Firefox versions kept that information and exported it in automatic bookmark backups, so anyone with a decent backup system will have this data forever. At this point in time I expect that almost everyone in need of that data already exported it somewhere else, because it was pretty obvious that it disappeared from the UI.

I backup my entire Firefox profile once a week. I just backed up my work profile and I still see descriptions in the bookmarks. I have several other profiles for different use cases. I'll very urgently go through all my Firefox profiles and try extracting my personal descriptions from my bookmarks. Thanks for the heads up!

Even with this change, I'm happy that Firefox bookmarks management is still objectively superior to that of Google Chrome's.

Comment 26

[Dave]

Is there an addon that can save store multiline text on a bookmark's right click properties yet?

Comment 27

[Dave]

It is really killing my productivity not having that feature anymore.

Comment 28

[Marco Bonardo [:mak]]

also remove most remaining uses of moz_items_annos, we may want to keep the table and some migration around though
https://searchfox.org/mozilla-central/search?q=moz_items_annos&path=&case=false&regexp=false

Comment 29

[Lata]

Hii! I have solved a few good first bug and interested in solving this bug. Can I get some guidance how to proceed?