Open Bug 1225916 Opened 9 years ago Updated 5 months ago

Allow reading and writing bookmark tags

Categories

(WebExtensions :: General, enhancement, P3)

Product:
WebExtensions
Component:
General
Type:
enhancement

Tracking

(Not tracked)

People

(Reporter: wbamberg, Unassigned)

References

(Depends on 2 open bugs)

Details

(Whiteboard: [bookmarks][design-decision-approved] triaged) 

The Chrome bookmarks API[1] doesn't let you query by tags. The Add-on SDK bookmarks API does[2]. It would be nice if the WebExtensions API did, too.

[1] https://developer.chrome.com/extensions/bookmarks#method-search
[2] https://developer.mozilla.org/en-US/Add-ons/SDK/Low-Level_APIs/places_bookmarks#search%28queries_options%29
I'm tracking implementation of the search functionality in Bug 1225743. I can see if it's viable to get tag support implemented in that bug or leave this open as a separate bug. So I'd put this on blocked until 1225743 is landed. :)
Depends on: 1225743
+1 about searching by tag! It's the one thing that makes Firefox bookmarks superior than Chrome's ones. Please please please don't drop this feature.
so the chrome.bookmarks.search that has just landed does not support searching by tag, but I'm pretty sure it can be amended to support this in the future
Mentor: kmaglione+bmo
Whiteboard: [bookmarks][berlin]
Whiteboard: [bookmarks][berlin] → [bookmarks][berlin][good first bug]
Hi,

If this bug is still open, I would like to work on this bug.
This would be my first bug for mozilla.

I can see that bug 1225743 is verified and fixed so I hope this bug is ready to get implemented.

Please help me to get started.
Flags: needinfo?(kmaglione+bmo)
Hi Vikram, thanks for contributing!

This bug is quite ambitious for a good first bug, I don't think it should be tagged as one, sorry. However, we can still try to get it solved. There are two ways I can come up how to do this:

Either we use the tagging service (https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Reference/Interface/nsITaggingService) to get a list of all urls for the tags in https://dxr.mozilla.org/mozilla-central/source/browser/components/extensions/ext-bookmarks.js#115 and do a search for all those URLS.

Or we change the query implementation in https://dxr.mozilla.org/mozilla-central/source/toolkit/components/places/Bookmarks.jsm#913 to also look in the tag folder and get the corresponding bookmarks as well. That sounds like some pretty complicated SQL query, though. (See https://developer.mozilla.org/en-US/docs/Mozilla/Tech/Places/Database).

Marco, maybe you have an opinion on what's the best thing to do here? :)
Mentor: jhofmann
Flags: needinfo?(mak77)
Whiteboard: [bookmarks][berlin][good first bug] → [bookmarks]
Also, if this sounds intimidating then I suggest Bug 1265317 which should be much easier to solve :)
Thanks Johann for pointers, I have started working on Bug 1265317.

Truly, this bug will be challenging for me but this will be a good learning. I would like to work on this and take it to completion. 

I want to visualize this issue. Could you help me to see functionally the meaning of 'Bookmark API should allow searching by tag', because I am able to search bookmark by tag in my firefox bookmark sidebar.

Trying to fill gaps.
Flags: needinfo?(jhofmann)
(In reply to Johann Hofmann [:johannh] from comment #5)
> Or we change the query implementation in
> https://dxr.mozilla.org/mozilla-central/source/toolkit/components/places/
> Bookmarks.jsm#913 to also look in the tag folder and get the corresponding
> bookmarks as well. That sounds like some pretty complicated SQL query,
> though. (See
> https://developer.mozilla.org/en-US/docs/Mozilla/Tech/Places/Database).

AUTOCOMPLETE_MATCH can match on tags, if you pass the tags list as the 4th param (IIRC). And you-re already using that...

The problem is how you want this API to look like... There is a problem I see here.
Tags currently are bound to bookmarks, but that's not really mandatory, through the tagging API one could just tag an unbookmarked url. So tags could also be seen as an alternative to bookmarking...
We have never specified a "correct" behavior here, cause tags have always been an half-done feature that lost traction along the way, everyone excited at the beginning, then no one cared anymore.

So, if there would be some sort of specification around those, first of all it should define whether we tag urls or bookmarks. And based on that the API will look very different, and it will bound tags to a certain behavior "forever".

As things stand now, I'd not bound a specification around something we never settled down.
Flags: needinfo?(mak77)
> As things stand now, I'd not bound a specification around something we never
> settled down.

How formal a specification are you hoping for and who are the people to help figure that out?
Whiteboard: [bookmarks] → [bookmarks][design-decision-needed]
Not extremely formal, just some consensus across people responsible for webextension API across browsers, maybe we could figure if any other browsers is interested in tagging support, and what are their ideas.
Otherwise, we could just take our own decisions, after some discussions around pros and cons.
But I can't exclude tags could end up on the "wrong" side of a great or dead decision.
(In reply to Marco Bonardo [::mak] from comment #10)
> Not extremely formal, just some consensus across people responsible for
> webextension API across browsers, maybe we could figure if any other
> browsers is interested in tagging support, and what are their ideas.
> Otherwise, we could just take our own decisions, after some discussions
> around pros and cons.
> But I can't exclude tags could end up on the "wrong" side of a great or dead
> decision.

It seems that the bookmarks API is not a part of the new working group for now. But I agree 100%, we should rather leave this stalled than shoehorn some solution that will not conform to future ideas but we would have to support in the future.

My idea was to include tags in the part that gets queried through the "query" parameter and add a new "tag" field to the search options, like "url" and "title".

I think removing these if tags gets killed would not badly break any extension except those that fundamentally rely on tags for some reason. One way to make sure that doesn't happen is to not offer the possibility to create tags via the WebExtension API.

> Truly, this bug will be challenging for me but this will be a good learning. I would like to work on this and take it to completion. 

If there is some consensus on what the API should look like you're welcome to try! But since we're just figuring it out for now I'd recommend not taking this bug.
Flags: needinfo?(jhofmann)
Hi Marco,

(In reply to Marco Bonardo [::mak] from comment #8)
> Tags currently are bound to bookmarks, but that's not really mandatory,
> through the tagging API one could just tag an unbookmarked url. So tags
> could also be seen as an alternative to bookmarking...
> We have never specified a "correct" behavior here, cause tags have always
> been an half-done feature that lost traction along the way, everyone excited
> at the beginning, then no one cared anymore.

This is probably not the right place to talk about it, but I just wanted to say my opinion.

Organize bookmarks with folders has never worked for me, as the URL content can be related to two different concepts, say A and B.
If you place the link in the A folder, when you'll be searching for that link you might associate it with the concept B and not A, because if you're current focus is (related to) B you tend to remember the B association more easily than the A association.
The result is that you try to search by keywords in the Awesome Bar, hoping the ones you can remember are in the URL.
If they're not, you're forced to rely on a search engine to find a link you already have in your machine, which is not logical (read: absurd).

On the other hand, if you're able to create those two relationships A <- link -> B, you'll have a more flexible mechanism to find things again.
Tagging a URL is also way faster because, instead of searching for a folder nested inside many others, you just press the shortcut to bookmark and fill the tags fields in, takes a second or two.
After I switched to tags I almost never missed a bookmark, meaning I'm less dependent on a search engine to find things I need.

Also, using a search engine to find a link you already have is good for the search engine business model, not for yours, and quite importantly it's not "green" at all as your searches keep pumping heat out of their CPUs: we keep doing them over and over, and we're millions doing that. 
After all Chrome is a Google product, which has little interest in letting you find things by yourself :/

So, why leave this super important feature behind and unfinished?

Is there a public place where you guys discuss these APIs?

Thanks!
You're right, this is not the right place. Quickly, the main reason is that we don't have infinite resources, the few we have must be dedicated to things that can help most people and, without doubts, current tagging experience is not one of those. To make it viable for most users you should completely redesign the UI interaction, that is expensive compared to the benefit. I can't tell if and when it will happen, but now it's not a priority for sure.
Whiteboard: [bookmarks][design-decision-needed] → [bookmarks][design-decision-needed] triaged
Clearing needinfo, since we're not sure if we want to implement this.
Flags: needinfo?(kmaglione+bmo)
Summary: bookmarks API should allow searching by tag → Allow reading and writing bookmark tags
Kris, thank you for changing the title/summary of this bug.  Now I agree that my new tags Bug 1276816 duplicates it :)

From searching the source code, it appears that the starting points for this bug would be in these files:

http://mxr.mozilla.org/mozilla-central/source/browser/components/extensions/ext-bookmarks.js
http://mxr.mozilla.org/mozilla-central/source/browser/components/extensions/schemas/bookmarks.json
http://mxr.mozilla.org/mozilla-central/source/toolkit/components/extensions/test/mochitest/test_ext_bookmarks.html

It also looks like chrome.bookmarks is mostly a wrapper around the Places API,

https://developer.mozilla.org/en-US/docs/Mozilla/Tech/Places/Places_Developer_Guide

so, fixing this bug would involve JavaScript similar to what I currently have in my extension.

I cloned mozilla-central this morning and am now writing this on NightlyDebug.  I think the next step is to wait for the design decision on whether or not Tags and the other three Firefox-only attributes are going to be obsoleted.

Bug 1225916 Tags (this)
Bug 1276817 Keyword
Bug 1276819 Description
Bug 1276821 Live Bookmarks
Preliminary comment: I inherited an ancient plugin https://github.com/wagle/tagspace-archive that lets you do arbitrary SQL queries to the places database.  If webextensions does that, then I think I'm set in converting/rewriting it (modulo me wrapping my brain around its javascript).

It currently works, albeit with my own use cases.

So the upshot is that I really need to query tags with complex queries, quickly (ie, without 6 levels of indirection slowing it down).

I expect to say something more useful in a couple days.
PS.  I have it up on a unlisted plugin "OldTagSpace" whose URL seems to be https://addons.mozilla.org/en-US/developers/addon/oldtagspace/versions/1781154
Well, I can say with relative certainty that WebExtensions won't ever be able to perform arbitrary SQL queries against the Places database. One of the goals of the the WebExtension APIs is to abstract away implementation details enough that we can make major platform changes without worrying about breaking extensions. And given that we definitely want to make major breaking changes to the places DB, we can't give extensions direct query access to it.
Ok, I live in my bookmark tag querier, and have been for years.  I require this to keep track of what I'm trying to do (1800+ page wiki).  You want to take away my cobbled together "wheelchair".

What do you propose that I do instead of using bookmark tags?  The original queries from the user are not SQL, but they get translated to SQL.  We could have the API do that translation, but whose translation do you use?  I propose a competition or something.

I was imagining that I'd prove my concept with a plugin first, then work on moving some of that code into firefox proper.  I worry that you are killing firefox by not allowing for such testing before migration.  Is firefox becoming closed source?  (Actually tricky question.)

My queries are roughly of the forms "Q and Q", "Q or Q", "not Q", and "? search_string", and "tag_name_prefix".  The last is a prefix because it starts partial queries when you start typing and allows for completions.

Note that there's nothing keeping my tags from referring to other things.  I would actually like tags to also refer to other tags, etc.  I don't know exactly what I want to do yet, I need a playground in which to try things out and see what works before changing firefox proper.  That's what plugins used to provide,  (Kinda sorta: things didn't appear to migrate from plugins to firefox much.)

So I don't mind having to port my plugin (and actually play with my ideas), I do greatly object to having no API to convert to.
(In reply to Perry Wagle from comment #20)

I object to your comments which I don't think further the conversation and suggest an attitude that I don't Kris deserves. The original bug and the point Kris made is that there should not be direct access to the places database, but instead an API to do queries to ensure the data integrity of the database.
And I addressed the direct access question with my translation idea of boolean expressions to SQL queries.

The proposal was to take my plugin away from me.  I take that threat with utmost seriousness, and am looking for alternatives.  Not reading me carefully led you astray.  Sorry.
(In reply to Perry Wagle from comment #22)
"You want to take away my cobbled together "wheelchair".", "Is firefox becoming closed source?", "I worry that you are killing firefox by not allowing for such testing before migration". 

No-one wants to remove these things from you right now or kill your Firefox and we are not becoming closed source.  

For example, you could propose and suggest an API, maybe in a Wiki or github page or blog post or whatever that outlines how an API could look and then supply some patches for this bug making Firefox work the way you want it. It of course has to pass approval by the module owners and needs to be discussed by the community.
I have re-read your comment and after a long day with lots of other problems going on at work, I realise I did over react a little. I apologize. I saw you on IRC today, if I see you tomorrow I'd like to take the time to chat to you in person, understand your suggestions. Sorry once again.
I rewrote this after your apology:

Upshot: you guys are scaring the sh*t out of me.

Proposing an API is daunting.  It assumes that I can wrap my brain around the Firefox source, fathom this mysterious direction (webextensions, electrolysis, etc) you guys are taking it, which is not at all obvious to me.  Its not in the current source code, nor in the documentation I've managed to find (generally you just say what you've done, and vague hints as to where you are going).  And sussing this out is something you propose I do every few months for the rest of my life, instead of getting my "real work" done.

Ok, fine.  I'm all for rewriting this large plugin I've inherited (and didn't write!) from scratch to The New Way.  Its big, slow and clumsy, and so far, impossible to comprehend.  My way of designing/evolving it's replacement would be to do it all in a new plugin and see what I want it to do and what I can make it do.

Since I need to experiment before reaching a design decision on the API, how about we shrink that to a shim that translates boolean expressions to SQL queries to wherever tags are today.  What boolean expressions would we be scared of, if any?  No sql injections, no little bobby tables https://xkcd.com/327/ ), assuming the strings in the boolean expressions get properly quoted during translation.  Grokking boolean expressions would be easier for that than SQL expressions.

I look forward to talking with you.  I did get a bunch of URLs on IRC.  I might be healthy enough to fathom my legacy plugin now.
What programming language would a tags API be written in?  A few months ago, I was warned off using Rust for a year, hence my procrastination in pursuing this problem, and waking up now that something seems to be about to happen.
Component: WebExtensions: Untriaged → WebExtensions: General
Priority: -- → P3
Whiteboard: [bookmarks][design-decision-needed] triaged → [bookmarks][design-decision-approved] triaged
Assignee: nobody → ntim.bugs
Hello, Tim.  I've written a patch to fix Bug 1276817 which I am sitting on until Bug 1343473 lands, which I hope will be any day now.  Bug 1276817 is quite similar to this one; it is to add support for bookmark Keywords.  I'd wanted to work on these two bugs and two others in parallel, but I understand Mozilla's policy is to keep bugs as small as practical, so they have been split into four bugs.  

My patch adds several hooks into which these asynchronously-fetched "Firefox only" properties (Keywords, Tags, Descriptions, Live Bookmark stuff) can be wired into, and of course I've wired my support for Keywords into these new hooks.  If my patch is more or less approved, you could add support for Tags into these same hooks.  Even though it works for me, my patch has not even been reviewed yet, and it's possible that I don't know what I'm doing and my patch may be mostly rejected.  In that case, I'm anxious to see you do it :)

In any case, I think you should look at my patch soon.  Posting it here would certainly cause confusion, so I've uploaded it to Pastebin for you: http://pastebin.com/0jKuy55K.  It should be there for 1 month.
Note that both keywords and tags are theorically unrelated to bookmarks and will in future move as such. They are a property of a uri, as bookmarked is.
Also note that keywords are likely to be converted to custom search engines.
Any API that binds those properties to bookmarks is thus likely to break when things will change.
(In reply to Jerry Krinock from comment #28)
> Hello, Tim.  I've written a patch to fix Bug 1276817 which I am sitting on
> until Bug 1343473 lands, which I hope will be any day now.  Bug 1276817 is
> quite similar to this one; it is to add support for bookmark Keywords.  I'd
> wanted to work on these two bugs and two others in parallel, but I
> understand Mozilla's policy is to keep bugs as small as practical, so they
> have been split into four bugs.  
> 
> My patch adds several hooks into which these asynchronously-fetched "Firefox
> only" properties (Keywords, Tags, Descriptions, Live Bookmark stuff) can be
> wired into, and of course I've wired my support for Keywords into these new
> hooks.  If my patch is more or less approved, you could add support for Tags
> into these same hooks.  Even though it works for me, my patch has not even
> been reviewed yet, and it's possible that I don't know what I'm doing and my
> patch may be mostly rejected.  In that case, I'm anxious to see you do it :)
Thanks for the pointers! I mostly plan to re-use some existing add-on SDK code: https://dxr.mozilla.org/mozilla-central/source/addon-sdk/source/lib/sdk/places/host/host-tags.js for this bug.

(In reply to Marco Bonardo [::mak] from comment #30)
> Also note that keywords are likely to be converted to custom search engines.
> Any API that binds those properties to bookmarks is thus likely to break
> when things will change.

It should be part of the engines management API (bug 1268401) I guess.

(In reply to Marco Bonardo [::mak] from comment #29)
> Note that both keywords and tags are theorically unrelated to bookmarks and
> will in future move as such. They are a property of a uri, as bookmarked is.

What API do you suggest for tags ? I plan to add simple tags property that returns a Set of tags to BookmarkTreeNode
Flags: needinfo?(mak77)
> (In reply to Marco Bonardo [::mak] from comment #30)
> > Also note that keywords are likely to be converted to custom search engines.
> > Any API that binds those properties to bookmarks is thus likely to break
> > when things will change.
> 
> It should be part of the engines management API (bug 1268401) I guess.

Probably, unfortunately we are lacking resources to fix the basic firefox implementation, so I can't give you an ETA.
But if you still want to continue, the API should be generic enough to support that future solution, that means it should not be bound to bookmarks in any way.

> (In reply to Marco Bonardo [::mak] from comment #29)
> > Note that both keywords and tags are theorically unrelated to bookmarks and
> > will in future move as such. They are a property of a uri, as bookmarked is.
> 
> What API do you suggest for tags ? I plan to add simple tags property that
> returns a Set of tags to BookmarkTreeNode

Since tags are not a property of bookmarks, but they are bound to uris, it's likely they will grow as an alternative to bookmarks. It's likely for the foreseeable future the firefox ui will keep showing them in the same bookmarking ui we already have, so from a visual point of view not much will change.
But from an API point of view, they should probably not be bound to bookmarks. As I said in comment 8, there is strict specification at this time.
The API should probably be generic: tag and untag a uri, get uris for a tag, gets tags for a uri, get all the tags. It should probably also provide some batch method to help building a tag cloud without having to fetch all the uris for all the tags (maybe the method returning tags could also return use counts). and likely it should have some sort of notification for added/removed tags.
This probably requires its own chrome.tagging. branch.
But please, don't rush an API that will enforce us to keep tags as they are today, cause it's not going to work.
Flags: needinfo?(mak77)
(In reply to Marco Bonardo [::mak] from comment #32)
> But from an API point of view, they should probably not be bound to
> bookmarks. As I said in comment 8, there is strict specification at this
> time.

typo, I meant "there isn't a strict specification". There is no spec at all in practice, cause known use-cases are very generic and uninspiring.
// describes a tag:
class Tag {
  String name; // tag name 
  Set<URL> tagged; // Set of tagged URIs
}

// Query tags by name or/and by URL associated to tags
// No parameter corresponds to querying all tags
// Returns a promise that resolves to an array of `Tag`
browser.tags.query({name, tagged});

// Tags a new URL
browser.tags.tag(name, url);

// Untags an URL
browser.tags.untag(name, url);

// Listeners
browser.tags.onTag.{add/remove/has}Listener((name, url) => {})
browser.tags.onUntag.{add/remove/has}Listener((name, url) => {}))



What do you think ?
Flags: needinfo?(mak77)
Tim,

With tags unbound from bookmarks, it looks like you are "just" writing a WebExtensions wrapper around nsITaggingService :))  Anyhow, as soon as you post a working patch here, I'm anxious to apply it and give it an "alpha" test.  

(In the world of my WebExtension, tags are bound to bookmarks, so I'll need to put "adapter" code into my WebExtension.  Not too big a deal, though – I just need to port from my old extension, which used nsITaggingService in the same way.)
(In reply to Tim Nguyen :ntim from comment #34)
> // Query tags by name or/and by URL associated to tags
> // No parameter corresponds to querying all tags
> // Returns a promise that resolves to an array of `Tag`
> browser.tags.query({name, tagged});

If I want get the tags for a page, will I get a list of Tag objects where every Set contains all the urls included the one I care about, or only the uri I queried? I'm asking mostly because it could be more expensive to return all the uris when I only care about the one I queried.

> // Tags a new URL
> browser.tags.tag(name, url);
> // Untags an URL
> browser.tags.untag(name, url);

The thing to evaluate is whether makes sense to do this one by one, or as a list. So basically, what's more common, adding one tag or adding multiple tags? I assume the latter could be done more efficiently in code.
Also, it seems strange that these don't take Tag object you defined, as well as the listeners. Wouldn't it be more coherent to have common input objects?

(In reply to Jerry Krinock from comment #35)
> With tags unbound from bookmarks, it looks like you are "just" writing a
> WebExtensions wrapper around nsITaggingService 

With an important difference, nsITaggingService is currently synchronous, but will become asynchronous in the future. This wrapper should start asynchronous so it will be ready for future changes.
Flags: needinfo?(mak77)
I think that the first and third issues raised by Marco arise from from declaring the "Tag" object.

Another issue with this declaration is that people already think of a "tag" as a string in a box in a tag cloud.  Let's call the latter a Physical Tag.  The proposed Tag object represents a relationship between a URL and a Physical Tag.  So we're going to create an incestuous homonym :(

I also think the "Tag" object is a layer of unnecessary complexity.  If we did not declare the Tag object, then the proposed browser.tags.query() function could be replaced by two more self-explanatory functions, something like getURLsForTags() and getTagsForURLs().

If in addition we modify the tag() and untag() functions to handle multiple tag names (Physical tags), as suggested in the second issue raised by Marco, then (surprise!) - browser.tags would declare four functions which exactly mimic the four functions of nsITaggingService.  Well, maybe mimicking something which has worked for more many years is not a bad idea :)

If we do really want the "Tag" object, consider renaming it, maybe as "Tagging", to at least eliminate the homonym issue.
TL;DR: I expect to describe my vision of "tags" in various stages over the next several months.  I would prefer people not further restrict their meaning and power until I get a chance to blow everyone's minds with a whole new set of ideas.  The long version below tries to convey what's going on in my head so that you can reason about this in my shoes instead of your own.  I'm panicking because I don't want to be accused of waiting too long, like with the cancel button, but I don't think I'll be convincing anyone of anything for a few months. 


I've been ill for a few months.  Somewhat better now, hence my suddenly reappearing at this late hour.


For years I've been using tags to manage my huge numbers of wiki pages and general bookmarks.  I got used to doing only what I could do with Firefox tags, which is a surprising lot, but at the same time limited and slow.  I can verbalize what I did do much better than what I plan to do soon.

A partial list of what I used to do with very large numbers of tags-as-complex-strings is to: (1) determine the sort order via prefixes, (2) modify sort orders by changing prefixes/infixes/suffixes on the fly, (3) list all the bookmarks with a Boolean query expressions (and, or, not, search), (4) mark bookmarks (mostly my wiki/journal/organizer now) with multiple tags from sub-lists of tags (radio button or otherwise), where, with the complex-strings naming of tags, each sub-list might have different purposes (dates, groups, categories, importance, priority, etc).  Since the tag naming is currently somewhat flexible, I can keep changing and tuning my "system", which I've arduously done several times in the past several years.

What I can't do is stuff like tag a list of tags, and other types of indirection (my memory is log-jammed at the moment,  but its in there).  Over the years, I've had a whole gob of things I wanted to be able to do, but couldn't without spending a year or two re-programming the tag system.  So each time I wanted/needed something new, I'd have to ignore it, and go back to what I *could* do.  This "solution" however, has me turned into a (log-jammed) pretzel, a micromanagement hell of tedious little manipulations <emph> THAT COULD AND SHOULD BE AUTOMATED AND ALSO MADE FASTER </emph>.  As of the past week, I am now mildly funded to spend a year implementing my vision, and the jammed logs unrolled as I can actually implement the stuff in my head.

Its very hard, without producing prototypes (which I hope to have in various stages during the next year) to show people to describe what I have in mind.  I have two sources of incoherency: one I think and program "spatially" and not "linguistically" (I have to translate in my head to English which is hard to impossible), and two my working memory for spatial is (for the past decade) now very confused, and cannot linearize things to be done unless I use my computer (ESR45 + tag querying + wiki) to keep track of things.

What I've done so far this past week is to make the bookmark editor resizable, which did a lot to first ground me in extension writing, and second to learn something about how to translate old-style extensions to "modern" Firefox.  Now maybe claims that the tiny old thing is too cluttered might go away.  If you use my vision of tags, you *will* want a cancel button and/or much better undo'ing, so I'm trying to find my old notes on that.  Finally, I am currently dovetailing between fixing the old bugs in my old overlaid XUL tagspace extension and a new-from-the-ground up pure webextension version.
Hmm..  In comment 38 above, I was mostly responding to comment 37.
Mentor: jhofmann
Mentor: kmaglione+bmo → bob.silverberg
Assignee: ntim.bugs → nobody
+1 to this bug being resolved.

**My reasons:

- Tags is one of the main reasons I still use Firefox. It's strength is amazing! Most people don't realize the benefits of multi-to-multi relation (tags) in stead of 1-to-multi (folders). I'm sure in the future, when people become more savvy, they will use it more and more. So, not working on ány "bugs" of it, will make general users not use it anymore* and eventually deprecated. (I'm afraid Firefox has joined the "popularity" trend instead of the quality.)
* Popularity of things not only comes from the user-side, the availability and ease-of-use influences them, so the developer side influences popularity too.

- I developed 3 FF bookmarks-tags addons and want to convert them to webextensions.
- I can use bookmarks with tags in one of these addons to simulate/replace another addon that is becoming developer-less (TabGroups & TabGroups Menu).

**Development note:
- Just a humble opinion here as a teaser. No need to go into it if you don't agree.
DO we really have to wait for the decisions how tags are bound (bookmarks or uri) and further discussed problems. Once those issues are decided or changed, the wrapper that is necessary to be written, is going to be a small change. It's the FF source code that will need the most changes and webextensions developer will still use the same functions and syntax. So why not just start with that? That way other people can already start converting to webextensions without the extra wait. Right...?
(OK. Some things need to be decided, but most of these are logical and Marco knows what and why. I'd say just go for what he/we decide...
Considered we started adding tags support to Bookmarks.insertTree, and that the discussion didn't proceed (no interest exposed from other vendors, nor from internal planning), it likely means things will stay as they are, so tags only for bookmarks.
I don't see us spending resources on expanding the concept to a new parallel bookmarking system, due to the lack of interest in both our and others planning.

It's likely the tagging backend API will merge into Bookmarks.jsm for simplicity, we'll evaluate that and maybe the API design that happens here will also help us there.

That means I am fine both ways, we could either have a separate tagging API, or put this directly in the chrome.bookmarks API as an additional property. For now the implementation could use the existing tagging API, and we can change the underlying code later.
The problems with merging into chrome.bookmarks may be:
1. tagging and untagging cannot just pass a "tags" property to the update() method, since how would you mark appending a new tag or untagging? maybe we could have "addTags" and "removeTags" properties? (where removeTags would be invalid for create() and throw or be ignored). Both would be arrays of strings.
2. folders cannot be tagged (throw or ignore again in case of misuse?).
3. the API is bookmark-guid specific, but it will effectively affect any bookmark with the same url. Probably could be solved just by documenting that properly.
4. we need hook points for querying the list of tags (with a count for each), the urls for a given tag, the tags for a given url. search() may probably help with querying bookmarks for a given list of tag (would it be an OR or an AND of the tags? what is more common?). Maybe there could be a getTags([optional]id) that without an id return list of tags, with an id return tags for the given bookmark (and thus uri).

I'm not expert of WebExt API designing, so I leave the decision on the API design to the add-ons team, I think I marked all the critical points to evaluate for taking a decision. I can only suggest to try designing both APIs and then compare them and see the pro/cons of the two approaches to decide.
Just want to add that the lack of tags is a blocker for my migration of the Auto-Sort Bookmarks[1] add-on to WebExtensions.

This add-on uses tags (aka annotations) to track folders to exclude from sorting.

[1] https://addons.mozilla.org/en-US/firefox/addon/auto-sort-bookmarks/
(In reply to Eric Bixby from comment #42)
> This add-on uses tags (aka annotations) to track folders to exclude from
> sorting.

dumb question since I don't know your add-on, couldn't you store the folder guids into a different kind of storage (WebExt have some), and just check the guid for equality?
Marco: I agree; I was thinking about that also.  Thus, it's less ideal, but not a blocker because there's a workaround.  This new approach means doing a little more work in the add-on to lookup the setting and then doing cleanup when the folder is deleted.
Possible noob questions, but..

1.
I'm wondering if this bug shouldn't be marked better for the Firefox 57 release?
See: https://wiki.mozilla.org/Bugmasters/Marking_bugs_for_the_Firefox_57_release

2.
Could I write a WebExtensions Experiment just for tags and then push it through the triage?
See: https://webextensions-experiments.readthedocs.io/en/latest/
(In reply to 0RUBn0 from comment #45)
> Possible noob questions, but..

> 2.
> Could I write a WebExtensions Experiment just for tags and then push it
> through the triage?
> See: https://webextensions-experiments.readthedocs.io/en/latest/

Yes, please do.
Since I was sent this direction, I'll add this comment here:

I have fully functional SQL query builder code that's been working peachy keen for years.  I just need to turn it into an API within the webexperiment framework I thought I was applying to in bug 1412747.

I have no idea whether that would be useful form your effort or not.  But I need my thing to work ASAP, with a literal dropdead deadline of June.
Should I port the completely functional query generation code from my legacy tagspace extension to an API for webextensions?

I thought that was a different approach than this bug.  But if you all want me to take over this bug, that's fine.
I'm trying to not spend 6 months on my own implementing something that won't fit into constraints known now by people other than me.  Yeah, I'm slow and confused.  Sorry.
(In reply to Perry Wagle from comment #50)
> Should I port the completely functional query generation code from my legacy
> tagspace extension to an API for webextensions?
> 
> I thought that was a different approach than this bug.  But if you all want
> me to take over this bug, that's fine.

I think you'd need to figure out a way to phrase what you want to do as an operation in terms of bookmark search, re mak's Comment 41. The input to the API would be abstract -- perhaps a list of tags that must all match -- and the implementation of the API would be inside or outside the tagging service. That might involve generating SQL, but I doubt it.

I suspect that taking full boolean expression strings is not an acceptable level of complexity; I would expect consuming extensions would need to do some amount of in-memory joining.
Is there a metric/threshold for what joins to let through, and which to shove off board?

On the one hand, the db engine is going to do joins faster, and speed is important.

On the other hand, screeching everything to a halt while the join is computed isn't good either.

How good is the multi-threading of firefox's version of sqlite?
Also, is there something wrong with the approach of https://developer.mozilla.org/en-US/Add-ons/SDK/Low-Level_APIs/places_bookmarks#search%28queries_options%29 ?

At the moment, it makes sense to me, but I could be wrong.
(In reply to Perry Wagle from comment #54)
> Also, is there something wrong with the approach of
> https://developer.mozilla.org/en-US/Add-ons/SDK/Low-Level_APIs/
> places_bookmarks#search%28queries_options%29 ?
> 
> At the moment, it makes sense to me, but I could be wrong.

The SDK will be removed in Firefox 57 (which arrives on the release channel in 2 weeks).
(In reply to Tim Nguyen :ntim from comment #55)
> (In reply to Perry Wagle from comment #54)
> > Also, is there something wrong with the approach of
> > https://developer.mozilla.org/en-US/Add-ons/SDK/Low-Level_APIs/
> > places_bookmarks#search%28queries_options%29 ?
> > 
> > At the moment, it makes sense to me, but I could be wrong.
> 
> The SDK will be removed in Firefox 57 (which arrives on the release channel
> in 2 weeks).

Yeah, well, I asked about the approach, and wondered why the deprecation?  I have no desire to waste time replicating known mistakes (that I don't know about).
:wagle, if your question is directed at a specific person, setting the needinfo? flag on the bug helps them know somebody is waiting on an answer.  In this case, I think :rnewman is probably best equipped to answer the questions from comments 53 and 54.
Flags: needinfo?(rnewman)
Was thinking someone else might want to chime in, like :ntim did, but thanks for the nudge.
(In reply to Perry Wagle from comment #56)
> (In reply to Tim Nguyen :ntim from comment #55)
> > (In reply to Perry Wagle from comment #54)
> > > Also, is there something wrong with the approach of
> > > https://developer.mozilla.org/en-US/Add-ons/SDK/Low-Level_APIs/
> > > places_bookmarks#search%28queries_options%29 ?
> > > 
> > > At the moment, it makes sense to me, but I could be wrong.
> > 
> > The SDK will be removed in Firefox 57 (which arrives on the release channel
> > in 2 weeks).
> 
> Yeah, well, I asked about the approach, and wondered why the deprecation?  I
> have no desire to waste time replicating known mistakes (that I don't know
> about).

Perry, if you mean the outer working of the functions, I don't think there's anything wrong with it. But if you mean the inner workings, as if you're planning to more or less reuse the code, there would be a problem because there is also talk about changing the inner workings of the querying as well as the schema of the bookmarks, especially tags, in the database.
I think if you click on Marco Bonardo's followed bugs, you might find the ones that describe it all. I don't know them by heart. I think if anyone has a good understanding and overview, it's Marco..
So, in short, it might be better to already plan the code in the direction of the new approaches so that in the future it doesn't has to be rewritten.
Of course, the whole drama* with FF57, might give another angle on it. I'm not sure. 

(* yes, I think the timing and bad planning of WebExtensions is and is going to be a drama. Just my opinion.)
(In reply to Perry Wagle from comment #53)
> Is there a metric/threshold for what joins to let through, and which to
> shove off board?

This is a design question more than anything. We have to balance ease of understanding, ability to change in the future, ease of implementation in the current system, and performance.

A design that is fast, but is only feasibly implemented on top of a particular database schema, is not ideal.

A design that is powerful but discouraging to add-on developers is not ideal.


> How good is the multi-threading of firefox's version of sqlite?

We expect all uses of Firefox databases to be asynchronous. That's not an answer to your question, but it's probably the only material thing.


> Yeah, well, I asked about the approach, and wondered why the deprecation?

Because we're switching to WebExtensions, in short. We don't want to maintain the Add-on SDK or couple Firefox internals to the API.


I'll leave another reply in an hour or two.
(In reply to Marco Bonardo [::mak] from comment #41)

> It's likely the tagging backend API will merge into Bookmarks.jsm for
> simplicity, we'll evaluate that and maybe the API design that happens here
> will also help us there.
…
> 3. the API is bookmark-guid specific, but it will effectively affect any
> bookmark with the same url. Probably could be solved just by documenting
> that properly.

Given that Sync associates tags with bookmarks, not with URLs, and apparently this is might also be a mismatch with the WE API, perhaps now is a good time to revisit that decision?

If the tagging service is (57+) only exposed and used for bookmarks, then perhaps we should just shift tags out from URLs to bookmarks in a migration and make the API per-bookmark?


> I can only suggest to try designing both
> APIs and then compare them and see the pro/cons of the two approaches to
> decide.

Indeed. Perry, I think the next step is to take a look at 

https://developer.mozilla.org/en-US/Add-ons/WebExtensions/API/bookmarks/search

and the rest of the bookmarks API, and "write the docs first" — try to come up with a concrete design (or two) that would meet your query needs that others can critique. I don't see anything too onerous about the old SDK search design, so you might use that for inspiration.

Once we get some consensus on what the API search extension might look like, we can similarly look at adding tags and presenting them in the node results, and then move down the stack to match the API to Places itself.
Flags: needinfo?(rnewman)
(In reply to Richard Newman [:rnewman] from comment #61)
> Given that Sync associates tags with bookmarks, not with URLs, and
> apparently this is might also be a mismatch with the WE API, perhaps now is
> a good time to revisit that decision?
> 
> If the tagging service is (57+) only exposed and used for bookmarks, then
> perhaps we should just shift tags out from URLs to bookmarks in a migration
> and make the API per-bookmark?

I don't think so, it would be extremely confusing for the user to have 2 bookmarks to the same identical page have different tags. Multiple bookmarks for the same url is already confusing enough, but that's what our ancestors decided.
I agree. I would rather see multiple bookmarks per url abandonded than the other direction and having even more confusion by also having different tags for same url.
Does anyone has a use case where for the ancestor scheme or know why the ancestors decided it?
I think it was done to avoid confusing users (it probably achieves a different kind of confusion).
First because the bookmarking system was built as a classical File Manager, so it was expected to have similar functionalities (IE favorites were indeed folders with files into them).
Then if you'd right click on the toolbar, for example, and select New Bookmark, and you don't recall you already have such a bookmark elsewhere, it would end up removing the original bookmark from its location, like if you're moving it. It may actually be a positive thing technically, but it would confuse some users.
If we'd completely redesign the system avoiding the folders hierarchy and going full tags, most of these issues would probably disappear.
I actually have had duplicate bookmarks recently. Some of them are added by mistakes, some of them are due to desktop / mobile using different hierarchies, and some are due to the fact that I use folders as tags--I have a folder on the bookmark bar to store URLs for later, and organized hierarchy in the menu for "normal" bookmarks.

Full tags sounds attractive to me! It means that I don't need to move bookmarks around. Any bug about that I can follow?
I must clarify, there is no official plan for that, I was just outlining why we are in the current situation and what could be an alternative way. But as I said, no short term plans.
Regarding duplicates, I have a feature in my bookmarks management applications to find duplicates, and due to user requests, a further feature to allow/ignore selected duplicates.  That was ten years ago, but I see the feature still being used.  For whatever reason, some people like to have duplicates in different folders.

Of course, one could argue the reason is: No one has ever explained to them the wonders of tags :))  But as long as the other big browsers continue to support folders which allow duplicates, and do not support tags, you're going to have users coming from other browsers with these expectations.
(In reply to 0RUBn0 from comment #63)

> Does anyone has a use case where for the ancestor scheme or know why the
> ancestors decided it?

There are several good reasons for it; some users use bookmarks to represent work items, research results, etc.

E.g., I might have toolbar folders for different tasks or days of the week, and I use "Open all in Tabs" to queue up my morning work -- on Monday and Wednesday I file expenses, each day I open my Bugzilla dashboard, etc.

Or I might have Google Maps in both my Travel Resources folder and my Motorcycle Trip Planning folder.

Or a link to a particular recipe in both "To Make" and "Christmas dinners" lists.

Or I might have a GitHub repo in a folder for a consulting client and also in my "Portfolio" folder.


One of the things I've learned from Mozilla's task continuity research and interacting with lots of Sync users is that there are countless ways that these simple tools are used, sometimes surprising, sometimes delightful, and sometimes horrifying!
As a reply to MArco and Newman: Then in my view, a bit like Marco says, the problem is not bookmarks, but folders. IF we redesign folders, there are no more use cases.

I propose it as follows: No more uni-hierarchical folders in the database, but everything tags. Then the folders on the UI can be special tags (marked "folder" in the DB) and this way several folders can have copies of the same bookmarks and you don't need several bookmarks for the same url anymore. There, all use cases solved. Anybody having had experiences with tags know that this is the way, that using tags system for folders is the solution.
Ever example written here fits in this design. And it makes the codebase much smaller.
But, maybe someone has still a use case that doesn't fit in this?

Before you answer, please think about this approach. If you're not used to complex tagging, you might want to read it twice or put it on paper to understand.

I hope I wasn't too self-righteous here... Heh..
(In reply to 0RUBn0 from comment #69)

> Ever example written here fits in this design. And it makes the codebase
> much smaller.

I think you might be leaping over some pretty big user experience questions here. There is a very deep groove of user experience that's unlikely to change.

And the obvious implementation of hierarchical tags has its own issues -- a 'folder' 'move' changes from a handful of writes into a write to every 'child' in the hierarchy.

> But, maybe someone has still a use case that doesn't fit in this?

Yeah, here's one: different titles (and thus different UI element strings) for the same bookmark. Your proposal of cross-cutting structure via tags means that the same URL in two different places can't have two different titles.

E.g., for a long time I had a Gmail link on my toolbar named "✉︎". Lots of pages have long titles, and that's not good for a toolbar.

Plenty more similar examples: e.g., "download files" and "upload files" in a 'workflow' folder, multiple photo albums accessible at the same URL….

> Before you answer, please think about this approach. If you're not used to
> complex tagging, you might want to read it twice or put it on paper to
> understand.

Heh. It's just turned 12 years since I was part of a WG attempting to standardize a semantic web ontology for tags. I think you'll find this bug has a lot of experienced folks in the CC list!
> a 'folder' 'move' changes from a handful of writes into a write to every 'child' in the hierarchy
I feel like a smaller code base and better design is better than this very small issue. A database is optimized for such things and it's not like you gonna have more than twenty writes. Even a hundred doesn't have much impact, unless we still see the DB design differenly.
Also, I wonder if it is really an issue, when your same argument also applies for multiple bookmarks to one url (and the trailing tags). You get extra writes here too.

>same URL in two different places can't have two different titles
I see.
So, it's only about titles? I'm not sure because your latter examples can be interpreted in several ways..
But if it is only about titles, wouldn't it still be more interesting and less confusing to have multiple titles for the same bookmark than multiple bookmarks for the same url? If the optimizations in the DB, talked about in other bugs, would be implemented, I think this would not be that difficult if (DB-) designed cleverly, including my proposal to let folders be special tags.

>experienced folks
Nice to know.
I didn't mean to be, what's it called, detrimental? It's just that I had some bad experiences with developers and decision makers that really don't grasp tags and their possibilities at all, not only regarding Firefox, but in my whole career as a developer and database designer. It's hard to put something into words that fits every reader's profile. I also haven't heard many of them here on bugzilla.  Might be subscribed to the wrong bugs, I don't know. Anyway, I'm informed now.
Just so you all know, I'm working on:

(1) figuring out the query generation in my tagspace plugin.  It constructs some non-trivial SQL queries, and I'm abstracting that out to see if I can produce an API capable of maintaining the-way-tags-were.  I absolutely require reasonably complex tag queries for when ESR52 goes away, else I would no longer be able to keep things straight by augmenting my mind with external aids.  I'm already terribly confused with my current setup.  I would be non-functional without it.

(2) trying to guess at the features I want to make bookmark processing faster, more powerful, and less tedious.  This will require many months of experimentation to get "right" for me and other people.

I should be able to say something actually concrete in a week or two.

Note:  If nothing else, I need to be able to make directed graphs of bookmarks.  Folders impose a hierarchy that simply isn't expressive enough.
Marco --

I've seen a number of pokes you've made over the years to split tagging off from the rest of bookmarks (places?).  Is there a latest version for me to see?

What I want is to automate the use of my personal-wiki as a bookmarking system, INSTEAD of the current one.  I'm not seeing a way to get from "here" to "there".

Blindly making all of it a separate app is likely to introduce latency/bandwidth barriers at the host process boundaries that slow everything down.  So I'd like the flexibility to empirically determine where those boundaries *should* be.  I expect having all the bookmark queries go "off-board" to the wiki would be terribly slow, but I'm like to either prove that, or else (pie-in-the-sky) have my program compute it for me.
Flags: needinfo?(mak77)
(In reply to Perry Wagle from comment #73)
> I've seen a number of pokes you've made over the years to split tagging off
> from the rest of bookmarks (places?).  Is there a latest version for me to
> see?

Our scope is just to move them to their own table IN Places and merge their APIs into the Bookmarks API.

> What I want is to automate the use of my personal-wiki as a bookmarking
> system, INSTEAD of the current one.

Not sure what this means and the relation with this bug.
Honestly, looks like you're working on something different and unrelated to this specific bug, and as such it's not the right place to discuss it.

> So I'd like the flexibility to empirically determine where
> those boundaries *should* be.

Anything external to Firefox must go through WebExtensions boundaries.
Flags: needinfo?(mak77)
(In reply to Marco Bonardo [::mak] from comment #74)
> (In reply to Perry Wagle from comment #73)
> > I've seen a number of pokes you've made over the years to split tagging off
> > from the rest of bookmarks (places?).  Is there a latest version for me to
> > see?
> 
> Our scope is just to move them to their own table IN Places and merge their
> APIs into the Bookmarks API.

Ah, ok.  I thought maybe the scope was more extensive than that.

> > What I want is to automate the use of my personal-wiki as a bookmarking
> > system, INSTEAD of the current one.
> 
> Not sure what this means and the relation with this bug.

Just my thinking about what happens after this so whatever that is could be made easier.

> Honestly, looks like you're working on something different and unrelated to
> this specific bug, and as such it's not the right place to discuss it.

I'm trying to work on a bunch of things.  It would be more related if more substantial changes were proposed.  But for the reason you give (not right place), I didn't go into details here.

> > So I'd like the flexibility to empirically determine where
> > those boundaries *should* be.
> 
> Anything external to Firefox must go through WebExtensions boundaries.

Definitely.  But how to determine what is external to what?  Measurement seems as valid as gut instinct.  One of the things I'm trying to implement elsewhere is having a programming/runtime environment that automates measurement to determine boundaries, but I was trying to avoid going into that as it is not relevant here in this bug.  I brought it up because it might have had something to do with the more substantial modification to tags I thought you might be proposing.  But you aren't.

Given that I'm doing this for free, and I project months of work (to replace my tag query extension), I'm trying especially to minimize the work I have to do, and I have to complete it before ESR52 and my ability to program at all goes away (ESR52 is already being rejected by webservers).

---

Now that I know what direction you are wanting to take tags, I'm now thinking more clearly about the task at hand, which was my goal.  I will propose something soon.
Do I understand the situation correctly:
- WebExtensions has no support of reading/writing tags/descriptions in bookmarks
- FF 57+ itself does support tags and descriptions in bookmarks, and its built-in sync feature preserve both mentioned bookmarks fields as well
- there are no plans to re-enable reading/writing tags/descriptions with WebExtensions bookmarks  API

Are all the three statements true?

To me it's beyond rational explanation why features still supported by browser itself are removed from WebExtensions API. Xmarks (one of affected extensions) stopped supporting tags/descriptions and became virtually useless to those using either of these features.

"Syncing: bookmarks  by means of manual importing/exporting is a stone age technology.
no, you didn't understand correctly.
Marco, will you be so kind to elaborate the scope of my misunderstanding?

Bookmarks have
- title
- URL
- short description
- tag(s)
- description

WebExtensions API only allows working with tags attribute. Do I understand correctly that all the rest of fields isn't supported?

There's no mention of them in docs. So the question is, will all the above fields/attributes be ever supported by WebExtensions Bookmarks API?
Tags will be supported, description and short description (what is it, never heard of it) is unlikely. It's all written in the previous comments and this bug is not resolved.
The fields value, in bookmarks manager, are:

    Name
    Location (URL)
    Tags
    Keyword
    Description

So far, according to your words, URL and tags might be supported in unknown future. Support for remaining fields (including "name") is not yet decided.
Name is the same as title.
Keywords will move elsewhere, it's not a bookmarks feature, it's a search feature in the wrong place (bug 648398).
So we remain with

Name (title)
Location (URL)
Tags
Description

the fate of the latter is still unclear. Note that built-in FF sync transfers all the above bookmark attributes; one should expect WebExtensions API should do the same.

Note: FF 57 is already published. Legacy add-ons are disabled. Modern (WebExtensions) implementation of bookmarks syncing add-ons halted, since the mentioned API is still under development.
(In reply to Konstantin Boyandin from comment #82)

> the fate of the latter is still unclear. Note that built-in FF sync
> transfers all the above bookmark attributes; one should expect WebExtensions
> API should do the same.

I think you've assumed that extensions should have all the capabilities of code running inside the browser. That is not the case. 

Certainly it's unlikely that one could ever correctly implement syncing via a WebExtension, simply because syncing — when done correctly — is very tightly coupled to the details of storage (e.g., change counters flipping within the same transaction as the change they track, database recovery events firing when files are found to be corrupt, atomically substituting GUIDs, and such), and the entire point of WebExtensions is that they are _not_ tightly coupled to the implementation of the browser.

Even if the WebExtensions bookmark API grew transactions and support for descriptions, it still wouldn't be enough to replace Firefox Sync.

If you want to do fine-grained syncing, then your extension needs to own storage, too. My understanding of the "WebExtensions way" is that you can write a sidebar and a toolbar button and your own syncing ServiceWorker to build your own bookmarks substitute, but it's not a design goal that you can replace Firefox Sync.
Would it be possible for a webextension to extend the standard bookmark sync with its own database?  Or would one be stuck re-implementing syncing?
Flags: needinfo?(rnewman)
(In reply to Richard Newman [:rnewman] from comment #83)
> (In reply to Konstantin Boyandin from comment #82)
> 
> > the fate of the latter is still unclear. Note that built-in FF sync
> > transfers all the above bookmark attributes; one should expect WebExtensions
> > API should do the same.
> 
> I think you've assumed that extensions should have all the capabilities of
> code running inside the browser. That is not the case. 

Your assumption (what I was assuming) is wrong.

All I need, as an extension developer, is access to entire bookmark data: title, URL, tags, description. The rest is up to me, how to do the syncing/whatever I need to do. Just allow me to access/init all the bookmarks attributes.

ATM WebExtensions Bookmarks API still doesn't allow accessing the entire bookmark object, as it is seen by user when bookmarks manager. That has already ruined the usefulness of Xmarks, which failed to transfer tags/description.

I also would like to remind that legacy API is no more available, and modern API is just not yet mature to replace it in full. Which is not what I expected from developers who were so ardently phasing out old API.

So my question is when the entire bookmark object can be handled by WebExtensions API. Surely it doesn't need 2 years to implement handling two remaining fields (tags and description)?

Thanks.
(In reply to Perry Wagle from comment #84)

> Would it be possible for a webextension to extend the standard bookmark sync
> with its own database?  Or would one be stuck re-implementing syncing?

Not really, no; separate storage implies separate sync.

> So!  In summary, I'm asking if modularizing tags in the above fashion is the approach to take or not.

I suspect there are two ways forward for you:

1. Whenever this bug is implemented, you'll be able to read and write tags. That should be everything you need to write a more powerful bookmark editor with arbitrary tag querying and editing. The constraint is that you have to wait for this bug to be implemented, and hope the result is what you need.

2. Store data outside of Firefox via Native Messaging, or in IndexedDB within Firefox, and expose your own UI for tag handling and navigation.


(In reply to Konstantin Boyandin from comment #87)

> All I need, as an extension developer, is access to entire bookmark data:
> title, URL, tags, description. The rest is up to me, how to do the
> syncing/whatever I need to do. Just allow me to access/init all the
> bookmarks attributes.

My comments were essentially the observation that an extension -- even one with access to those fields -- doesn't have enough hooks to implement correct synchronization.

As such, I do not believe that Xmarks is a viable thing to implement as a WebExtension (nor, indeed, as a legacy add-on!), and so I don't think your articulated use case is strong motivation to do the work.

There are probably other bookmark-related add-ons that _do_ have good reasons to access tags and descriptions (e.g., backup, tag editing), but I'm afraid third-party bidirectional sync isn't persuasive to me: it will get the synchronization wrong, and that will impact Firefox users.

(For context, I've been working on synchronizing Firefox data for about 7 years now.)


> I also would like to remind that legacy API is no more available, and modern
> API is just not yet mature to replace it in full. Which is not what I
> expected from developers who were so ardently phasing out old API.

This has been eloquently put by the endlessly patient folks working on add-ons, but: 

1. This change couldn't wait until full feature parity was reached, for all kinds of reasons.
2. Full feature parity is not a goal; many of the capabilities exposed to XUL add-ons were, with the benefit of hindsight, a bad decision. A large and deep extension surface is not a good thing for Firefox.

I understand that those decisions leave a small set of add-on developers, and a small set of users, unhappy. Everyone is doing the best they can.

This bug is not a good place to discuss this, and you're not going to change the past, so I suggest this thread ends here.


> So my question is when the entire bookmark object can be handled by
> WebExtensions API. Surely it doesn't need 2 years to implement handling two
> remaining fields (tags and description)?

As you can probably tell from Marco's comments throughout this bug, it's not quite as simple as slapping a couple of fields in an object somewhere. But this request has been approved and triaged, so now it's all about resources and taking appropriate care to coordinate with other changes.
Flags: needinfo?(rnewman)
> (In reply to Konstantin Boyandin from comment #87)
> This bug is not a good place to discuss this, and you're not going to change
> the past, so I suggest this thread ends here.

I will answer by email and would appreciate the appropriate place to discuss such issues.

A developer forum where can be a real response from real developers (preferably, different from silence).

Thanks in advance.
(In reply to Richard Newman [:rnewman] from comment #90)
> (In reply to Perry Wagle from comment #84)
> > So!  In summary, I'm asking if modularizing tags in the above fashion is the approach to take or not.
> 
> I suspect there are two ways forward for you:

> 1. Whenever this bug is implemented, you'll be able to read and write tags.
> That should be everything you need to write a more powerful bookmark editor
> with arbitrary tag querying and editing. The constraint is that you have to
> wait for this bug to be implemented, and hope the result is what you need.

Maintaining the status quo with respect to tags isn't going to do me any good fast enough.  I now need something "by yesterday".  That is The Web is squeezing out ESR52 more and more on a daily basis as we speak, and I suspect an implementation of a tags API wouldn't be in standard Firefox even by June 2018, when ESR52 goes away entirely.

I need much tighter integration with my personal wiki to make headway.  I need to work on that as a webextension because I need to experiment for a year or so to evolve this thing to the point that others might want to use it.

What I need is the hooks to do my own tag/triple system.  I don't think they are there now.  And, I don't yet know what is missing.

> 2. Store data outside of Firefox via Native Messaging, or in IndexedDB
> within Firefox, and expose your own UI for tag handling and navigation.

Well, I will now see if I can do it this way.  There were all sorts of obstacles earlier.  Maybe not any more.  But discussing it here seems off-topic to this bug, as :mak noted above.

---

I imagine I will continue to flip-flop on what approach to take.  Sorry.  Trying to fix that mental issue with my technology.

If someone else wants to take over this bug, please speak up.  Else, I will probably keep poking at it over time, as I go around and around.
Assignee: nobody → bob.silverberg
I made an extension where users can search bookmarks by the folder hierarchy they are placed in or by domain or path. It is a really hackish solution, I change the title of the bookmarks. Here it is https://addons.mozilla.org/ro/firefox/addon/filter-awesomebar/

Being able to read and write tags would be awesome!
we started working on the tags backend, I'd suggest to just wait a little bit more.
Flags: needinfo?(bob.silverberg)
Assignee: bob.silverberg → nobody
Mentor: bob.silverberg
There's nothing to test for now, we are converting some of the data, then we'll start working on the backend APIs. ETA is this quarter, unless we hit huge roadblocks.
Flags: needinfo?(mak77)
Will it be even possible to add AND/OR/NOT expression queries over the tags database AND the bookmarks (still "places"?) database?  I'm nervous about being painted into a corner if you can't possibly query over both databases simultaneously.
Sending in AND/OR/NOT expressions in as nested JSON is what I saw somewhere do a while ago.  Not finding the URL anywhere yet, though.
Product: Toolkit → WebExtensions
Hi,

I was following this thread and ended up implementing a WebExtensions Experiments API to retrieve tags (getTagsForURI) and URIs (getURIsForTag) from tags. You can find the result here: https://github.com/azappella/webextension-experiment-tags (instructions on how to use it are in the README). 

I realize that this is only a partial implementation of the underlying tagging API, but I needed something quick and fast to retrieve tags. It seems the tags API might merge with the bookmark API soon (https://bugzilla.mozilla.org/show_bug.cgi?id=1448885), so it might not be relevant at that point. 

Hopefully, it can still be useful to someone. 

Cheers,

Andrew
With very few tweaks

    https://github.com/wagle/webextension-experiment-tags/tree/wagle-prototype-0001-snapshot-0001

(notice the branch name, which is actually a tag), has Andrew's API working for a simple prototype bookmarks lister sidebar webextension.

For a un-pretty demo of that from a regular webextension, see

    https://github.com/wagle/tagspace-tng/tree/wagle-prototype-0001-snapshot-0001

let me know what, if anything, is confusing.
My (not here yet) replica of the legacy extension I inherited first wants to know what ALL the tags are, to display them in a fast scrollable list, then a list of bookmarks that possess 1-or-more tags to likewise display in a scrollable list.

The idea is you start with an empty "tags-query" (meaning all tags).  Display a list of all the tags satisfying that query, and all the bookmarks having any of those tags.  Then you use a hybrid gui/text boolean query system to focus the tags-query (ands, ors, and nots of tags or tag-queries) to produce a shorter list of tags according to what matches "the venn diagram" of the current tags-query.

When you've focused the list of bookmarks "enough" to make you happy, you can do various operations, including "start all in tabs".

In the legacy edition of "tagspace" running in ESR52, the empty tags-query displays 10281 tags and 13202 bookmarks (its a bit bloated, but pruning it is CURRENTLY "micro-management hell", which I hope to fix with automation).  I usually start up to 70 bookmarks at a time.
Sorry to comment on this long discussion. However, I feel that need to say, that I agree with Perry, that the process of waiting has been quite painful up to this point: there seems to be no single design document available and upcoming milestones are still in the mist...

TLDR;

To IMHO there is 2 separate issues: allowing setting/getting tags for bookmark items and querying.

As I understand Perry is very interested particular in the querying part, which I believe needs no (!) changes to the API only some minor implementations if focussing on boolean "and" querys:

1. browser.bookmarks.search allows specifying search terms: not much would be broken allowing search terms in google style like "site:com" or in this case "tag:xxx" (BTW: a boolean algebra using all uppercase OR and NOT would also not break this interface ...)

2. browser.bookmarks.getSubTree could be used to get all tags. Currently the bookmarks dialog (Library) already exposes this virtual subtree (also for the recent tags). So one would only need to officially document those hard-coded ids (return tags as BookmarkTreeNode with type "tag")


For the getting setting part I do not see much problems simply extending the calls with one more optional argument...


Is this the way you are thinking when merging things into the Bookmarks API??

My bookmarks are also unusable at the moment because folders don't suit my needs and I heavily used tags together with the tagsifter extension.
End user here. There's no possible effective, comprehensive, complete cross-browser bookmarks syncing without this bug being tackled. Please solve this.

If Firefox Sync is able to read/write bookmarks' tags, why not simply copy/adapt code from Firefox Sync into WebExtension API?

Ah, sorry: hit send a bit early there. I meant to strip out the docs and add a tags field to the struct.

What brought me to this issue?

I want to auto-tag with regex rules and then sync those string labels.
As an (RDFS) https://schema.org/Thing, tags could have associated urls (and multiple names for e.g. multiple languages)

Reframing the bookmarks problem as JSON-LD is a worthwhile effort, IMHO.
That would insinuate moving from
"a tag is an edge to a string"
to
"a tag is an edge to a URI which identifies a thing with attributes"

Current:

tag: str

Suggested Linked Data approach to start the discussion before standardization with WebExtensions:

tag: {key: str, name: [str], urls: [str]}
tag: {id: str, name: [str], urls: [str]}

Which committee would need to review the proposed WebExtensions experiment?
(Is this the current URL to that work: https://github.com/azappella/webextension-experiment-tags/blob/master/experiments/tags/api.js )

What would need to be done to support a more complex tag object that FF, Chromium, and WebKit could easily migrate to?

Any updates on this?

I mean allowing addons to edit/use tags ... would improve the quality of some bookmark and tab management addons.

Who are the decision-makers to work with in regards to the WebExtensions Bookmarks API && tags on bookmarks?

With third-party app support for tags via the WebExtensions API, wouldn't you think we'd get native serverside bookmarks with tags functionality?

What are the

  • [ ] specific
  • [x] tasks
    necessary to add tag support to the WebExtensions AP?

Before specifying that tags are just strings for every bookmarks extension, IMHO URIs are the way of the web and tags as the :about property of a https://schema.org/Thing as the most extensible way forward
{:name:, :url, :about,
:educationalAlignment, }

(We can use W3C SHACL to limit which RDF data to accept and store at data validation time. JSON-LD is one representation of RDF; and JSON-LD would let us look our bookmarks together with string tags and URIs/URLs)

From https://bugzilla.mozilla.org/show_bug.cgi?id=1225916#c115 :

https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/bookmarks/BookmarkTreeNode

Severity: normal → --
Type: defect → enhancement
Severity: -- → N/A
You need to log in before you can comment on or make changes to this bug.