1457658 - WebExtensions API for annotating tabs

Status: REOPENED

(WebExtensions :: Frontend, enhancement, P5)

Description

[Peter Simonyi]

Attachment: Example screenshot

I'd like to propose a new API for WebExtensions:

    browser.tabs.setStatusIcon(tabId, icon);

This would add (or remove, when icon === null) an icon to the tab just before the close button.

Firefox currently has a mute/autoplay status icon that appears there; this would allow extensions to do something similar.  Extensions could use it for any kind of tab annotation they need.

See the attached screenshot for an example.  An extension has marked two tabs, each with a different status icon, in addition to one tab showing that it's muted (Firefox default behaviour).

Rationale:

I have an extension (https://addons.mozilla.org/en-US/firefox/addon/tab-teleporter/) that could benefit from this.  As a workaround, it prepends emoji to the page title so there's an extra icon in the tab.  However, this requires the <all_urls> permission, which is tremendous overkill (and "a little scary", as a friend commented).

I suspect that this feature will also satisfy many of the non-theme use cases presented in bug 1320585.  Providing status icons instead of just changing colours of existing elements could even be an accessibility improvement for some of those.

There are at least two other kinds of tab annotation in Firefox today: contextual identities are marked with a coloured bar under the tab, and tabs received from sync get a cyan dot under the favicon.  However, neither of these can be reused for multiple purposes.  With icons, each extension can have its own space, and if the user installs lots of extensions that add status icons so there are too many, they will just get cut off fairly obviously.

Other issues:

Would it be better to have a tabs.clearStatusIcon(tabId) instead of tabs.setStatusIcon(tabId, null)?
Is there any reason to have a getter function?
Should the icon be clickable, like the mute icon?  (This could be done later.)

Comment 1

[Peter Simonyi]

Attachment: Bug 1457658 - tabs.setStatusIcon WebExtension API

Review commit: https://reviewboard.mozilla.org/r/240558/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/240558/

Comment 2

[Peter Simonyi]

Attachment: statusicontest.xpi

Here's a demo extension.  (Make a copy and give it a different ID to see how multiple extensions would interact.)

Unfortunately, tabs are XBL bindings, and it seems that means JS can't mess about with their internals, so this demo also requires the (WIP) patch above.

Comment 3

[Caitlin Neiman [:caitmuenster]]

Hi Peter, we're going to try to get to this at the next WebExtensions APIs triage on May 8. :) Would you be interested in joining us at the meeting? 

Here’s a quick overview of what to expect at the triage: 

* We normally spend 5 minutes per bug
* The more information in the bug, the better
* The goal of the triage is to give a general thumbs up or thumbs down on a proposal; we won't be going deep into implementation details

Relevant Links: 

* Wiki for the meeting: https://wiki.mozilla.org/WebExtensions/Triage#Next_Meeting
* Meeting agenda: https://docs.google.com/document/d/1TRkZ2u3GQXwlHpfP4-P_SXcwIwvlQz5Sm-C8I89Ya_o/edit#
* Vision doc for WebExtensions: https://wiki.mozilla.org/WebExtensions/Vision

Comment 4

[Tim Nguyen :ntim]

> Is there any reason to have a getter function?

Yes, so sidebar tab extensions can provide this UI as well.

Comment 5

[Peter Simonyi]

(In reply to Caitlin Neiman [:caitmuenster] from comment #3)
> Hi Peter, we're going to try to get to this at the next WebExtensions APIs
> triage on May 8. :) Would you be interested in joining us at the meeting? 

Thanks, I think I can manage IRC at least.

(In reply to Tim Nguyen :ntim from comment #4)
> > Is there any reason to have a getter function?
> Yes, so sidebar tab extensions can provide this UI as well.

Good point.  I was thinking of a getter that would return only the current extension's icon (which is why it seemed unlikely to be useful), but it sounds as if you'll want something to return the icons set by all extensions.  Since that necessarily provides information about other extensions, it should probably require the 'management' permission as well as 'tabs'.

Something like tabs.getStatusIcons(tabId) returns an array of objects with an extension ID and the icon info object set by the extension.  Or it could instead return an object mapping extension IDs to the icon info object, I suppose, since I'm proposing that each extension be limited to one icon per tab.

Comment 6

[Caitlin Neiman [:caitmuenster]]

Mike to follow up with UX.

Comment 7

[Mike Conca [:mconca]]

Markus, the add-ons team is looking for UX to have final call on this feature request. Extension developers would like the ability to set an icon in the tab, very similar to the audio (or mute) icon that Firefox itself displays when a tab is playing audio.

On the one hand, there are many use cases where this could be beneficial for extensions, allowing them to visually communicate state to the user. On the other hand, people are only used to seeing Firefox icons in that space. It was pointed out in our design decision meeting that users (currently) would have no way of knowing which extension an icon belongs to, or what it might mean. So there are Project Jazz considerations here, too.

Is placing icons in the tab something that extensions should be allowed to do?

Comment 8

[Markus Jaritz [:designakt] (UX)]

Attachment: Indicators-in-Tab.png

(In reply to Mike Conca [:mconca] (Denver, CO, USA UTC-6) from comment #7)
> On the one hand, there are many use cases where this could be beneficial for
> extensions, allowing them to visually communicate state to the user. 

Understanding the use cases better would be helpful to understand if this proposal is the only or best way to solve them.
The only use case I see currently is marking the teleport target for the mentioned extensions... and I don't think this case alone justifies adding the complexity to the already crowded tab UI. 

> On the other hand, people are only used to seeing Firefox icons in that space. 
> It was pointed out in our design decision meeting that users (currently) would
> have no way of knowing which extension an icon belongs to, or what it might
> mean. So there are Project Jazz considerations here, too.

With every UI surface we introduce for extensions, we also need to account for unjustified or unhelpful use of that UI, and the impact it has on related UI. This risk seams to be big on this one, as placing something in the small space of a tab, should be very well considered. (but would be opened up to any extension with this API)

> Is placing icons in the tab something that extensions should be allowed to do?

With current knowledge of the usefulness and potential risk, I do not want extensions to be able to place random elements in tabs.

Comment 9

[Markus Jaritz [:designakt] (UX)]

Closing this as wont-fix because of limited usefulness and high risk for existing UI. (see comment 8)

Comment 10

[kernp25]

I think a notificationbox or PopupNotifications makes more sense!

https://bugzilla.mozilla.org/show_bug.cgi?id=1331557

Comment 11

[Karoly Pados]

> With every UI surface we introduce for extensions, we also need to account
> for unjustified or unhelpful use of that UI, and the impact it has on
> related UI. This risk seams to be big on this one, as placing something in
> the small space of a tab, should be very well considered. (but would be
> opened up to any extension with this API)
> 
> ...
> 
> With current knowledge of the usefulness and potential risk, I do not want
> extensions to be able to place random elements in tabs.

As the above argument seems to weigh a lot in the final decision, I would like to point out it is somewhat moot and pointless. You have to note that WebExtensions *already* have a way to manipulate what is shown in the tab, since they can attach custom and random text to the tab title. This text can be tag-like (e.g. "[EXAMPLE]") or unicode symbols (e.g. "♫"), and it is extremely simple to implement using a content script.

My point is, extensions can already do what you are trying to prevent them to do, except visually much nastier. If an extension wants to add a badge or tag to a tab header, it will do it today and now. But the proposed API would allow WebExtensions to make such badges visually nicer and compacter (a 16x16 icon will take up a less space than almost any random text segment, while at the same time being more distinguishable and appealing).

So the original argument the negative decision was based on will not prevent any misuse. It only prevents a nicer GUI and a more compact tab header for any addon that would misuse the header anyway.

Comment 12

[Karoly Pados]

> The only use case I see currently is marking the teleport target for the
> mentioned extensions... and I don't think this case alone justifies adding
> the complexity to the already crowded tab UI. 
> 

Here is another addon that wants to benefit from badges, to mark tabs set for active reload: https://addons.mozilla.org/en-US/firefox/addon/reloadmatic/

In general of course, any extensions that wants to indicate the addon-specific status of a tab is a potential user. In light of this, I'll change jobs and move to another city if Tab-Teleporter and ReloadMatic are the only ones that would find the proposed API useful. To dismiss it by saying that "marking the teleport target for the mentioned extensions" is the only foreseeable use-case is - sorry to be blunt - very short-sighted.

Comment 13

[Markus Jaritz [:designakt] (UX)]

(In reply to Karoly Pados from comment #11)
> You have to note that WebExtensions *already* have a way to manipulate what 
> is shown in the tab, since they can attach custom and random text to the 
> tab title. This text can be tag-like (e.g. "[EXAMPLE]") or unicode symbols 
> (e.g. "♫"), and it is extremely simple to implement using a content script.

This sound's more like an argument to not introduce this API.
If one already can add items, why would we need to introduce a new UI for the same thing?

> My point is, extensions can already do what you are trying to prevent them
> to do, except visually much nastier. If an extension wants to add a badge or
> tag to a tab header, it will do it today and now. But the proposed API would
> allow WebExtensions to make such badges visually nicer and compacter (a
> 16x16 icon will take up a less space than almost any random text segment,
> while at the same time being more distinguishable and appealing).

I understand what you are saying... And I think it is great that extensions that really want to do something like this, can already do so, without introducing an new UI for it.
Please consider that any new and nicer UI also comes with a cost of building and maintaining said UI, plus dealing with all the edge-cases the new UI brings. (e.g. What colors can be used? How to deal with light/dark themes? How many annotations per Tab? What annotation disappears first if the tab shrinks, or more annotations are added? Can it be clicked? How can the user know what extensions created this annotation? Does it have it's own tooltip?… )

> So the original argument the negative decision was based on will not prevent
> any misuse. It only prevents a nicer GUI and a more compact tab header for
> any addon that would misuse the header anyway.

There are not many (no) popular extensions using the current way of annotations:
So, either the original decision either prevents misuse good enough;
or it does not, but the feature of annotations is not very popular.

In both cases, adding new UI in the already crowded tab does not improve the situation. 
(As it crowds the tab even more, and create a lot of additional work, for very limited benefit.)

Comment 14

[Markus Jaritz [:designakt] (UX)]

(In reply to Karoly Pados from comment #12)
> > The only use case I see currently is marking the teleport target for the
> > mentioned extensions... and I don't think this case alone justifies adding
> > the complexity to the already crowded tab UI. 
> > 
> 
> Here is another addon that wants to benefit from badges, to mark tabs set
> for active reload:
> https://addons.mozilla.org/en-US/firefox/addon/reloadmatic/
> 
> In general of course, any extensions that wants to indicate the
> addon-specific status of a tab is a potential user. In light of this, I'll
> change jobs and move to another city if Tab-Teleporter and ReloadMatic are
> the only ones that would find the proposed API useful. To dismiss it by
> saying that "marking the teleport target for the mentioned extensions" is
> the only foreseeable use-case is - sorry to be blunt - very short-sighted.

I did not say teleport-target is "the only forseeable use-case" ! It is the only I was aware of.

My comment only reflects my current view of the situation, if there are more applications, please share:
- all examples of extensions doing annotations
- all use-cases you are aware of, that are actually helpful, and can not be solved with other UI
(please be specific, as it is not possible to build a good UI on a broad generalization)

Provided such information, I am happy to re-evaluate the usefulness of a tab-annotation-API.

(I tried to come up with use-cases myself, but could not come up with any that would require a tab-annotation, or be way better with a tab-annotation. That is besides the teleporter, which I personally really like.)

Comment 15

[Peter Simonyi]

Here are a few extensions that already use content-script hacks (adding symbols to the page title, or changing the favicon).  There might, of course, be more out there -- this feature is kind of hard to search for.

- Tab Teleporter, my original use case above.

- ReloadMatic https://addons.mozilla.org/en-US/firefox/addon/reloadmatic/
mentioned in this bug; marks tabs that it will automatically reload

- Unread Tab https://addons.mozilla.org/en-US/firefox/addon/unread-tab/
marks tabs with a side-pointing triangle when they're opened in the background until they are selected by the user

- Snooze Tabs https://addons.mozilla.org/en-US/firefox/addon/snoozetabs/
former Test Pilot experiment, flashes the favicon on tabs that come back from being snoozed away

- Tab Numbering https://addons.mozilla.org/en-US/firefox/addon/tab-numbering/
marks tabs that can be selected by keyboard shortcut with the superscript number that's part of the shortcut

This technique has a lot of downsides: as the author of ReloadMatic notes, it will clash with any other extension doing the same thing; it requires excessive permissions; it fails on protected pages like AMO and about: pages; and the available symbology is limited to what's common in system fonts.

---

Here follow, in no particular order, some more ideas about the kinds of extensions that could benefit from a way to put some mark on individual tabs.  It's probably not exhaustive -- could you list all the ways browserAction would be used? -- but just the result of me sitting down and thinking about it for a bit on the Victoria Day weekend.  It's probably good that most of these are either responses to user action or fairly ephemeral, since that should result in less contention for limited space.

In general, this is for information that is relevant to a specific tab or the page open in a tab when the tab is not (necessarily) the foreground tab.


Some extensions try to manage "old" tabs (e.g. Dustman AutoClose Tabs, Tab Auto Close).  They could add an icon to tabs that are getting old as a sort of background warning before they are actually removed or archived.  (The extensions cited actually seem to remove the tabs with no warning or archive feature!  Surely someone wants this kind of thing with some unobtrusive warning.)  Actually, I'd be interested in a way to just mark tabs with their age (e.g. a bar scaled to relative age) so I can clean them up myself.

Some websites try to show notifications in the tab by changing the favicon, flashing a title change, etc.  An extension could intercept these techniques and replace them with an icon, giving the user more control about how to present these things (do you want new messages to flash, or just show a static speech balloon?).

I've been thinking of making something to detect opening the same page in multiple tabs; it would help to show which tabs are the duplicates on the other tabs themselves.  (Existing extensions like this just list the page titles, which is not very useful in the case of duplicates!)

An extension could flag tabs that are using a lot of resources (data, CPU, automatic reloading from a <meta> refresh).  I'm not sure how much CPU performance data is available to extensions, but they can definitely track network usage.

An extension could make an alternative loading progress indicator for slow connections.  On very slow connections, there's usually a point when it's reasonable to start reading some time after the loading starts but well before all the ads and images have finished.

## Explaining why a tab is here

Firefox Sync's send tab to device feature highlights received tabs with a little dot under the favicon.  It's a great way to quietly indicate where this tab came from, especially when you don't get around to looking at it for a while.

There are a few sync alternatives out there.  None of the ones I've seen have a feature like send tab to device yet, but if they did they'd benefit from being able to mark the tabs so received.

Some extensions also open tabs without direct user interaction, and if they want to do it in the background, being able to indicate that they're the source would be nice.  Examples: Snooze Tabs (several extensions use this concept), daily schedule openers (Morning Coffee Quantum, Today's Pages, Simple Browsing Schedule -- at least some of these seem to require user interaction to open the scheduled pages, possibly because unexplained tabs showing up is bad UX).

The icon for this kind of thing would generally go away once the user selects the tab.

## Highlighting during user interaction

Extensions that involve searching for tabs (e.g. Tab Finder, TabSearch, Tabhunter, Conex, etc.) could use this to implement a kind of highlight-all function analogous to the highlight-all typeaheadfind mode.

Extensions that involve selecting multiple tabs manually or by search for some action would also benefit from being able to mark the tabs they'd affect (e.g. MultiSelect Tabs, Tabby).

Currently, extensions that do these kind of search/select operations on tabs have to implement their own list of tabs to show the selection or search results.  They'd probably need to continue to do so since the tab bar can overflow, but being able to show a mark on visible tabs would help because there's a spatial component to "which tab is that".

Comment 16

[Markus Jaritz [:designakt] (UX)]

Thank you for this extensive list of examples and ideas. Seams there are some more use-cases of such a feature than I came up with. Thank you very much for taking the time. This was a great help for me to get a better understanding of the situation.

Your list made me realize that there are apparently a few different forms of annotations / notifications:
- a simple highlight indicator (like send tab to device uses)
- a text / number indicator (like a badge)
- a visual indicator (icons and symbols)
- an interactive area (like mute / close)

You examples also helped me see annotations from a different angle. Annotations would not necessarily need to be adding more elements to a tab as it was initially suggested, but could be enabling extensions more controlled ways to use existing UI, like:
- trigger the tab-highlight
- replace / prepend the tab-title
- replace the favicon
(all with options to re-set to what the website had specified, and without giving the extension access to all the tabs content -- to prevent the more intense permission message)

Any of these 3 options would allow for some of what the initial proposal asks for without adding to much complexity, and could be extended if we see useful uptake of such simple annotation option.

(This would not allow for multiple extensions annotating the same tab, and also not for interactive annotations, but those seam to be more advanced asks that do not need to be solved immediately.)

Comment 18

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'WebExtensions::Frontend' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.