1364404 - Create a Floating Panel API

Status: NEW

(WebExtensions :: General, defect, P5)

Description

[Lusito]

Attachment: detached_bar.png

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:53.0) Gecko/20100101 Firefox/53.0
Build ID: 20170413192749

Steps to reproduce:

Note: This "bug" is created as a result of Bug 1344573 being way too broad. So please don't close this as a duplicate!

Use-case: An add-on wants to show a floating panel with interactive menu elements, positioned in window coordinates.

Examples for add-ons that could use this:
- A translation add-on, which translates words below the cursor and wants to display the translation results with some links to more details.
- A custom search add-on which allows using regular expression to search the page.


Actual results:

The API doesn't exist


Expected results:

I'd like to see an API to create a floating panel, which can be positioned freely in window coordinates.
This panel would always contain the add-on icon (and it's name as tooltip on the icon) so the user can identify the source of the panel.
This panel can be used to show interactive tooltip kind of panels as seen in the attached screenshot.

The content of the panel would be HTML.

Most use-cases I can think of would use a floating bar with an icon on the left.
Some might also want to add some height to it, so maybe a different style could be added, where the add-on icon is on the top-left and the add-on name would follow it, with the HTML content below the icon & name "titlebar".

Comment 1

[Shane Caraveo (:mixedpuppy)]

> - A custom search add-on which allows using regular expression to search the
> page.

can Bug 1332144 help with that?

Comment 2

[Lusito]

I have not read every comment on Bug 1332144, but it does not seem to describe the change of ui needed for such a searchbar. (extra checkboxes, buttons, etc.), tho it might simplify the actual search code.

bug 1340930 seems like it would solve it though.

Comment 3

[Lusito]

Attachment: overlay_proposal.txt

Second iteration of the API Proposal

Comment 4

[Lusito]

Attachment: panel_types.png

CreateTypes visualized with demo use-cases

Comment 5

[Lusito]

Attachment: overlay_proposal.txt

Third iteration of the API proposal

Comment 6

[Shane Caraveo (:mixedpuppy)]

Some quick reaction.

I like the images, gives me a good idea what you're thinking.  I'm not convinced about providing full screen mode, maximize, minimize.  I'd prefer to push those out for later.

I'd like to see the API match browser/page/sidebarAction APIs.  It can have more or less if necessary, but that is the direction I was thinking.  Maybe call it panelAction, IDK.  

I think it should be tied to the tab like pageAction is.  I'm not certain about whether the UI should be constrained to the tab as well, but I'm leaning towards that.

I'm assuming only one panel is allowed.

I'm assuming this will be backended by xul:panel (as the *action panels are), so we should also inherit some features, possibly some limitations.

Comment 7

[Lusito]

>I'm not convinced about providing full screen mode, maximize, minimize.
I guess I could have explained that one further.. I did not want a fullscreen mode.
The minimize/maximize should is just a callback so that the add-on can provide a bigger version of the UI.
For example show one entry vs show 5 entries in a list. It would be up to the add-on to specify a different size for the panel.

>I'd like to see the API match browser/page/sidebarAction APIs
I'll take a look at it at the weekend, busy until then.

>I think it should be tied to the tab like pageAction is.
In the case of a Download-Manager panel, it would be better if it was global.

>I'm assuming only one panel is allowed.
No. You might want a search panel and a translation panel open at the same time.

>I'm assuming this will be backended by xul:panel (as the *action panels are), so we should also inherit some features, possibly some limitations.
I'm missing the relevant information to say anything about that.

Comment 8

[Lusito]

Attachment: panelAction_proposal.txt

I have attached the API as far as I have come with it.
The API now assumes one Add-On can have one Panel which is bound to the tab (but I'd still want two different addons to have their panels shown at the same time)

I'm not sure it's such a great idea to match the API to sidebarAction or browserAction.
Those APIs access every property via asynchronous getters & setters (one for each property).
That would be a pain if you need a couple of the properties. Like width, height, top, left, ...

I've split the update method into as many setters as I thought was reasonable, but only one getter.

I've also replaced the maximize-button with a way to add custom buttons to the titlebar, which can be added to the API at a later time.

Looking through the sidebarAction API, there seems to be no sendMessage method.. Is there another way to communicate with it? If so, the sendMessage method here might not be needed.

I'd still prefer to bind it to a window instead of a tab, to allow add-ons to have panels exist across tabs (like a facebook messenger/chat or a download manager add-on for example). I guess you could synchronize the content on tab-switch but that might involve some flickering.

Comment 9

[Shane Caraveo (:mixedpuppy)]

(In reply to Lusito from comment #8)
> Created attachment 8874141 [details]
> panelAction_proposal.txt
> 
> I have attached the API as far as I have come with it.
> The API now assumes one Add-On can have one Panel which is bound to the tab
> (but I'd still want two different addons to have their panels shown at the
> same time)

Which means 20 addons could have their panels shown.  It's best to avoid trying to develop a windowing system for addons based on panels.  Keep the focus narrow.  A panel overlay for content that doesn't interfere with (ie. injection into) the content dom is very interesting.  Recreating window management when Firefox already has it is a much harder sell.

> I'm not sure it's such a great idea to match the API to sidebarAction or
> browserAction.
> Those APIs access every property via asynchronous getters & setters (one for
> each property).

It doesn't have to be like that, just have a set/getProperties instead, but consider the spirit of those APIs and keeping a similarity for consistency in APIs.

> I've also replaced the maximize-button with a way to add custom buttons to
> the titlebar, which can be added to the API at a later time.

I would drop this for now (at least initially).  Panels can resize based on content.

> Looking through the sidebarAction API, there seems to be no sendMessage
> method.. Is there another way to communicate with it? If so, the sendMessage
> method here might not be needed.

There's runtime.sendMessage, and extension.getViews(type) which returns a window object, which I think you can use postmessage.  

> I'd still prefer to bind it to a window instead of a tab, to allow add-ons
> to have panels exist across tabs (like a facebook messenger/chat or a
> download manager add-on for example). I guess you could synchronize the
> content on tab-switch but that might involve some flickering.

A window can be used for something that needs to remain open regardless of tabs.  

Something like facebook messenger would actually be better suited in a window that is not attached to a specific window.  A good example of what that could look like is gtalk in chrome.

Comment 10

[Lusito]

Attachment: panelAction_proposal.txt

I've attached a reduced API using your recommendations.

>There's runtime.sendMessage, and extension.getViews(type) which returns a window object, which I think you can use postmessage.  
Those are not specific to one panel tho, so I'd have to decide in the panel if the received message was actually meant for this panel. Each tab could have one panel receiving this message, so the message would possibly be needlessly relayed to a lot of panels. So I guess a sendMessage method should stay in this API.

I kept sendMessage in the spirit of the other sendMessage methods rather than porting it to *Action like detail objects.

Is this good enough for now or do you need anything else?

Comment 11

[Lusito]

Alternatively: Add extension.ViewType.panel and allow to search using a tabId in extension.getViews(), then panelAction.sendMessage() could be omitted.

Comment 12

[Shane Caraveo (:mixedpuppy)]

Comment on attachment 8874161 [details]
panelAction_proposal.txt

This is looking good, a few more comments below.  I think it's a good time to broaden feedback, and perhaps develop the API as an experiment.

>permissions required:
>	"panelAction" and if the used url is not an internal add-on html, the urls used.

The comment is confusing, I'm not clear what you mean.  If panelAction is not internal, what will the value be?  I'm not certain I want to allow arbitrary web urls here.


>	sendMessage(tabId, message)

Still not entirely convinced about this, but can shelve that for now.

>	panelAction.Panel
>		focused: boolean. Whether the panel is currently focused.
>		visible: boolean. Whether the panel is currently visible.

IMO a panel should always be visible (within the tab) until closed or dismissed.  What is the intended use for these?  Wouldn't window.focused in the panel and the use of the visibility api handle this?

>Events:
>    onFocus: callback gets the tabId of the panel.
>    onBlur: callback gets the tabId of the panel.

The content of the panel should be able to handle this without additional APIs

>    onShown: callback gets the tabId of the panel.
>    onHidden: callback gets the tabId of the panel.

These should be handled by using visibility api in the panel content.

Comment 13

[Lusito]

>The comment is confusing, I'm not clear what you mean. 
Say someone provides a small search engine that could be contained in a very small space or something similar.
For my add-on for example, I would like to show http://pocket.dict.cc/ as a "popup" result.
browserAction supports remote documents as well, so I don't see why this one should be different.

>IMO a panel should always be visible (within the tab) until closed or dismissed.
Not sure how quick this is to load, I thought it might help loading times if it wasn't recreated everytime, but just hidden instead. If that's not wanted, I guess a few changes need to be made.

>Wouldn't window.focused in the panel and the use of the visibility api handle this?
My concern was that the content script might not have loaded yet when this is accessed (especially for remote documents). So my background script might never be notified when the panel lost focus or was closed before the content script was running.

As for the rest of your comments: These have the same reason as above..

Comment 14

[WM]

Hope this could be implemented as a replacement for Addon-SDK panel

Comment 15

[Caitlin Neiman [:caitmuenster]]

Hi Lusito, this has been added to the agenda for the June 20 WebExtension APIs triage meeting. I hope you'll be able to join us! 

Wiki: https://wiki.mozilla.org/Add-ons/Contribute/Triage#Next_Meeting

Agenda: https://docs.google.com/document/d/1s2j85VfYKTDftppFU-K7pgRs6sEuMTHVXqf3RL5iYvs/edit#

Comment 16

[Lusito]

I hope I can make it.. depends on when I can get off from work.

Comment 18

[Luca Greco [:rpl] [:luca] [:lgreco]]

I've marked Bug 1364401 a duplicate of this issue, so that we can take into account also that scenario when we will design the "floating panels" API.

Also, I've just pushed on github the sources of a "WebExtension Experiment API" addon which I created sometimes ago to experiment a bit on a similar kind of "Floating/Overlay panels" API:

- https://github.com/rpl/webext-experiment-overlayPanel

I've not yet tried it on a recent version of Firefox, it is completely possible that it needs to be updated to make it compatible with the changes that have been introduced in the meantime in the WebExtensions internals. (I'm going to give it a try as soon as possible).

Comment 19

[Lusito]

I've seen these kinds of api scripts in bug tickets, but I'm unsure how they can be used.. is there a guide somewhere? Or is this only available for mozilla staff?

Comment 20

[Lusito]

So, since I have no idea how to do this in firefox itself, I created a mockup app:
http://lusito.info/panel-action/
It's still missing a couple of things and for the next couple of weeks, I won't have much time on my hands, but feel free to fork it on github:
https://github.com/Lusito/panel-action

Comment 21

[Jonathan Kingston [:jkt] he/him]

As you mention :rpl it looks like this no longer works I get the following error in Nightly.

> TypeError: browser.overlayPanels.create is not a function

I also get various complains about the manifest shape in your example extension.

Thanks

Comment 23

[Stephan Sokolow]

> A window can be used for something that needs to remain open regardless of tabs.

Just something to be aware of:

I use about:config keys (pinned using user.js, in case the browser thinks it's smarter than I am) to aggressively redirect "open new window/tab" to "navigate current tab" unless explicitly triggered by a middle-click, "Open in New Tab" context menu choice, or external `firefox <URL>` process call.

If something finds a way bypass that block, I uninstall it. If I then can't find a replacement which allows me to manually trigger such behaviour ONLY when I want it, I currently find a way to shove it into a sidebar bookmark.

However, once I finish arranging replacements for my legacy extensions and migrate off 52ESR, I'll make do with a browser action that I have to manually re-open every time I interact, since Tree Style Tabs will be taking up my one sidebar slot.

Comment 24

[Tim Nguyen :ntim]

I think an API like this should simply be integrated to the windows API.

Chrome used to properly support the "panel" type for windows.create(), this used to create a floating chat-like panel. Perhaps, that same type could be re-used to create floating panels as proposed here.

Comment 25

[The 8472]

usecase: A virtual keyboard overlaid over some percentage of the webpage.

Sidebars are obviously inadequate for this. Injecting it in the page DOM easily runs into conflicts with pages that do more advanced layout things, that act on keyboard events etc.
Popups are not adequate either because they would not stay floating over the content when you move focus back to the <input> element into which the virtual keyboard is entering characters.

So my requirements aprox. are:

- positionable relative to the content viewport
- always on top of the content, even when content has focus
- no window decorations borders or title bars would take up too much space (depending on platform).
- (optional) support transparent background

The overlays from bug 1340930 seems to be what I actually need, but that's only intended for internal use(?) so I would have to do with the closest approximation available through panels.

Comment 26

[Firefox Bug Husbandry Bot]

Bulk move of bugs per https://bugzilla.mozilla.org/show_bug.cgi?id=1483958

Comment 27

[Bik]

I hope a floating panel API or similar can be implemented again. This was useful before the WebExtension introduction. Thanks for all work on this!

Comment 28

[Sergio]

As it seems some bugs are being closed cause "there has been very little interest" on them i want to confirm that I'm still interested on this bug.

Comment 29

[Lusito]

Caitlin would it be possible to somehow re-raise awareness for this ticket? Maybe at the next "Add-ons Show & Tell"?

It's been quite some time with no progress even though there is a complete API proposal with fully working mock demo that has never been commented on by anyone from Mozilla. There are quite a few developers who are looking forward to an API like this.

http://lusito.info/panel-action/
https://github.com/Lusito/panel-action

Comment 30

[Caitlin Neiman [:caitmuenster]]

Thanks Lusito! This is on Jim & Philipp's (engineering & product management, respectively) radar for discussion next week.

Comment 31

[Lusito]

Thanks Caitlin. Looking forward to the result of that :)

Comment 32

[Lusito]

Jim has anything been discussed about this ticket yet? Just wondering if there's anything I can contribute.

Comment 33

[Jim Mathies [:jimm]]

We're happy to take submissions but the team hasn't prioritized this for internal development.

Comment 34

[Bik]

@Jim Thank you for this feedback, hopefully the priority for this issue can change! Thanks for considering this.
@Lusito Thank you very much for being willing to contribute!

Comment 35

[BugBot [:suhaib / :marco/ :calixte]]

The severity field for this bug is relatively low, S3. However, the bug has 18 votes.
:robwu, could you consider increasing the bug severity?

For more information, please visit auto_nag documentation.

Comment 36

[BugBot (nomail) [:suhaib / :marco/ :calixte]]

The last needinfo from me was triggered in error by recent activity on the bug. I'm clearing the needinfo since this is a very old bug and I don't know if it's still relevant.