1296885 - WebExtensions doesn't have a means to setup a protocol handler

Status: RESOLVED DUPLICATE of bug 1271553

(WebExtensions :: Request Handling, defect)

Description

[Cameron Kaiser [:spectre]]

+++ This bug was initially created as a clone of Bug #590682 +++

Per bug 590682, 70% of the top 30 addons have a protocol handler of some sort. There is currently no pathway for these addons to migrate to WebExtensions, at least not based on the MDN docs as of this writing.

There needs to be a way to establish the protocol handler, and for those extensions that implement channels or use sockets, a means to pass network data through the addon to the browser.

Comment 1

[Tom S [:evilpie]]

Could somebody clarify that 70% and actually look into if those protocol handlers are required or if the same thing could be solved differently. I wanted to check that, so I downloaded the 20 most popular addons from amo and only Greasemonkey seems to implement an nsIProtocolHandler.

Comment 2

[Kris Maglione [:kmag]]

We'll discuss this, but I should warn you at the outset that this will almost certainly not happen.

Comment 3

[Boris Zbarsky [:bzbarsky]]

I would very much like to know what sort of protocols are being implemented in addons here.  If nothing else, whether they only need navigation to work or general fetches is relevant for some standards work we're doing...

Comment 4

[Andrew Swan [:aswan]]

(In reply to Cameron Kaiser [:spectre] from comment #0)
> There is currently no pathway for these addons to migrate to
> WebExtensions, at least not based on the MDN docs as of this writing.

That's certainly not true.  Webextensions capabilities like web_accessible_resources (https://developer.mozilla.org/en-US/Add-ons/WebExtensions/manifest.json/web_accessible_resources) cover some (hopefully many or most) uses of protocol handlers.

I'm sure there are add-ons that do other things that aren't currently possible in webextensions but I suspect this among the top uses and it is covered.

Comment 5

[Cameron Kaiser [:spectre]]

(In reply to Boris Zbarsky [:bz] from comment #3)
> I would very much like to know what sort of protocols are being implemented
> in addons here.  If nothing else, whether they only need navigation to work
> or general fetches is relevant for some standards work we're doing...

Besides Chatzilla and IRC, there is also (disclaimer: this is an addon I maintain) OverbiteFF and Gopher, PCMAN BBS and Telnet, and FireFTP and FTP. This was just a cursory initial search of AMO, along with https://addons.mozilla.org/en-US/firefox/search/?q=protocol&appver=45.0&platform=mac(In reply to Andrew Swan [:aswan] from comment #4)

(In reply to Andrew Swan [:aswan] from comment #4)
> > There is currently no pathway for these addons to migrate to
> > WebExtensions, at least not based on the MDN docs as of this writing.
> 
> That's certainly not true.  Webextensions capabilities like
> web_accessible_resources
> (https://developer.mozilla.org/en-US/Add-ons/WebExtensions/manifest.json/
> web_accessible_resources) cover some (hopefully many or most) uses of
> protocol handlers.
> 
> I'm sure there are add-ons that do other things that aren't currently
> possible in webextensions but I suspect this among the top uses and it is
> covered.

Given that chrome:// and resource:// URLs already exist and do what it appears this feature does, I submit the reason that addons are using actual protocol handlers purely for internal resources is that the use case is not, in fact, covered.

Comment 6

[Cameron Kaiser [:spectre]]

(And more to the point, it doesn't cover the case of those addons that actually do implement _network_ protocols instead of using a protocol handler as essentially an alias.)

Comment 7

[Boris Zbarsky [:bzbarsky]]

OK, so the irc case is basically a navigation case.  So is telnet.  ftp can probably be more or less navigation.  Gopher I expect needs fetch hooks, right?

Put another way, there are actually two separate problems that are both solved by protocol handlers right now but don't have to be solved via the same mechanism in general:

1) Some way to hook navigation to a URL that allows the hooker to respond with some arbitrary HTML/whatever content, running with either system permissions or not.

2) Some way to hook subresource (image, script, etc) fetches from a url.

Generally speaking, a solution for #2 would also solve #1, but not vice versa, right?

Comment 8

[Cameron Kaiser [:spectre]]

Gopher would need a fetch hook. Right now it goes out, gets either a menu or a resource, and either formats the menu as HTML (and returns that) or gives the resource back as raw data. That would probably work also for FTP. So I guess that would be your option 1, but I'm not sure what option 2 means.

Comment 9

[Boris Zbarsky [:bzbarsky]]

Option 2 is what you need to make <img src="gopher:whatever"> work (and similar for <script>, <video>, etc).  I think gopher needs that, but none of irc/telnet/ftp do.  For those it's enough to make <a href="irc:whatever"> and maybe <iframe src="irc:whatever"> work, so for those hooking navigation but not fetch is enough.

Comment 10

[Cameron Kaiser [:spectre]]

I get it now. Yes, option 2 would be the most seamless approach (it's more or less how I have it implemented now and those things all work), but if option 1 were the only choice it would still be substantially better than nothing, of course.

Comment 11

[Honza Bambas (:mayhemer)]

Attachment: addons-implementing-nsiprotocolhandler.txt

This is the list of addons from the top 30 that do implement their own nsIProtocolHandlers

Comment 12

[Jason Duell]

Update: we threw the whole necko team into looking at the top 30 addons list in comment 11, both to test for e10s bustage and examine how they use nsIProtocolHandler. The status is actually much better than we were expecting. We checked 27 of them (3 weren't on AMO) and all of them actually are either already working under e10s or have plans to work under e10s (or are abandoned and already don't work on nightly with or without e10s turned on). FireFTP, for instance, does all its work in the Chrome/parent process, so even though it implements a full protocol stack on top of sockets, it still works.

There are 16 more addons with >10K users which also use nsIProtocolHandler, and we'll be looking at them as well in the next few days.  Beyond that, for reference, there are 32 addons with between 1K and 10k users, and 41 with less than 1K users.  We're not planning to look at all of those.

There are clearly some addons that will break: overbite (gopher://) is the one we know about for sure.  But I think at this point the addons bustage is low enough that we can conclusively say that we don't need to block e10s on providing a webextension API.  (Of course we'd like to have one as soon as possible).

As far as what a webextension API should look like, we might as well make it compatible with the Electron API for protocol handlers:

  http://electron.atom.io/docs/api/protocol/

I'm not 100% clear on whether that API is useful for the Overbite case. It looks like if we also provide some sort of socket access, the 'buffer' version of the API might work,  assuming you can call the 'callback()' function asynchronously--I'm not sure from the docs.  Of course we'd need to think carefully about how to provide a socket API to addons.

Comment 13

[Cameron Kaiser [:spectre]]

(In reply to Jason Duell [:jduell] (needinfo me) from comment #12)
> FireFTP, for instance, does all its work in the Chrome/parent process,
> so even though it implements a full protocol stack on top of sockets, it
> still works.

Is there a way to force this to happen with a more generic approach? It sounds like the only reason it happens that way is because it opens a specific XUL chrome interface (like Chatzilla), and that does all the socket work.

> As far as what a webextension API should look like, we might as well make it
> compatible with the Electron API for protocol handlers:
> 
>   http://electron.atom.io/docs/api/protocol/
> 
> I'm not 100% clear on whether that API is useful for the Overbite case. It
> looks like if we also provide some sort of socket access, the 'buffer'
> version of the API might work,  assuming you can call the 'callback()'
> function asynchronously--I'm not sure from the docs.  Of course we'd need to
> think carefully about how to provide a socket API to addons.

Actually, I think that general idea could work (looking at |registerBufferProtocol|). There would need to be a socket API, but this idea could conceivably do its socket work, create a Buffer with the response and set the equivalent MIME type based on the Gopher item type, and then hand that to the callback once the socket transaction was complete.

This wouldn't work for something more interactive, and Gopher is a very trivial protocol, but it would at least cover this use case.

Comment 14

[Jason Duell]

> > FireFTP, for instance, does all its work in the Chrome/parent process, so
> > even though it implements a full protocol stack on top of sockets, it still
> > works.
> 
> Is there a way to force this to happen with a more generic approach? It sounds
> like the only reason it happens that way is because it opens a specific XUL
> chrome interface (like Chatzilla), and that does all the socket work.

:billm, is there any straighforward way to migrate content from a child process to the parent?  I.e. could we recognize that a particular protocol (like overbite's gopher://) needs to run in the parent and automagically have any top-level navigation to such a URI switch to use the parent process? (we don't need to support loading gopher:// subresources within an http:// page).

I'm not suggesting we implement it for gopher if it doesn't exist already.

Comment 15

[Jason Duell]

:bz might know the answer here too (see comment 13 onwards).

Comment 16

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

The code in E10SUtils.jsm determines which process a URL should run in. It's kind of ad-hoc right now. Bug 1095484 tried to add a more general mechanism, but it got bogged down somehow. I don't remember the details. The bug has more information.

In general, though, we do have a mechanism to switch which process we load a top-level document URL in.

Comment 17

[Cameron Kaiser [:spectre]]

(In reply to Bill McCloskey (:billm) from comment #16)
> The code in E10SUtils.jsm determines which process a URL should run in. It's
> kind of ad-hoc right now. Bug 1095484 tried to add a more general mechanism,
> but it got bogged down somehow. I don't remember the details. The bug has
> more information.
> 
> In general, though, we do have a mechanism to switch which process we load a
> top-level document URL in.

What about reversing that bug so that an add-on would have to request to load in the parent? That should require a much smaller patch and should be simple for add-ons to adopt (if I read it right, I could simply add the "must run in parent" flag and that would be enough). If this seems worth exploring, I can follow up in that bug and leave this for the WebExtensions case.

Comment 18

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Yes, please follow-up with Blake in that bug.

Comment 19

[Cameron Kaiser [:spectre]]

Looks like bug 1277410 was where that work was being considered.

Comment 20

[Jason Duell]

I'm dup-ing this to bug 1271553.  There's a lot of good info in both bugs, but that one seems further along (has a patch).