1215035 - [meta] Allow classic extensions to use WebExtensions APIs

Status: RESOLVED WONTFIX

(WebExtensions :: Untriaged, defect)

Description

[Wladimir Palant]

Migrating larger extensions to WebExtensions all at once is not going to be realistic, for several reasons:

1. The testing effort after a complete rewrite is simply not manageable.

2. A complete rewrite would bind all resources for a considerable amount of time (months) and severely limit the ability to respond to urgent issues during that time (e.g. Firefox changes).

3. The currently available APIs are very incomplete, yet it is hard to say what exactly is missing without trying it out.

4. Backwards compatibility constrains would cause enormous delays (nobody wants extensions that work only in the most recent nightly builds).

The solution is to migrate extensions gradually, converting parts of the codebase to WebExtensions as the necessary APIs become stable enough. Addressing backwards compatibility will likely mean including parts of the WebExtensions codeback in the extension for Firefox versions that don't have it yet.

This is already possible right now. However, it requires making many assumptions and accessing internal APIs. There should be a proper way to create "hybrid" extensions, this is a meta bug for the work to be done here.

Comment 1

[Erik Vold]

I can do this one.  Andy I just want to confirm that it won't be veto'd?

Comment 2

[Andy McKay]

(In reply to Erik Vold from comment #1)
> I can do this one.  Andy I just want to confirm that it won't be veto'd?

Seems like a good idea to me to allow more access to the WebExtensions API, as long as we understand that this is a temporary step. There's a lot of details that would need to be worked out from this one high level bug though :)

Comment 3

[Kris Maglione [:kmag]]

We should talk about how we want to do this. In particular, it should definitely be done using the mechanism created in bug 1148188 to create, and automatically tear down, the APIs.

We'll also need to decide how isolated we want the code using the APIs to be. Will the code with access to the APIs need to run in isolated globals (namely, background pages)? What mechanisms will it have to communicate with privileged add-on code (message passing? APIs private to the add-on?) ...

I'm available to talk these over in #webextensions

I'm deassigning this because it's a meta bug, but I don't want to discourage you from working on it.

Comment 4

[Wladimir Palant]

(In reply to Kris Maglione [:kmag] from comment #3)
> We'll also need to decide how isolated we want the code using the APIs to
> be. Will the code with access to the APIs need to run in isolated globals
> (namely, background pages)? What mechanisms will it have to communicate with
> privileged add-on code (message passing? APIs private to the add-on?) ...

There is an issue here. If the "classic" part and the WebExtensions part are supposed to be completely isolated from each other then chrome.webRequest has to be used in the latter. However, at least in case of ABP the filters will live in the "classic" part (not going to change for a while, because of backwards compat). chrome.webRequest requires synchronous decisions, so for this to work the "classic" part has to be messaged synchronously...

Comment 5

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Could you have a copy of the filters in both sides?

Comment 6

[Kris Maglione [:kmag]]

(In reply to Wladimir Palant from comment #4)
> There is an issue here. If the "classic" part and the WebExtensions part are
> supposed to be completely isolated from each other then chrome.webRequest
> has to be used in the latter. However, at least in case of ABP the filters
> will live in the "classic" part (not going to change for a while, because of
> backwards compat). chrome.webRequest requires synchronous decisions, so for
> this to work the "classic" part has to be messaged synchronously...

I don't think they need to be completely isolated. Like I said,
I'd like them to be able to communicate, it's just a question of
how, and how much. Some kind of structured-clone-based message
passing would be an obvious start. Allowing the privileged part
of the add-on to define new APIs for the WebExtension part is up
for discussion.

Comment 7

[Wladimir Palant]

(In reply to Bill McCloskey (:billm) from comment #5)
> Could you have a copy of the filters in both sides?

That would mean additional 30 MB of memory and a royal pain keeping both sides in sync.

But that's not even the issue. After thinking through it, having the "classic" and WebExtensions part run in parallel worlds brings us back to square one - no gradual migration part, giving a working extensions to our users requires either full WebExtensions support or none whatsoever.

It goes like that: using chrome.webRequest means that the requests will have to be remembered in content scripts. This in turn means that all functionality currently accessing content (lib/child/ on our e10s branch) will have to run in WebExtensions (so far as expected, I don't think this could be avoided even with my original suggestion). No synchronous communication with the "classic" part means then that the abstract modules making actual decisions will have to move into WebExtensions as well. And that in turn will require the entire UI implementation to be made aware of WebExtensions as well, it currently expects to access core modules synchronously.

Comment 8

[The 8472]

(In reply to Wladimir Palant from comment #4)
> chrome.webRequest requires synchronous decisions

maybe this should be improved then? At least the underlying http channels provided can suspended/resumed, which would allow async handling. sadly no such thing is possible with content policies.

Comment 9

[sblask]

This could possibly solve the problem of https://bugzilla.mozilla.org/show_bug.cgi?id=1214790 ?

Comment 10

[Wladimir Palant]

Yes, it might. This will boil down to a fairly complicated data migration approach however, not sure whether all add-on authors will be happy with it.

Comment 11

[Andy McKay]

Wladimir, does the https://developer.mozilla.org/en-US/Add-ons/WebExtensions/Embedded_WebExtensions meet the needs addressed in comment 0? At this point I'm not sure what else there is we would want to do.

Comment 12

[The 8472]

I have tried writing a hybrid extension recently and ran into the issue of translating internal identifiers (tab/window/request IDs) into their bootstrapped equivalents. I think if i whack the web-ext internals hard enough the information might fall out somewhere, but i think at a minimum the wiki could use a code example of an approved way to get to that data.

Example: Webrequest onbeforerequest is not powerful enough. So i use a contentpolicy. But after recording what i'm interested in I want to do the rest of the work in web-ext. For that I would have to pass the information back through the messaging API and match it up with the web-ext equivalents.

Comment 13

[Andy McKay]

(In reply to The 8472 from comment #12)
> I have tried writing a hybrid extension recently and ran into the issue of
> translating internal identifiers (tab/window/request IDs) into their
> bootstrapped equivalents.

Is there a bug filed for this? I gather people have been thinking about it and it might be a good thing to fix.

Comment 14

[Wladimir Palant]

(In reply to Andy McKay [:andym] from comment #11)
> Wladimir, does the
> https://developer.mozilla.org/en-US/Add-ons/WebExtensions/
> Embedded_WebExtensions meet the needs addressed in comment 0?

Somewhat. With the messaging between the classic and WebExt part being asynchronous, this doesn't really facilitate gradual migration in our case. Problem is that WebRequest requires a synchronous response - so asynchronous messaging won't do the trick. This means that all the data necessary to make a decision needs to live in the WebExt part, and this in turn means that essentially all the code needs to live there. But at this point I no longer believe that we'll be able to do a gradual migration...

Comment 15

[Andy McKay]

Given the 2017 plans, we've got the embedded extensions to provide a transitionary period, but after that classic extensions will no longer be viable.

Comment 16

[Wladimir Palant]

Let's go with WONTFIX - as mentioned a bunch of times, embedded extensions won't do the job for us.