1427747 - provide a way to filter/supply responses to moz-extension protocol requests

Status: RESOLVED WONTFIX

(WebExtensions :: Request Handling, enhancement, P5)

Description

[Mossroy]

We need to trap HTTP requests coming from a webextension to the same webextension, in order to provide the content from an external source.

We first wanted to do that with a ServiceWorker. I opened Bug 1344561 (see its second comment for the explanation of the use-case we have). I was told to use the webRequest API to do that, instead of a ServiceWorker (because they are disabled inside WebExtensions on Firefox).
Unfortunately, I can not make it work because these requests can not be trapped by a webRequest.

Based on https://developer.mozilla.org/en-US/Add-ons/WebExtensions/API/webRequest/RequestFilter :
"Only requests made using HTTP or HTTPS will trigger events, even though match patterns can match some other protocols"
So I suppose it's the expected behavior (not a bug).

Unfortunately, it makes the webRequest API useless for our need.
Could the RequestFilter be opened to other protocols? (at least moz-extension)
Should I reopen bug 1344561? (it's not a duplicate of Bug 1255894 as it does solve the issue)

I've created a test-case if you need : https://github.com/mossroy/webextensions-examples/tree/requestfilter-on-moz-extension-protocol/http-response

Comment 1

[Caitlin Neiman [:caitmuenster]]

Hi Mossroy, this has been added to the agenda for the January 9, 2018 WebExtensions APIs triage meeting. Would you be able to join us? 

Here’s a quick overview of what to expect at the triage: 

* We normally spend 5 minutes per bug
* The more information in the bug, the better
* The goal of the triage is to give a general thumbs up or thumbs down on a proposal; we won't be going deep into implementation details

Relevant Links: 

* Wiki for the meeting: https://wiki.mozilla.org/WebExtensions/Triage#Next_Meeting
* Meeting agenda: https://docs.google.com/document/d/15JYw3L1490dKbr6yTLz1uirH8rfhcSiLJCubSWDlnfs/edit#
* Vision doc for WebExtensions: https://wiki.mozilla.org/WebExtensions/Vision

Comment 2

[Mossroy]

I would be very glad to join this triage meeting.

Unfortunately, I won't be available on January 9 at that time.
It looks like you have this meeting every 2 or 3 weeks : I should be available at the same time on the following Tuesdays : January 16, 23 or 30 (but maybe not February 6)
If it's possible for you to reschedule the discussion about this bug for the following meeting, that would be the best option for me.

If it's not, there is a test-case, and more explanations on why we need it in the second comment of bug 1344561. If you need more information/explanations, I'm available to provide them.

Comment 3

[Caitlin Neiman [:caitmuenster]]

Hi Mossroy! No worries if you aren't able to attend. We'll refer to 1344561 for context and needinfo if any questions come up during the review.

Comment 4

[Mossroy]

OK.
Please keep in mind that, for us, the best solution would be to use the ServiceWorker API, as it also works outside an extension, and in other browsers, and is already mostly implemented.

Comment 5

[Shane Caraveo (:mixedpuppy)]

As was previously discussed in bug 1344561, we "explicitly not to allow service workers for non-content origins".  However, that seems to be a chrome-compat issue if I read bug 1344561 comment 0 correctly.  As well, now that we only have web extensions, it might be easier to open up on that restriction (bkelly?).  Even with access to using service workers, I'm not entirely certain this will work since you'd be using a moz-extension protocol.

The issue with using the WebRequest api is probably that the moz-extension protocol is not handled via nsHttpChannel.  We don't get the same notifications, so it likely doesn't go through the StreamFilter, I'd have to dig to see if/what issues there would be doing that (but maybe kris has some quick input).

Comment 6

[Kris Maglione [:kmag]]

Yes. I'd rather people use service workers than StreamFilter for this.

StreamFilter *could* work with moz-extension requests, in principle, but exposing those requests to the webRequest API will be a major effort.

Comment 7

[Ben Kelly [:bkelly, not reviewing]]

(In reply to Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) from comment #6)
> Yes. I'd rather people use service workers than StreamFilter for this.

Conversely I don't want to weaken invariants that service workers only run with content principals.  Its very security sensitive code and this would complicate things in a way which increase our risk profile across all sites.

Comment 8

[Ben Kelly [:bkelly, not reviewing]]

(In reply to Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) from comment #6)
> StreamFilter *could* work with moz-extension requests, in principle, but
> exposing those requests to the webRequest API will be a major effort.

And doing this for service workers would also be a major effort.  And one that has the potential to break a web exposed, standardized feature.

Comment 9

[Kris Maglione [:kmag]]

I think there's probably a compromise, where an extension should be able to use service workers to generate content with a content principal, and use something like content scripts if it needs slightly elevated privileges. I don't particularly like the idea od dynamic content with extension principals, either.

That said, as far as the service worker code is concerned, extension pages *do* have ContentPrincipals. We just attach an extension ID to them when they come from certain origins, and use that to grant them certain elevated privileges.

Comment 10

[Ben Kelly [:bkelly, not reviewing]]

To clarify we have invariants that we use only https:// content principals.  Things like file:// content principals would be very unsafe with service workers.

Also, I have performance concerns about how this would work.  Does it require extra process hops for every FetchEvent?

If you really want to move this forward someone needs to do a detailed design about how it would work that shows its both safe and performant.  I don't expect the service worker team to have any time for this since we're already overwhelmed trying to keep up with the spec and other browser changes (multi-e10s).

Finally, I think google made a bad design choice integrating extensions and service workers.  Service workers have a lot of complexity oriented towards remotely loaded content sites.  Extensions don't need any of that and can have a simpler system.

Comment 11

[Mossroy]

(In reply to Shane Caraveo (:mixedpuppy) from comment #5)
> However, that seems to be a chrome-compat issue if I read bug 1344561 comment 0 correctly.
Yes it is. You can test by yourself with https://addons.mozilla.org/fr/firefox/addon/kiwix-offline/ and https://chrome.google.com/webstore/detail/kiwix/donaljnlmapmngakoipdmehbfcioahhk : both use the same code. The ServiceWorker mode only works on Chrome.

I understand there are technical difficulties (both for ServiceWorker and WebRequest APIs), even if it's disappointing for us.

Don't forget we only need to trap the local requests (coming from some relative URLs in the DOM of a web page inside the extension) : with this scope, I don't see the security concern.

(In reply to Ben Kelly [:bkelly] from comment #10)
> Extensions don't need any of that and can have a simpler system.
Which system?
We'd be interested in another workaround if there is. Because we don't have any for now, except parsing the DOM to "inject" manually the content (what we called the "jQuery" mode), which is ugly and has many flaws.

Comment 12

[Ben Kelly [:bkelly, not reviewing]]

(In reply to Mossroy from comment #11)
> (In reply to Ben Kelly [:bkelly] from comment #10)
> > Extensions don't need any of that and can have a simpler system.
> Which system?
> We'd be interested in another workaround if there is. Because we don't have
> any for now, except parsing the DOM to "inject" manually the content (what
> we called the "jQuery" mode), which is ugly and has many flaws.

I was not referencing another existing system.  I was trying to say that features like:

* Service worker state lifecycle going from installing, waiting, active
* Automatic remote update checks

Are not needed for a web extension network interception API.  The browser handles updating and installing extensions.  An API used by web extensions does not need any of that complexity.

For example, chrome ran into some confusion around this:

https://bugs.chromium.org/p/chromium/issues/detail?id=533065

(In reply to Kris Maglione [:kmag] (long backlog; ping on IRC if you're blocked) from comment #6)
> StreamFilter *could* work with moz-extension requests, in principle, but
> exposing those requests to the webRequest API will be a major effort.

I've been thinking about this statement some more.  I agree its probably a major effort, but it is a boundable effort.  Ultimately we have the tools in gecko to intercept a network call, run some js, and handle the result.  It could even use primitives like IPCStream we've added for service workers without depending directly on service workers itself.

In contrast, trying to integrate web extension support into service workers creates a difficult to quantify complexity cost.  Consider that we originally shipped service workers in FF42 and we are *still* not comfortable with its stability enough to enable it in ESR.  We have been trying to re-write it for two years now.

In addition, the spec and our implementation continue to change in ways that are oriented towards web content; not extensions.  For example, we just started rejecting CORS responses for same-origin mode requests.  Could we have done this without breaking extensions?  I have no clue and would not know how to answer that question if extensions were involved.  On the other hand, what is the risk changes made for extensions will break content service workers?

I really encourage the web extensions team to investigate an extension-specific solution.  The complexity cost and barriers to continued change caused by coupling these features seems high.  Having separate implementations gives us the most flexibility to make the best design choices for the different use cases.

In terms of "compat with chrome extensions", I'm not sure how important that is.  If its critical we could investigate still exposing an extension-specific implementation through `navigator.serviceWorker` properties, etc.

Those are my concerns and fears anyway.

All that being said, though, if we could define something a bit more concrete maybe its less risky than I think.  I haven't been able to find anything to read that actually describes what chrome does, though.  All I can find are some chromium bugs.  This is still not marked complete:

https://bugs.chromium.org/p/chromium/issues/detail?id=545535

It seems like they are currently working on exposing extension APIs in the service worker thread itself:

https://bugs.chromium.org/p/chromium/issues/detail?id=721147

How stable is the chrome extension+SW stuff?

Comment 13

[Shane Caraveo (:mixedpuppy)]

Having a short chat with robwu since he'd written up a doc about using fetchevent in extensions a while back.  The proposal morphed into something that looks like this:

browser.webRequest.onBeforeRequest.addListener((details) => {
  var status = 200, statusText = ‘OK’, headers = new Headers();
  var response = new Response(body, {status,statusText,headers});
  return {response}; // ← Ability to set Response is new
}, {urls:[‘https://example.com/*’]}, [‘blocking’]);

I wonder if that would be easier to implement than getting streamfilter to work with non-http requests.

Comment 14

[Andy McKay]

Sorry for the slow update.

The general feeling in the meeting was that this was not a good idea for a couple of reasons, if I'm correct, one of them the way HTTP channels are created in this case and how they are different. But mostly that's agreeing with comment 6.

I'd rather recommend we point to a way to integrate with service workers. The comments from Ben (comment 12) are great and insightful and point out that we'll have to think about this carefully and get some smart people involved in a solution.

Comment 15

[Ben Kelly [:bkelly, not reviewing]]

So, I recently spoke with a chrome engineer about how their system works.  He believes that their extension support for service worker only allows the addon to control chrome-extension:// origin pages.  It is not something that lets a chrome extension intercept content page network requests.

Of course, he wasn't 100% sure either because their support for extension service workers is not very well documented.

Shane, do you think this should be re-opened?  It seems the decision in comment 14 may have been based on incorrect data.

Comment 16

[Shane Caraveo (:mixedpuppy)]

I'm torn.   I think either direction is a lot of work, and I'm uncertain of the value.  In a perfect world we'd be able to just expose FetchEvent to our background scripts and have it work with any request the extension has permission for.  Lacking that, I'd go for compatibility with chrome on this.  In either case, we should probably have an open bug tracking support for this functionality.

Comment 17

[Ben Kelly [:bkelly, not reviewing]]

I think the underlying issue is there is no proposed model for "just expose FetchEvent".  From my talks with the chrome service worker team they don't support that either.  So its hard to evaluate what that option even means.

AFAIK both firefox and chrome addons are using redirects to data URLs to achieve this functionality today.

Comment 18

[Tomislav Jovanovic :zombie]

(In reply to Ben Kelly [:bkelly] from comment #17)
> AFAIK both firefox and chrome addons are using redirects to data URLs to
> achieve this functionality today.

Yeah, either data: or blob URIs, but that has security implications, because it formally changes the URI of the redirect.

I'm not sure of the name, but I think we have something like "internal redirects" that are not observable by content, and using that might be the simplest way to achieve this.

Comment 19

[Ben Kelly [:bkelly, not reviewing]]

We do have internal redirects and service worker interception uses them.  But we would have to consider what it means for addons to expose something using those.  They are not an adequate API unto themselves.

BTW, service worker interception now changes the final result URL as well if its provided a Response with a URL.  So its pretty similar to redirecting to a data/blob URL.  (With some exceptions here for navigation interceptions, etc.)

Comment 20

[Tomislav Jovanovic :zombie]

Sorry, I was partly documenting an irc conversation with Shane without citing all the context.  I was thinking in terms of existing webRequest api, where we support redirecting, but which produces an observable URI change, so redirecting to a data: or blob URI doesn't cover all the use cases.

Adding a flag to do the internal redirect under specific conditions (with security in mind) was my proposed "easy" solution that should cover most of those use cases.

Comment 21

[Ben Kelly [:bkelly, not reviewing]]

I re-opened bug 1344561.  See bug 1344561 comment 6.