1344561 - consider supporting moz-extension:// service workers

Status: REOPENED

(WebExtensions :: Request Handling, defect, P3)

Description

[Mossroy]

Attachment: testcase-serviceworker-webextension.tar.gz

It seems that registering a ServiceWorker from inside a WebExtension is not allowed (i.e when called from a moz-extension:// javascript resource).
It fails with : SecurityError: "The operation is insecure."

I suppose it comes from the security restrictions applied to ServiceWorkers, that require an HTTPS connection (cf https://www.w3.org/TR/service-workers/#security-considerations).

Could it be considered to allow ServiceWorkers for WebExtensions? I don't see the security threat as the code is local (as when executed from http://localhost, which allows ServiceWorkers). Moreover, these extensions are signed by Mozilla.

Steps to reproduce :
- open the attached test-case, and install it in Firefox as a temporary webextension (through about:debugging)
- click on the newly-created "testcase" icon : it opens a new tab with the index.html page, served as moz-extension://uid/index.html
- this page tries to register a ServiceWorker and fails

If you open the same testcase with the index.html served through a local webserver i.e as http://localhost : the ServiceWorker is registered.
If you open the same testcase as an extension on Chromium 56 : the ServiceWorker is registered

Comment 1

[Ben Kelly [:bkelly, not reviewing]]

Our position to date has been explicitly not to allow service workers for non-content origins.  It seems most extension or addon systems have other methods designed for intercepting network traffic.  Those seem more appropriate to use instead of using trying to (ab)use the service worker API.

Loosening our origin restrictions on service workers increases the risk of security bugs.

Can you describe the use case for a service worker in an extension?

Comment 2

[Mossroy]

We're creating a browser extension (at least on Firefox and Chrome) for Kiwix : an offline wikipedia reader (see http://www.kiwix.org/).
Basically, all the content of wikipedia (and some other websites) can be compressed inside a big file (openZIM format) : you manually copy it locally on your computer/device, and the extension should allow to browse its content while being offline.

See the code on https://github.com/kiwix/kiwix-html5 (initially targeted on Firefox OS, still in beta state).

When you read a web page inside a ZIM file, you get the HTML content, which contains links to other pages, and other resources (images, CSS, javascript etc). These resources are also stored inside the ZIM file, so we need a way to "inject" them inside the browser.
Initially, we were doing that with jQuery : parsing the HTML and injecting the content into the DOM. But it can be slow, and you still miss some resources this way (for example a background image set by a CSS).

A Service Worker is the best way we found to do that properly.
Its purpose is to trap all the HTTP requests sent by the page, read the corresponding content from the archive file, and send it back to the browser. It's faster, generic and exhaustive.

Comment 3

[Ben Kelly [:bkelly, not reviewing]]

It seems to me this should be possible with the webRequest API:

  https://developer.mozilla.org/en-US/Add-ons/WebExtensions/Intercept_HTTP_requests
  https://developer.mozilla.org/en-US/Add-ons/WebExtensions/API/WebRequest

But it looks like that API does not support modifying the response body because chrome was worried about perf:

  https://bugs.chromium.org/p/chromium/issues/detail?id=104058#c19

Personally I'd rather see this feature added to webRequest so we don't have to add a bunch of special-case logic to our security-sensitive content API.  If we are trying to maintain compat with chrome, though, that may not be possible.

In any case, the ability to modify the response body is a missing feature.  We should discuss how to handle that.

If the decision is made we need service workers in WebExtensions it will probably not happen for a very long time.  We're pretty buries on the service worker team trying to fix things for out multi-e10s efforts.  Sorry!

Comment 4

[Honza Bambas (:mayhemer)]

When you are designing the API, please be aware of bug 1261585.  

I believe an API you would like to provide is similar to how nsITraceableChannel works today.  We have currently big problems with how the nsITraceableChannel API is defined.  The main problem is that consumers of that API wants to see the final data before we give them to the final listener (like HTML parser, image decoder, etc).  But there are content decoders involved, which are normally added and running on the content process.  But if the parent process wants to see the decoded content and intercept it (by means of ability to modify it), it becomes pretty complicated.

Bug 1261585 outlines the problems and after several months we was not able to find a fix that would not be super-fragile or super-ugly and also work.

Dragana know more than me at the moment (CCed)

Please be careful when designing this API regarding our experience with nsITraceableChannel.

Thanks.

Comment 6

[Ben Kelly [:bkelly, not reviewing]]

I spoke with a chrome engineer about what they do.  I also talked with Shane a bit in the context of bug 1427747.  I think I was confused about what was being asked for.

I incorrectly thought the extension wanted to register a moz-extension:// origin service worker and receive FetchEvents for normal https:// requests from content pages.  Basically I thought folks wanted this in order to implement something like:

https://developer.mozilla.org/en-US/Add-ons/WebExtensions/API/webRequest/filterResponseData

I now understand (I think) that what is desired is a moz-extension:// origin service worker that receives FetchEvents only for network loads to its moz-extension:// origin.  So the service worker is not chosen by the document's controlling service worker, but instead by the URL of the resource load.

To do this we would have to do some largish things:

1. Expand URL protocol restrictions from http/https to include moz-extension.
2. Expose ServiceWorker binding objects in the background moz-extension windows.
3. Make ServiceWorkerManager smart enough to run the moz-extension service worker script in the WebExtensions process.
4. Make the nsIChannel code smart enough to do the foreign fetch style interception for moz-extension URLs.  This might require a new channel implementation if we don't want to put this in SimpleChannel.

I still think its very weird to do the normal service worker life cycle in the context of an extension.  I would propose we also do something like:

5. Add the service worker script to the extension manifest
6. Fire install and activate as part of the extension installation process.  If these fail then the extension installation would also fail.
7. Add code to remove any service workers for an extension if the extension is uninstalled.

Things like (3) will require the service worker e10s refactor to complete first.

Comment 7

[Mossroy]

Thanks a lot for taking the time to think again about this issue.

I think you understood correctly our need. But, to make sure, I'll try to rephrase below :
We don't need to intercept the fetch requests coming from any other window/tab than the one created by the extension (with a moz-extension://xxx prefix).
We only need to intercept the fetch requests coming from an iframe of this tab. Because we populate this iframe with some HTML content we read from an external source. This iframe contains some relative URLs like ../another_page.html, ../some_css.css, ../some_javascript.js. Our extension needs to capture the fetch requests for these URLs (that are with the same moz-extension://xxx prefix), and respond with the appropriate content (also read from an external source).

The HTML content injected in our iframe might also contain some links to absolute public URLs (http:// or https://, or even other schemes). It would be perfect if we could intercept them too, but it is less important.

If necessary, declaring the serviceworker in the extension manifest (5) is not a problem for us. And we probably can handle the lifecycle you propose in (6) and (7) (needs some testing, though).

If you need to test or better understand what we're trying to achieve, you can run the existing browser extension (and choose the ServiceWorker mode) after downloading a simple ZIM file like https://download.kiwix.org/zim/wikipedia/wikipedia_en_ray_charles_2015-06.zim : https://chrome.google.com/webstore/detail/kiwix/donaljnlmapmngakoipdmehbfcioahhk on Chrome, and https://addons.mozilla.org/fr/firefox/addon/kiwix-offline/ on Firefox (both are based on the exact same javascript code)

Implementing this would be very helpful for us, as the same ServiceWorker would work in both Chrome and Firefox, and we don't have another viable solution : the webRequest API can not do what we need (see bug 1427747), and our fallback "jQuery" mode is not sustainable.

Comment 8

[Ben Kelly [:bkelly, not reviewing]]

(In reply to Mossroy from comment #7)
> We don't need to intercept the fetch requests coming from any other
> window/tab than the one created by the extension (with a moz-extension://xxx
> prefix).
> We only need to intercept the fetch requests coming from an iframe of this
> tab. Because we populate this iframe with some HTML content we read from an
> external source. This iframe contains some relative URLs like
> ../another_page.html, ../some_css.css, ../some_javascript.js. Our extension
> needs to capture the fetch requests for these URLs (that are with the same
> moz-extension://xxx prefix), and respond with the appropriate content (also
> read from an external source).

Ok, this matches what Matt on the chrome team told me as well.  They don't do foreign fetch type behavior and require a chrome-extension:// controlled window like you mention here.

> The HTML content injected in our iframe might also contain some links to
> absolute public URLs (http:// or https://, or even other schemes). It would
> be perfect if we could intercept them too, but it is less important.

You would be able to intercept them, but they would be cross origin.  In addition, CORS probably would not work unless the site is using `Access-Control-Allow-Origin:*`.  So the requests would have to be no-cors.  Its not clear how useful this would be if you are stuck only getting opaque responses in most cases.

> Implementing this would be very helpful for us, as the same ServiceWorker
> would work in both Chrome and Firefox, and we don't have another viable
> solution : the webRequest API can not do what we need (see bug 1427747), and
> our fallback "jQuery" mode is not sustainable.

Understood, but we probably will not have bandwidth to work on this for some time.

Comment 9

[Mossroy]

I'm glad to see that bug 1231208 (service worker e10s redesign) has made some progress, which gives hope for this one to be implemented at some point.
I just wanted to say that we're still highly interested in this improvement, for our extension to work as well on Firefox as on Chromium-based browsers.

Comment 10

[[ex-Mozilla] Agi Sferro | :agi]

Another use case for service workers in the extension context is using the Web Push API which is only available in service workers.

Comment 11

[Emmanuel Engelhart]

I would like to put a request for update on this ticket. The Kiwix Wikipedia Offline reader extension still maintains with big effort the support for Firefox via a JQuery based/DOM manipulation hack... but this is really expensive and situation is getting worse over time. Do we have any chance to see this ticket implemented anytime in a similar manner like it works on Chrome?

Comment 12

[alex.parrill]

I'm interested in this as well, for a similar reason: viewing inter-linked web pages offline.

The webRequest API does not look like it supports completely supplanting a request like the service worker fetch event listener can - even if you use the filter content API, Firefox will still fetch the original page, so it won't be able to work offline. Redirecting means breaking links, and even if it didn't, bug #707624 means you can't redirect to data: pages anyway.

I find it frustrating that progressive web apps, something that can be installed simply by visiting a page, have better support for intercepting requests than extensions, which require signing and manual installation.

Comment 14

[Mossroy]

For information, it was becoming too much of a problem for us not to have ServiceWorkers available in our Kiwix offline Firefox extension.

So we implemented a kind of workaround: switch to a PWA version (where ServiceWorkers are fully available).
It's not satisfactory at all, because it means the extension is not 100% offline any more: the user needs a first-time Internet access before being able to use the extension offline. It prevents its usage in some cases (extension manually installed with the .xpi file, in a fully offline scenario)

It was not easy to implement properly (see the size of the PR: https://github.com/kiwix/kiwix-js/pull/771), and forces us to deploy/maintain the PWA too, but at least we can move on and use the same codebase for all recent browsers.

We're still hoping this will eventually be implemented for Firefox. When it will be, our extension should be able to make use of it (we switch to the PWA only if SW registration fails)