1712096 - Allow extensions to replace the response body without network activity

Status: NEW

(WebExtensions :: Request Handling, enhancement, P3)

Description

[Rob Wu [:robwu]]

There are extensions that want to replace response bodies without network activity. For example:

• Decentraleyes is an extension that bundles libraries, and redirects common script requests to them.
• Content blockers suck as uBlock Origin redirect some script requests to "surrogates" to block scripts and provide a small replacement to minimize site breakage.

Methods to implement this has varied, but it's generally implemented by using the webRequest.onBeforeRequest API in combination with a redirect to a moz-extension:-URL (or a data:-URL). These methods have disadvantages:

• CSP can block data:-URLs, and prevent the resource from loading.
• SRI can prevent the script response from being accepted. (bug 1321916 offers more context)
• CORS can prevent the resource from being loaded. This was recently resolved for moz-extension: by disabling CORS for web-accessible moz-extension: URLs (bug 1694679). If we offer alternatives that do not require that, then we can consider removing this exception again in manifest v3, possibly with an opt-in (bug 1696580 enables us to add resource-specific properties).
• moz-extension:-URLs require an extension resource to be declared in web_accessible_resources, which allows anyone that get hold of the URL to load the resource.
  • The UUID may inadvertently be leaked, via unexpected features of the browser (e.g. bug 1588957 / bug 1705523) or due to extensions injecting the resource directly.
  • A known web-accessible script can be abused to bypass CSP (if the script/library has a vulnerability/feature that offers such functionality).
  • While there is work in progress to allow extensions to restrict the pages that can access the resource (bug 1697334), it doesn't solve the issue for extensions that want to use the resource in potentially any web page.

There is already the webRequest.filterResponseData API that allows extensions to replace a response body, but this is not a feasible method for extensions that do not want any network request to happen.

In order to allow extensions to replace the response body without privacy or security risks, we should introduce an API that allows extensions to replace response bodies for specific requests, e.g. at the onBeforeRequest stage. This can be part of the blocking webRequest API and/or Declarative Net Request (DNR - bug 1687755).
Binding the API to the webRequest/DNR APIs allows us to offer the desired functionality to extensions (and offer request-specific opt-out of security features such as SRI, CORS, CSP) without requiring us to weaken security aspects for all web-accessible resources.

The exact shape of the API will be formed when this patch is being worked on. It might be nice to use the existing Response type from the web platform, but if that's not feasible, then a bunch of new properties is also okay.

browser.webRequest.onBeforeRequest.addListener(details => {
  return {
    // TODO: some new property to set the Response body.
  };
}, {
  urls: ["https://example.com/script.js"],
  types: ["script"],
}, ["blocking"]);

Comment 1

[Thomas Wisniewski [:twisniewski]]

This will also be very useful for SmartBlock and Tinker Tester Developer Spy.

The ability for this to process POST requests and serve varying responses will be very powerful.

Note that it would be very handy to be able to specify not just the response body, but also its headers.

Comment 2

[Arthur Edelstein [:arthur]]

This proposed enhancement would be very useful to allow extensions to support alternative and experimental protocols, including anonymizing networks such as Tor, decentralized web protocols such as ipfs, and dat, and decentralized naming protocols such as .bit (Namecoin).

For these applications, the ability to stream a dynamically-generated response would be critical. Also it would be good to be able to suppress DNS/OCSP requests associated wtih HTTP requests that are being overridden. In other words, effectively a dynamic "proxy".

Comment 3

[Karl Dubost💡 :karlcow]

I wonder if this would help to have an even better DITM addon
and I wonder if the webcompat team had opened a bug on overriding resources in the devtools.

Comment 4

[Karl Dubost💡 :karlcow]

arai-san, do you think that could help DiTM too. (I think it is broken now).

Comment 5

[Tooru Fujisawa [:arai]]

(In reply to Karl Dubost💡 :karlcow from comment #4)

arai-san, do you think that could help DiTM too. (I think it is broken now).

I haven't hit the trouble due to reasons listed in the comment #0 (CSP, SRI, CORS, etc) personally, so I'm not sure if this API improves the DITM feature (maybe you hit such issue?).

but if the modification can be done without actual network traffic, as described in comment #0, that's more straight solution for the original purpose of the extension,
so, in term of simplicity of the extension code, the API helps much,

and also the API could improve user experience on slow server/network or large file, given this API doesn't have to wait for the actual server response.

btw, if you found any breakage in DITM, can you open an issue on github?

Comment 6

[KOLANICH]

CSP can block data:-URLs, and prevent the resource from loading.

It is a good thing allowing stacking malicious content blockers and ensuring each one works.

SRI can prevent the script response from being accepted. (bug 1321916 offers more context)

It is an issue. In my fork (the original extension seems to be prety unmaintained and the author has said me that he won't merge my changes (because doing it means maintaining the extension) and to create a new one on AMO, but I'm not going to do it because of AMO harmful policies enforcing 2FA, instead I recommend users to disable extension signing (it is possible even in release builds by patcching the browser resources) and install extensions not from AMO) of ResourceOverride I had to work around it by using MutationEvents (unfortunately MutationObservers can only bubble and cannot capture, so by the time I get the info, the resource is already set and blocked by SRI, MutationEvents allow me to interfere before the request happening, so it[s a shame they are deprecated and gonna be removed).

A known web-accessible script can be abused to bypass CSP (if the script/library has a vulnerability/feature that offers such functionality).

Web-accessible resources access to which is allowed by default were a mistake. Web-accessible resources should have an API to enumerate them and give access to a page to them selectively by mapping a resource object (it should also be possible to create an object from a Blob) to page-visible URI. Usually not during a webrequest, but during an extension start up. After a resource is registered in runtime, it becomes available to a page. The resources shipped within an XPI still have to be registered within manifest, but this doesn't make them available to pages by default. Also instead of registering the resources directly within manifest it should be possible to refer a JSON file within an XPI. A dev should have a way to add arbitrary JSON-serializeable metadata to each resource within the JSON.

This way we want to

• provide the devs with convenient API to use the preregistered resources: they don't have to scan the dir themselves, they don't have to create the resource objects themselves, they only have to iterate a list of resource objects and decide for each if it is available to the web page. No need to mess with webRequest and potentially introduce timing attack-suspectable side channels.
• encourage them to register the static resources within manifest, this way enabling external introspection of this list the same way web_accessible_resources allow it. Of course they can easily avoid registering, but enforcing this is out of scope of the extension level, AMO can enforce this requirement for example by static analysis and human vetting.

Comment 7

[KOLANICH]

Qlso

CSP can block data:-URLs

data URIs should be avoided because they are inefficient.