1323433 - Implement a declarative content script API

Status: RESOLVED WONTFIX

(WebExtensions :: General, enhancement, P2)

Description

[Rob Wu [:robwu]]

A declarative API for registering content scripts can benefit user script managers such as GreaseMonkey and Tampermonkey (bug 1305856).

Chrome has some declativeContent APIs - https://developer.chrome.com/extensions/declarativeContent, consisting of the following APIs:

Matching on conditions (URL/css):
- chrome.declarativeContent.onPageChanged
- chrome.declarativeContent.PageStateMatcher

Actions (+bugs to missing/broken functionality):
- chrome.declarativeContent.SetIcon (https://crbug.com/462542)
- chrome.declarativeContent.ShowPageAction (https://crbug.com/429603)
- chrome.declarativeContent.RequestContentScript (https://crbug.com/377978)

Chrome's implementation is designed to be fast for the browser to support (where browser = Chrome, not necessarily Firefox), but it is not as developer-friendly as it can be (e.g.  strings as content script code are not supported - https://crbug.com/377978).
Therefore we should explore other options before committing to implementing declarativeContent.

I have user script managers in mind, so the API requirements are:

1. Must support loading content scripts at document_start, document_end and document_idle.
2. Must support loading content scripts that are not bundled with the extension.
3. Must support loading content scripts based on URLs.
4. Must support content scripts in child frames (not just the main frame).
5. Should execute the content script only once per page load if a condition matches multiple times.
6. Should be effective when the browser just starts up (before the background page has loaded).
7. Should allow the browser to not regress performance on pages where the content scripts are not needed.
8. Could support loading content scripts based on URL changes that do not reload the page, e.g. through the history.pushState, history.replaceState APIs or location.hash changes.

Chrome's declarativeContent API only supports 3, 6, 7 and 8 (and fails at 1, 2, 4 and 5).


Before adding new APIs, let's see how the requirements can be satisfied with the current APIs:


## Option 1 (existing APIs): Using browser.webNavigation events + browser.tabs.executeScript
(satisfies all requirements except 6).
Firefox's current implementation does not support injection of scripts at document-start.
Events that trigger before the page is loaded (webNavigation.onBeforeNavigate and sometimes webRequest.onCompleted) are run too early, for more details see bug 1290016.

Even if there was an event like browser.tabs.hypothethicalOnDocumentStart event, then the asynchronicity of the APIs means that the script is not guaranteed to execute at document-start.

And finally, to support custom content scripts, this API involves lots of IPC to send the content script from the extension process to the browser, from the browser to the content process. So from the performance POV this option is terrible.


## Option 2 (existing APIs): Using content_scripts + synchronous XMLHttpRequest (+webRequest...)
(does not work at all due to bugs/features in Firefox)
With a manifest-declared content_scripts, the user script manager can run at document-start (or later if wanted). Then the user script manager can send a synchronous XMLHttpRequest, which the background page intercepts and redirects to the actual user script code from the extension.
In theory, this should work with the existing APIs. It works in Chrome, but not in Firefox (webRequest seems not fired for synchronous XHR from content scripts. Also, redirecting xmlhttprequest to data:-URLs does not work (the load just fails - "NetworkError: A network error occurred").

Besides not working, this hacky method does not solve the IPC issues from the previous section.


## Option 3 (new APIs): support extended version of declarativeContent.RequestContentScript
Support the full declarativeContent API with additions to support the other requirements.

Requirements 1, 3 and 5 are straightforward.
Requirement 2: Allow "scripts" to take a list of code in addition to files. The existing content script execution logic in ExtensionContent.jsm already assumes that js is a URL, so from Firefox's implementation POV, this is doable.
Requirement 8 is nice-to-have but not a blocking requirement. AFAIK user script managers match based on the URL at page load and ignore later URL changes.
Requirement 4 does not really fit in Chrome's API design, because the onPageChanged trigger is based on the tab URL, not frame URLs. Maybe we should add a new property frameUrl to declarativeContent.PageStateMatcher ?


## Option 4 (new APIs): add a way to synchronously read content script options.
Instead of adding a complete new API of content script injection, we can also add an API to allow extension pages to write new values and make those available from content scripts.

Example API:
// background.js, (browser action) popup.js, (options_ui) options.js, or any other extension page
browser.contentScriptOptions.set({
   // tabId: someInteger,          // Optional, restrict to tab.
                                   // Default = apply to all tabs.

   matches: ['*://example.com/*'], // Restrict to specific origins/URLs.
   key: 'myContentScript',
   value: '// some code',          // Strings only, for ease of implementation.
                                   // Extensions can always use JSON.stringify if wanted.
});


// contentscript.js (content scripts).
// MAYBE also in extension pages if it does not complicate the implementation?
// Maybe follow the API of browser.storage.local.get ? I.e. supporting {key:defaultValue}, 'key' and ['key', 'anotherKey'] to retrieve values.
var contentScriptOptions = browser.contentScriptOptions.get('myContentScript');
// ^ frozen object.
// ^ throws if the key is not found?
// ^ Behind the scenes, the value is cached (=== may be true or false, whatever is easiest to implement).

// MAYBE also support detecting changes (not required - extensions can use extension messaging to signal changes):
// browser.contentScriptOptions.onChanged.addListener

manifest.json:
"content_scripts_options": {
    "myContentScript": {
        "maximumSize": 5000000,             // The maximum size for the options, in bytes.
                                            // (note: not necessarily equal to # of characters, e.g. with UTF-16)
        "matches": ["*://*.example.com/*"], // Restrict to the following origins/URLs.
        "persistent": true,                 // Whether to persist the value across browser restarts and add-on updates.
        "synchronous": false,               // Whether to make the key available via the synchronous API.
    },
},

Comment 1

[:shell escalante]

complex - but not P1/P2... but want sometime

Comment 2

[u287251]

For what it's worth, I like very much option #4.

The way I see it, something as simple as a mere localStorage-like API available from both an extension background page and an extension's content script instance would help solve a lot of headaches when it comes to easily exchange information between an extension background page and its content scripts in a non-asynchronous manner. Such a localStorage API would solve the user script issue *and* other issues which can't be solved by any other mean than synchronous communication between an extension's content scripts and its background page.

Not being able to avoid round-trip messaging such as [content script world => background world => back content script world] makes it impossible to accomplish some tasks properly, such as executing dynamically generated content script code at document_start time[1]. But such storage would have other uses than just user scripts.[2]


[1] Such limitation is one of the reason uBO-Extra extension is needed on Chromium (related issue: https://github.com/uBlockOrigin/uAssets/issues/227).

[2] For instance, consider this issue: https://github.com/gorhill/uBlock/issues/728. If uBO's content script was able to fetch the whitelisted status of a page from such localStorage-like API, there would be no need for the round-trip messaging, and no need for the extension subsystem to be setup -- the content script would just abort itself with no further overhead.

Comment 3

[The 8472]

Synchronous storage gives us configuration and logic-based decision-making on |document_start|. The question is how dynamic script-loading should be implemented.

Greasemonkey loads scripts stored in the filesystem via Cu.Sandbox + subscript loader to let each script + their dependencies run in clean contexts without scripts interfering with each other or the extension itself.

Tampermonkey seems to use eval() in the content script sandbox, which strikes me as suboptimal on several levels, especially since it would mean storing the source strings in that storage.

So instead of trying to build some high level API on the background page level Option 4 + exposing some pared-down version of the Sandbox constructor + subscript loader on the content script side may also do the job. In general sandboxes might be useful for execution of untrusted code.

Comment 4

[ntninja]

Hi! Just wanted to chim in and mention that I am affected by this with my extension "User-Agent Switcher": When you use that extension to set a different User-Agent and visit http://www.runoob.com/try/try.php?filename=try_nav_all , then the original User-Agent (and other values) will be displayed, because (of course) the content script to override those values is run too late to matter anymore.

I implemented a workaround based on option 2 for this that will ship soon:
https://gitlab.com/alexander255/user-agent-switcher/commit/847d91e6c24796d5c960cb6f76e3fff03d50e8a3
I'm neither proud of that workaround and don't think it should be necessary just to have access to the extension settings (it's basically a poor-mans CPOW-replacement), so I thought you should know that people will do / are doing these things for lack of a better alternative. Peviously I had used `browser.tabs.executeScript` which had the great advantage that I would not have to load *any* content-script (or the tab tracker for that matter) as long as the user didn't choose an overriding User-Agent from the list.

(Note that my particular use-case may also be served by https://bugzilla.mozilla.org/show_bug.cgi?id=1414078 .)

Comment 6

[Mike Conca [:mconca]]

A November 2017 analysis of the Chrome extensions webstore shows that the set of declarativeContent API are the #111 to #115 most popular API, used by 566 extensions that represent 1.2M users.

Comment 7

[Rob Wu [:robwu]]

The RequestContentScript action is just one of the available actions in declarativeContent, for more actions see https://developer.chrome.com/extensions/declarativeContent#event-onPageChanged
The supported actions are:
declarativeContent.RequestContentScript
declarativeContent.SetIcon
declarativeContent.ShowPageAction

Did you just look at requested permissions, or specifically the declarativeContent.RequestContentScript substring?

Comment 8

[Mike Conca [:mconca]]

(In reply to Rob Wu [:robwu] from comment #7)
> Did you just look at requested permissions, or specifically the
> declarativeContent.RequestContentScript substring?

I am viewing this bug in the larger context of the declarativeContent API, not just for the RequestContentScript action.  That said, I looked at every chrome.declarativeContent.* API that is available and falls within the top 250 used Chrome APIs.  The specific results are:
# 111 - chrome.declarativeContent.onPageChanged.removeRules
# 112 - chrome.declarativeContent.PageStateMatcher
# 113 - chrome.declarativeContent.onPageChanged.addRules
# 115 - chrome.declarativeContent.ShowPageAction

As you can see, ShowPageAction is the only action that shows up in the list, with neither RequestContentScript nor SetIcon showing up in the top 250 API.  When looking at permissions, approximately 800 extensions request declarativeContent.

Comment 9

[Rob Wu [:robwu]]

Does it really make sense to implement declarativeContent.RequestContentScript when Firefox already has the contentScripts.register API (bug 1332273)?

The implementation in Chrome is not officially supported:
"WARNING: This action is still experimental and is not supported on stable builds of Chrome."
https://developer.chrome.com/extensions/declarativeContent#type-RequestContentScript

and there is no solid commitment from Chrome's end to keep supporting the API:
"Since the documentation is clear that this isn't supported (even though it is available), we can still make changes if we need to in the future"

Chrome's implementation has issues too:
- only files are supported (strings are not): https://crbug.com/377978
- CSS injection is not supported: https://crbug.com/708115
- runAt is not supported.

The only feature that the declarativeContent API provides that is not directly offered by contentScripts.register is to inject scripts based on css rules. However, that can easily be implemented by extension authors in content scripts too.


I think that this bug can be closed because bug 1332273 provides the desired functionality from the initial report.

Comment 10

[Mike Conca [:mconca]]

(In reply to Rob Wu [:robwu] from comment #9)
> I think that this bug can be closed because bug 1332273 provides the desired
> functionality from the initial report.

contentScripts.register (bug 1332273) is *SO* close, but I think there are still reasons for implementing declarativeContent:

1) It doesn't match CSS rules, as pointed out in comment 9. Doing this in the browser is (theoretically) much more performant than extensions injecting themselves into every page and using Javascript to decide on actions.

2) Depending on what the content script does, using declarativeContent can avoid the need to declare host permissions in the manifest file. This actually might be the most important reason - users being scared off by broad host permission prompts is one of the biggest complaints developers have.

3) Supporting declarativeContent allows developers to maintain a single source code base for Chrome and Firefox. Yes, the RequestContentScript action is currently marked experimental, but adoption by Firefox might encourage Chrome to make part of their release channel. That would be a win for developers. 

Ideally, Firefox would offer a declarativeContent API for content script injection that is backward compatible with Chrome's implementation, doesn't suffer from any of the Chromium issues identified in comment 0, and at the same time, is more powerful (e.g. allows the use of runAt).  This is, I think, what Option #3 in comment 0 is proposing.

Comment 11

[Rob Wu [:robwu]]

(In reply to Mike Conca [:mconca] (Denver, CO, USA UTC-6) from comment #10)
> 1) It doesn't match CSS rules, as pointed out in comment 9. Doing this in
> the browser is (theoretically) much more performant than extensions
> injecting themselves into every page and using Javascript to decide on
> actions.

How common is it for extensions to match on a subset of CSS selectors and (repeatedly) run a script based on that?

I don't know how you're getting statistics, but you'll need to look for PageStateMatcher AND RequestContentScript AND css["']:?  to find code like this:

conditions: [ ...
  new chrome.declarativeContent.PageStateMatcher({ css: ... }) ...
],
actions: [ ...
  new chrome.declarativeContent.RequestContentScript

If the number of stars are any indication of the popularity of a feature under development, then 30 stars in 4 years is meager: https://crbug.com/377978

I am also interested in knowing how much the "css" PageStateMatcher feature is used in general (e.g. for the declarative pageaction actions).
 

> 2) Depending on what the content script does, using declarativeContent can
> avoid the need to declare host permissions in the manifest file. This
> actually might be the most important reason - users being scared off by
> broad host permission prompts is one of the biggest complaints developers
> have.

Is this comment specific to the RequestContentScript action? In this case an extension does still need the permission before a script is executed.
If this is about the declarative pageaction actions, we need to take care of the following:

declarativeContent.PageStateMatcher has very flexible CSS and URL matching options.

declarativeContent.ShowPageAction offers the ability to show/hide the pageAction button based on the above pattern, regardless of extension permissions.

Unlike Chrome, Firefox offers pageAction.isShown that allows extension to determine whether the page action is shown.
If we land this declarativeContent API, we should make sure that the pageAction.isShowing API does not reveal the state of tabs that the extension cannot normally access (otherwise it would be a privacy/security issue).

> 3) Supporting declarativeContent allows developers to maintain a single
> source code base for Chrome and Firefox. Yes, the RequestContentScript
> action is currently marked experimental, but adoption by Firefox might
> encourage Chrome to make part of their release channel. That would be a win
> for developers.

Contrary to the documentation, the API is already available on Chrome's release channel.
I want to understand that the developer expectations of this (poorly documented and buggy) API match with the actual behavior of the API.

For pageactions actions, the exact implementation is not that significant, because the changes are not observable by extensions.
For RequestContentScript, the script is run again whenever the match state changes (not only CSS changes, but also URL changes, e.g. through history.pushState/replaceState).  This is more powerful than the contentScripts API, which only injects when a page is loaded.
I want to know whether extension developers use RequestContentScript to have a declarative content script API (in that case, they should use browser.contentScripts API) or whether they use it to run code whenever the page state (URL/CSS) changes (which is more comparable to the dynamic content script injection API via tabs.executeScript).

> Ideally, Firefox would offer a declarativeContent API for content script
> injection that is backward compatible with Chrome's implementation, doesn't
> suffer from any of the Chromium issues identified in comment 0, and at the
> same time, is more powerful (e.g. allows the use of runAt).  This is, I
> think, what Option #3 in comment 0 is proposing.

Yes, that is what option 3 is proposing.

Comment 12

[David Durst [:ddurst]]

Why would we not pursue option 3? I'm less interested in existing dev experience with this (wrt RequestContentScript) than with what it should be doing.

Comment 13

[Rob Wu [:robwu]]

(In reply to David Durst [:ddurst] from comment #12)
> Why would we not pursue option 3?

"Option 3" from comment 0 is not the same as "Implement RequestContentScript".

Comment 0 was a quest to get user script managers to work. These only need URL-based matchers, which can be checked at initial document load. This is how requirements 1 and 5 are automatically satisfied:

> 1. Must support loading content scripts at document_start, document_end and document_idle.
> 5. Should execute the content script only once per page load if a condition matches multiple times.

However, with the introduction of the contentScripts API in bug 1332273, and the even more specialized userScripts API in bug 1437098, there is no need to meet these requirements.


The objective of this bug has now changed. The desire is to implement "declarativeContent" to improve the API compatibility between Chrome and Firefox. The most significant difference is that scripts could be injected more than once (and in quick succession) when the URL and/or CSS changes.


> I'm less interested in existing dev experience with this (wrt RequestContentScript)
> than with what it should be doing.

"what it should be doing" is open to interpretation.

- Chrome's documentation: unspecified (officially: not supported on release).

- Chrome's original intention:
  * Basically what we implemented in our contentScripts API: https://crbug.com/377978#hc0
  * Inject scripts only once per page, even if matched multiple times: https://crbug.com/377978#c1

- Chrome's implementation:
  * Inject scripts multiple times whenever the state changes.
  * Works on release. Has many bugs.

Chrome's idea of what the API should be doing is inconsistent (and long-term API stability is therefore at risk).
To improve API compatibility, we need to look at the developers' expectations instead.

Comment 14

[Shane Caraveo (:mixedpuppy)]

I’ve been thinking about the declarative content stuff.

This api is really strange.  Why wouldn't it just be implemented like any other listener:

browser.tabs.onPageChanged.addListener(details => { ... }, filters);

where filters is:
{
  urls: array of match patterns,
  css: array of compound css selectors,
  options: object with other options (windowId?)
}

I do really like the idea of an onPageChanged event, *IF* we can do it in a very efficient way.  Maybe someone who knows the DOM/CSS stuff well could chime in.

I care less about the “actions”, though compatibility would be nice, I wonder if it could be polyfilled given css selector support.  IMO an event listener function allows an extension to avoid content scripts but still react to page changes.

I don't think that the ability to avoid a couple permissions is really important, but again...compatibility.  Perhaps the listener on an API like this could simply gain any temporary permissions necessary for compatibility via a polyfill.

Comment 15

[Andrew Swan [:aswan]]

(In reply to Shane Caraveo (:mixedpuppy) from comment #14)
> This api is really strange.  Why wouldn't it just be implemented like any
> other listener:

Well from the name, the point is that it is declarative -- no extension code needs to execute in order to do simple common operations.

Comment 16

[Rob Wu [:robwu]]

I have analyzed the API usage of the declarativeContent API of extensions in the Chrome Web Store.

1065 extensions (11.1M users) in the Chrome Web Store request the declarativeContent permission.
671 extensions (1.6M users) actually use the declarativeContent API.
22 extensions (63k users) use the declarativeContent.RequestContentScript API.
20 extensions (63k users) use the "js" property.
5 extensions (1.5k users) use the "css" property (1.5k users), even though this is not supported by Chrome.
No extension uses the "matchAboutBlank" property.
3 extensions use the "allFrames" property (of which 2 change the default from false to true).

Because Chromium's API has significant flaws (see last part of comment 13) and the API has relatively low usage, we are not going to implement the declarativeContent.RequestContentScript API.

Developers who seek a declarative content script API can already use browser.contentScripts.register instead:
https://developer.mozilla.org/en-US/Add-ons/WebExtensions/API/contentScripts/register