1437098 - [META] Implement a userScripts API

Status: RESOLVED FIXED

(WebExtensions :: General, enhancement)

Description

[Luca Greco [:rpl] [:luca] [:lgreco]]

This is a tracking issue for the implementation of a userScripts API.

This new API is going to be similar to the contentScripts.register API (Bug 1332273) but it will execute each registered userScript into an isolated sandbox (instead on a sandbox per extension / target frame) and the userScripts will not have direct access to any of the WebExtensions APIs (the extension will be allowed to provide its own API methods if needed).

Additional details about the API Design and its planning are available on the following wiki page:

- https://wiki.mozilla.org/WebExtensions/UserScripts

As an additional side note, this API is designed to cover the needs described by Bug 1353468 without providing direct access to Cu.Sandbox to the extension code.

Comment 1

[sdaniele3]

One point that's missing from the design details is the way scripts are matched against URLs. In the WebExtension ecosystem content scripts (and the upcoming user scripts, as they seem to be using the same interface) determine URL matches using the following conditional:

[@match x OR @match y] AND [@include x OR @include y]
AND NOT [@ex_match x OR @ex_match y]
AND NOT [@exclude x  OR @exclude y]

Unfortunately most user script managers do not implement matching in this manner. Instead they use an "OR" between matches and includes:

[[@match x OR @match y] OR [@include x OR @include y]]
AND NOT [@ex_match x OR @ex_match y]
AND NOT [@exclude x  OR @exclude y]

From my testing (using the contentScript API for the time), trying to re-implement this functionality within a WebExtension is not trivial. In an attempt to work around this I tried registering each script twice. Once for all the matches and another with a global match but added the includes. An oversight on this was the interaction when a URL matched both a match and an include. The script would be executed twice, which is undesirable. I tried another idea or two but with even less success.

It would be great if an option or flag to adjust the matching behavior could be added (might be useful in the contentScript API too).

Comment 2

[Luca Greco [:rpl] [:luca] [:lgreco]]

Hi sdaniele3,
thanks a lot for pointing this out, we were not aware of these differences.

Do you mind to add an example of this scenario with some actual urls/patterns from your use cases? 

I'd like to check if there are other ways to workaround these differences. 

Thanks in advance.

Comment 3

[sdaniele3]

I would reckon that the majority of cases may have to do with Userscript managers accepting RegExp in `@includes` (which Firefox does not allow). Unfortunately I don't have any specific details regarding actual patterns as I personally only use simple scripts. However, I have reached out on the Greasyfork and OpenUserJS.org GitHub repos asking for additional information, should it be available.

Greasyfork: https://github.com/JasonBarnabe/greasyfork/issues/536
OpenUserJS.org: https://github.com/OpenUserJS/OpenUserJS.org/issues/1333

Comment 4

[sdaniele3]

So, from the kind folks over at OpenUserJS.org and Greasyfork I got my hands on a bunch of script meta data. Here are the results.


Total of of scripts: 5591

Have about:blank
#:  0
%:  00.00
Have both match and include
#:  197
%:  03.52
Have Regex (include or exclude)
#:  222
%:  03.97
Have match and Regex (include or exclude)
#:  5
%:  00.08
Have Regex in include:
#:  211
%:  03.77
%%: 95.05
Have Regex in exclude:
#:  32
%:  00.57
%%: 14.41


The first point I see is that matching against `about:blank` is probably not needed and could be removed. Further the usage of both `@match` and `@include` is a small percentage, the same goes for the usage of Regex.

Digging deeper, the vast majority of the `197` items that have both `@match` and `@include` use them as duplicate matchers. That is, each `@match` is copied as an `@include`, I assume this was done for compatibility between different managers that didn't support one or the other. About 10% of these scripts `@include` something that is not present as a `@match`. So that makes a `00.35 %` compatibility issue (disregarding Regex) using `@match AND @include` rather than the current `OR`.

As for 'use cases' there wasn't anything [1] that struck my attention. Perhaps if Regex (or simplified) support was available them there might be some valid use cases. However, of the `5` scripts that had `@match` and Regex, all of them were using Regex as a glorified glob (wildcards only; no alternatives or groups).

To the point, adding an `OR` option may not be required. With such a small percentage of scripts that would break it's probably not worth implementing. Along with that, it's probably not even worth implementing a workaround in script managers.

Examining Regex, which is the other backwards compatibility issue, there are `222` scripts that use it. Some are used as glorified globs but the majority use at least one Regex feature. Of these I saw maybe three that used 'advanced' features such as look-ahead and look-behind. As for the rest of them the most used features are any-of (i.e. `[a-z]`), repetitions, alternatives, and the lazy modifier (are glob wildcards greedy? does it event matter for globs?). I think there might be some credence in providing 'simplified' Regex in Firefox (full Regex would be great though). For the purpose of further argument on this point I'm going to disregard backwards compatibility and assume authors will rewrite their Regex.

Of the Regex features that are used alternatives is not required. If there are alternatives then they can be written as additional rules. The lazy modifier was used ten times and therefore can probably be discarded as well. That leaves any-of and repetitions. Repetitions can be defined as one to x; zero would just be considered an alternative. I don't have a specific count on the number of scripts that used those features but it's at least `60.00 %` of the scripts that utilize Regex.

If Regex (or simplified) support is not added to Firefox then it's not a deal-breaker. Although ugly, in order to maintain backward compatibility user script managers can inject some conditional expressions before the script and return early if a match is not found.

[1] There was one script that had an interesting glob: `@include: *://*oper.ru/*/view.php*`
As far as I can tell this can't be translated into a `@match` due to the host portion. As an example the above glob would match `http://baroper.ru/any/view.php`. This can probably be written off as a one-time outlier.

Comment 5

[sdaniele3]

After I submitted my previous post I had another thought. User script managers generally also support the wildcard `.tld`. But I think adding that functionality may be dependent on Bug 1315558.

Comment 6

[Anthony Lieuallen]

So I've slept on it a while, and I have to retract my previous statements (elsewhere) in support of this API.

Things looked good at first read above.  Obviously I forgot about regex and .tld support.  And what other details might I have missed?  What other subtle interactions of include and exclude or other behavior will cause painful backwards compatibility issues for my users?

More importantly: this just puts me as an extension developer in a sad and frustrated place.  If I adopt this, I become just a wrapper around some very-high-level API which I have little or no control over.  I expect to find myself stuck in more "sorry, Mozilla decided X and gave me no choise, so you don't get that feature added/bug fixed" style interactions with my users.

Why not give me primitives to accomplish what I want, and also be useful for all other extensions?  Surely user scripts are not the only place where some sort of sandboxed/scoped execution of javascript is useful?

Comment 7

[Luca Greco [:rpl] [:luca] [:lgreco]]

(In reply to Anthony Lieuallen from comment #6)
> So I've slept on it a while, and I have to retract my previous statements
> (elsewhere) in support of this API.

Hi Anthony,
I'm really sorry to hear that, especially given that the patches based on the userScript API design are in a stage that allows to actually "test drive" the API, verify what works and what doesn't, and what would still be missing.

> 
> Things looked good at first read above.  Obviously I forgot about regex and
> .tld support.  And what other details might I have missed?  What other
> subtle interactions of include and exclude or other behavior will cause
> painful backwards compatibility issues for my users?
> 
> More importantly: this just puts me as an extension developer in a sad and
> frustrated place.  If I adopt this, I become just a wrapper around some
> very-high-level API which I have little or no control over.  I expect to
> find myself stuck in more "sorry, Mozilla decided X and gave me no choise,
> so you don't get that feature added/bug fixed" style interactions with my
> users.
> 
> Why not give me primitives to accomplish what I want, and also be useful for
> all other extensions?  Surely user scripts are not the only place where some
> sort of sandboxed/scoped execution of javascript is useful?

Providing direct access to the Sandbox constructor is something that has been discussed and it is not a viable option.

The proposed API design aims to provide as much freedom as possible to the extensions (e.g. by allowing the extensions to define their own custom API methods to the userScripts sandbox), as well as to provide a more easy and consistent way to ensure that these scripts are actually going to be able to run when they are expected to (which is particularly tricky for ones that should run on "document_start" if the user script is injected on the fly using an API like tabs.executeScript, because that API cannot guarantee that the script will actually be able to be executed on its target at during the desired phase of the target document loading).

About "being locked-in by adopting the userScripts API", we tried to design the userScripts API in a way that should not limit what an extension can or cannot do on its own when it adopts this API:

Given that the userScripts.register API is basically providing most of the options already supported by the contentScripts.register API (plus the additional ones which are only available to the userScripts.register API), and that the extension can inject its own custom API methods to make available to the userScript sandboxes, if something is currently possible using one of the other WebExtensions APIs, then it should also be possible for an extension that combines it with the userScripts API.

On the other specific issues already mentioned:

- tld: this seem (at least at a first glance) to be a missing WebExtensions APIs feature more than just something missing specifically from the userScripts API design (but if there is any additional relationship between the tld feature request and this API design, it would be great to better define what is the relationship between them, so that it can be taken into account)

- "differences in the include/exclude properties behaviors": thanks to the details collected by sdaniele3 (thanks a lot! these details are really helpful) we can get a better sense of the issue, its impact and what can be reasonable workarounded by the extension itself (and which part of it can't, or can't be workarounded in a performance sensible way)

About these two issues it would also be interesting to learn what Greasemonkey is currently doing to workaround them (if any, and eventually what are the issues related to the workarounds that are currently used, e.g. do these workarounds present performance issues or do not provide a consistent behavior? e.g. like the tabs.executeScript workaround described above), so that we can keep it into account and verify if something in the userScripts API design could be preventing it or making it unreasonably harder.

Comment 8

[Shane Caraveo (:mixedpuppy)]

(In reply to Anthony Lieuallen from comment #6)

> Why not give me primitives to accomplish what I want, and also be useful for
> all other extensions?  Surely user scripts are not the only place where some
> sort of sandboxed/scoped execution of javascript is useful?

This is a very vague request.  As has been said, we're never going to expose primitives such as direct access to sandbox apis.  

I read two things into this:

- you want to do your own matching (e.g. as you possibly could with tabs.executeScript)
- you want a sandbox that does not require attaching to a content page

I think the first is a reasonable consideration and could be a followup.  The second I'm lacking any concrete current use case that couldn't be handled other ways, but am open to what that would be.

Overall I think we've tried to be very responsive to your needs without compromising the intention/goals of webextensions.  Yes, you will never get access to low level primitives again, but there have been a lot of requests for changes and additions to webrequest apis that we do consider and approve.

Yes, ultimately you will be "stuck" with whatever APIs get approved for addition.  Having userScripts or not doesn't change that.

Comment 9

[sdaniele3]

> About these two issues it would also be interesting to learn what Greasemonkey is currently doing to workaround them

In the current codebase Greasemonkey does all matching itself [1] and if a script matches a URL the content is executed using `tabs.executeScript`. The biggest problem is the timing issue (as previously discussed). Another, potential, issue may be performance. Unless I'm mistaken Firefox does URL matching for content scripts in C code rather than Javascript. I don't quite understand the relationship between Firefox C code and Firefox Javascript code so I may be off base, but I'd think that the C code would perform better?

As for workarounds using the `(content|user)Script` API interfaces (testing on Beta), I've made some proof of concepts. The one that I've gotten to work the best is probably a performance nightmare. Rather than restate what I did I'm just going to link the GitHub comment. As another point, it's an attempt to work around the "OR" issue, regex issue, and tld issue. https://github.com/greasemonkey/greasemonkey/issues/2663#issuecomment-366484164

I would also like to point out that even if the userScript registration API supported the above features for matching it would also be great to manually query currently running scripts. The reason is that many managers like to list "current number of running scripts on the tab" or "which scripts are currently running on the tab". Without such an interface the userscript manager would have to self-implement pattern matching and that would result in inconsistencies and, probably, performance issues.

I'd envision the API to be something like `browser.userScripts.runningOn(tabId)` which would return a list of "scriptMetadata" (that was passed into `.register`) objects.

> tld: this seem (at least at a first glance) to be a missing WebExtensions APIs feature more than just something missing specifically from the userScripts API design (but if there is any additional relationship ...

I don't think there's any "additional" relationship. But it would need to be applicable in the (content|user)Script match/include patterns.

[1] Greasemonkey has to include it's own MatchPattern implementation which is not ideal. This may be somewhat outside the scope of /this/ particular issue, but it would be great if some interface could be created to access Firefox's implementation. This doesn't have to be direct access but an API like `browser.<something>.matchpattern.matches(url, [pattern, pattern])` would be sufficient. This would eliminate any inconsistencies. Bug 1395278 asked for something similar, but was shot down. The reasoning doesn't make much sense to me (too broad?). Although, this may not be required depending on how the userScript API turns out.

Comment 10

[Anthony Lieuallen]

Just thought of another subtle issue that is missing from the designs above.

In GM 3 users could control which order their installed user scripts would run.  In some circumstances, script A will change the page in a way that script B can't understand, but not vice versa.  So the user chooses to have script B run first.

Assume GM adopts the user script API designed above.  How do we provide this feature to our users?

Comment 11

[sdaniele3]

> Users can turn Greasemonkey on and off at runtime.  If I turn it off then scripts don't run until I turn it back on.  I do this by interacting with the extension through the browser action's popup.

I believe scripts are run in the order they're registered. Therefore the simplest would be to unregister and re-register. Downside is that every time you save a script (for development or whatever) you have to unregister everything to ensure the order. Not particularly difficult though.

Comment 12

[sdaniele3]

Gah, bad copy / paste.

Replace the quote with

> Assume GM adopts the user script API designed above.  How do we provide this feature to our users?

Comment 13

[Luca Greco [:rpl] [:luca] [:lgreco]]

(In reply to Anthony Lieuallen from comment #10)
> Just thought of another subtle issue that is missing from the designs above.
> 
> In GM 3 users could control which order their installed user scripts would
> run.  In some circumstances, script A will change the page in a way that
> script B can't understand, but not vice versa.  So the user chooses to have
> script B run first.
> 
> Assume GM adopts the user script API designed above.  How do we provide this
> feature to our users?

Hi Anthony,
thanks for pointing out this issue and describe the reasons behind it, it is definitely helpful.

I confirm sdaniele3's comment 11, the scripts are executed in the order they're registered, and so currently the only way to change the execution order of two registered scripts (content scripts or user scripts) is to unregister them and then register them again in the correct order.

Making the API able to change the order of the registered scripts is something that is going to require additional changes to some of the "low level" internals, I'm going to take a look at the needed changes and I'll file a separate bugzilla issue to evaluate it with the rest of the team.

Comment 14

[sdaniele3]

> Making the API able to change the order of the registered scripts is something that is going
> to require additional changes to some of the "low level" internals, I'm going to take a look
> at the needed changes and I'll file a separate bugzilla issue to evaluate it with the rest
> of the team.

Changing the internals to support ordering may be less important if the following (quoted from
bug 1445909) is implemented.

(Luca Greco)
> To be able to temporarily disable a registered script we can evaluate to define a script API
> method (e.g. `script.setEnabled(true/false)`) which could prevent the script from being
> executed without unregister it completely.

The reasons a script may need to be unregistered / reregistered are, generally, the following.
A script is installed, a script is updated / saved (in an editor), a script is disabled / enabled
or (assuming ordering is maintained) when the script execution order changes. Along those lines
the most common use case would be, from my experience, enabling and disabling a script. If the
overhead of unregistering and reregistering all scripts when a single script is disabled can
be eliminated then the benefit may not be worth the time / complexity of internally supporting
script order. If the extension only has to unregister / reregister all scripts with infrequent
operations (install / save / update / order change) then I think it's a viable option.

Comment 16

[robbendebiene]

Is it be possible to allow the "userScripts" permission to be used as an optional permission?
(I'm the developer of Gesturey a mouse gesture extension and I want to support custom user scripts on certain gestures. Since this is a very advanced feature which only a minority of users will use, I don't want to declare this as a core permission.)

Comment 18

[erosman]

In case it is not already part of the API ....

Would it be possible to add/attach a CSS registering option to the same API?

Similar to the "content_scripts" in manifest.json

  "content_scripts": [
    {
      "matches": ["://*.mozilla.org/*"],
      "js": ["script.js"],
      "css": ["style.css"]
    },


While user CSS is not a script, a method to permanently register/unregister CSS in the exact way a userscript is registered, would be great.

It is possible to inject CSS as script via userScript API, but it would be better for it to have its own sub-section.

I assume it should not be complicated to attach a method to register both script & css within the same API!!?

Comment 19

[Luca Greco [:rpl] [:luca] [:lgreco]]

(In reply to erosman from comment #18)
> In case it is not already part of the API ....
> 
> Would it be possible to add/attach a CSS registering option to the same API?
> ...
> While user CSS is not a script, a method to permanently register/unregister
> CSS in the exact way a userscript is registered, would be great.

This is already covered by contentScripts.register (https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/contentScripts/register).

Comment 20

[erosman]

AFA I can see, ScriptOptions parameters for the userScripts.register() are identical (CSS omitted) to contentScripts.register(). That is great and helps and simplifies extension coding immensely.

Please continue in keeping them identical.

Comment 21

[robbendebiene]

Sorry if this is the wrong place to ask for, but is it possible to run a user script in a specific tab like the currently active one? (perhaps similar to tabs.executeScript())

An example use case would be an user script that can be applied to the current tab/website on browser action click or another trigger. In my specific case I want users to be able to run scripts on certain mouse gestures.

Comment 22

[Anthony Lieuallen]

FYI I want to re-iterate that I want a way to turn x-rays off.

Comment 23

[sam.sandberg]

Hello, I'm new here.

Am I understanding correctly that this API a potential alternative to Greasemonkey/Tampermonkey/etc.?

Or is it just a safer alternative approach for allowing those extensions to function?

Thanks in advance! 🙏

Comment 24

[erosman]

(In reply to sam.sandberg from comment #23)
It is a safer API for script-mangers (and similar) to use to inject JavaScript into web content/pages.

Comment 25

[Sergio]

As it seems some bugs are being closed cause "there has been very little interest" on them i want to confirm that i'm still interested on this bug.

Comment 26

[Philipp Kewisch [:Fallen][📱📆]]

Not quite sure where to attach this, but the draft at https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/userScripts is not very elaborate and seems to have remnants of the contentScripts page. As the initial version is enabled in 68 I feel we need to complete this documentation.

Comment 27

[Irene Smith]

(In reply to Philipp Kewisch [:Fallen] [:📆] from comment #26)

Not quite sure where to attach this, but the draft at https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/userScripts is not very elaborate and seems to have remnants of the contentScripts page. As the initial version is enabled in 68 I feel we need to complete this documentation.

This work is being tracked here: MDN/Sprints 2136

Comment 28

[arthaey]

I'm just a Firefox + Tampermonkey user hoping to stop getting CSP errors for my userscripts. I'm excited to see this is so close!

What work is left for this ticket? MDN/Sprints 2136 is closed as done.

The related Trello card has 3 items left:

• this ticket (kinda circular 😊)
• enable API on release channels: Bug 1514809 (done as of FF68)
• Android WebExt support: Bug 1263005 (still open, unassigned)

Does Android support block this ticket from being completed, then?

Comment 29

[Luca Greco [:rpl] [:luca] [:lgreco]]

(In reply to arthaey from comment #28)

I'm just a Firefox + Tampermonkey user hoping to stop getting CSP errors for my userscripts. I'm excited to see this is so close!

What work is left for this ticket? MDN/Sprints 2136 is closed as done.

As you have mentioned in Bug 1437861 comment 57, this API is already enabled by default on all channels since 68 and so it can already be used.

The userScript API docs on MDN and a new example for the mdn/webextensions-examples github repo are under ongoing work,
and under this meta there are some enhancements linked as a dependency (like Bug 1437867 and Bug 1445909).

The related Trello card has 3 items left:

• this ticket (kinda circular 😊)

eh :-)
this is a meta issue, it is only used to group together bugzilla issues related to the userScripts API.

• enable API on release channels: Bug 1514809 (done as of FF68)
• Android WebExt support: Bug 1263005 (still open, unassigned)

It looks that the work needed for the MDN docs was being tracked from the same MDN trello card, but from an API implementation point of view Bug 1263005 isn't related at all to the userScripts API, and so it has never been a blocker for it.

Comment 30

[Luca Greco [:rpl] [:luca] [:lgreco]]

Closing as resolved-fixed, because as mentioned in comment 29 the API has been already enabled by default
(followups and enhancements not yet closed moved to Bug 1595853).