[META] Implement a userScripts API

ASSIGNED
Assigned to

Status

ASSIGNED
a year ago
3 months ago

People

(Reporter: rpl, Assigned: rpl)

Tracking

(Depends on: 4 bugs, Blocks: 1 bug, {meta})

60 Branch
Dependency tree / graph

Firefox Tracking Flags

(firefox60 affected)

Details

(Assignee)

Description

a year ago
This is a tracking issue for the implementation of a userScripts API.

This new API is going to be similar to the contentScripts.register API (Bug 1332273) but it will execute each registered userScript into an isolated sandbox (instead on a sandbox per extension / target frame) and the userScripts will not have direct access to any of the WebExtensions APIs (the extension will be allowed to provide its own API methods if needed).

Additional details about the API Design and its planning are available on the following wiki page:

- https://wiki.mozilla.org/WebExtensions/UserScripts

As an additional side note, this API is designed to cover the needs described by Bug 1353468 without providing direct access to Cu.Sandbox to the extension code.
(Assignee)

Updated

a year ago
Keywords: meta
(Assignee)

Updated

a year ago
Depends on: 1332273
(Assignee)

Updated

a year ago
Depends on: 1437861
(Assignee)

Updated

a year ago
Depends on: 1437864
(Assignee)

Updated

a year ago
Depends on: 1437867

Comment 1

a year ago
One point that's missing from the design details is the way scripts are matched against URLs. In the WebExtension ecosystem content scripts (and the upcoming user scripts, as they seem to be using the same interface) determine URL matches using the following conditional:

[@match x OR @match y] AND [@include x OR @include y]
AND NOT [@ex_match x OR @ex_match y]
AND NOT [@exclude x  OR @exclude y]

Unfortunately most user script managers do not implement matching in this manner. Instead they use an "OR" between matches and includes:

[[@match x OR @match y] OR [@include x OR @include y]]
AND NOT [@ex_match x OR @ex_match y]
AND NOT [@exclude x  OR @exclude y]

From my testing (using the contentScript API for the time), trying to re-implement this functionality within a WebExtension is not trivial. In an attempt to work around this I tried registering each script twice. Once for all the matches and another with a global match but added the includes. An oversight on this was the interaction when a URL matched both a match and an include. The script would be executed twice, which is undesirable. I tried another idea or two but with even less success.

It would be great if an option or flag to adjust the matching behavior could be added (might be useful in the contentScript API too).
(Assignee)

Comment 2

a year ago
Hi sdaniele3,
thanks a lot for pointing this out, we were not aware of these differences.

Do you mind to add an example of this scenario with some actual urls/patterns from your use cases? 

I'd like to check if there are other ways to workaround these differences. 

Thanks in advance.
Flags: needinfo?(sdaniele3)
(Assignee)

Updated

a year ago
Assignee: nobody → lgreco
Status: NEW → ASSIGNED

Comment 3

a year ago
I would reckon that the majority of cases may have to do with Userscript managers accepting RegExp in `@includes` (which Firefox does not allow). Unfortunately I don't have any specific details regarding actual patterns as I personally only use simple scripts. However, I have reached out on the Greasyfork and OpenUserJS.org GitHub repos asking for additional information, should it be available.

Greasyfork: https://github.com/JasonBarnabe/greasyfork/issues/536
OpenUserJS.org: https://github.com/OpenUserJS/OpenUserJS.org/issues/1333
Flags: needinfo?(sdaniele3)

Comment 4

a year ago
So, from the kind folks over at OpenUserJS.org and Greasyfork I got my hands on a bunch of script meta data. Here are the results.


Total of of scripts: 5591

Have about:blank
#:  0
%:  00.00
Have both match and include
#:  197
%:  03.52
Have Regex (include or exclude)
#:  222
%:  03.97
Have match and Regex (include or exclude)
#:  5
%:  00.08
Have Regex in include:
#:  211
%:  03.77
%%: 95.05
Have Regex in exclude:
#:  32
%:  00.57
%%: 14.41


The first point I see is that matching against `about:blank` is probably not needed and could be removed. Further the usage of both `@match` and `@include` is a small percentage, the same goes for the usage of Regex.

Digging deeper, the vast majority of the `197` items that have both `@match` and `@include` use them as duplicate matchers. That is, each `@match` is copied as an `@include`, I assume this was done for compatibility between different managers that didn't support one or the other. About 10% of these scripts `@include` something that is not present as a `@match`. So that makes a `00.35 %` compatibility issue (disregarding Regex) using `@match AND @include` rather than the current `OR`.

As for 'use cases' there wasn't anything [1] that struck my attention. Perhaps if Regex (or simplified) support was available them there might be some valid use cases. However, of the `5` scripts that had `@match` and Regex, all of them were using Regex as a glorified glob (wildcards only; no alternatives or groups).

To the point, adding an `OR` option may not be required. With such a small percentage of scripts that would break it's probably not worth implementing. Along with that, it's probably not even worth implementing a workaround in script managers.

Examining Regex, which is the other backwards compatibility issue, there are `222` scripts that use it. Some are used as glorified globs but the majority use at least one Regex feature. Of these I saw maybe three that used 'advanced' features such as look-ahead and look-behind. As for the rest of them the most used features are any-of (i.e. `[a-z]`), repetitions, alternatives, and the lazy modifier (are glob wildcards greedy? does it event matter for globs?). I think there might be some credence in providing 'simplified' Regex in Firefox (full Regex would be great though). For the purpose of further argument on this point I'm going to disregard backwards compatibility and assume authors will rewrite their Regex.

Of the Regex features that are used alternatives is not required. If there are alternatives then they can be written as additional rules. The lazy modifier was used ten times and therefore can probably be discarded as well. That leaves any-of and repetitions. Repetitions can be defined as one to x; zero would just be considered an alternative. I don't have a specific count on the number of scripts that used those features but it's at least `60.00 %` of the scripts that utilize Regex.

If Regex (or simplified) support is not added to Firefox then it's not a deal-breaker. Although ugly, in order to maintain backward compatibility user script managers can inject some conditional expressions before the script and return early if a match is not found.

[1] There was one script that had an interesting glob: `@include: *://*oper.ru/*/view.php*`
As far as I can tell this can't be translated into a `@match` due to the host portion. As an example the above glob would match `http://baroper.ru/any/view.php`. This can probably be written off as a one-time outlier.

Comment 5

a year ago
After I submitted my previous post I had another thought. User script managers generally also support the wildcard `.tld`. But I think adding that functionality may be dependent on Bug 1315558.
So I've slept on it a while, and I have to retract my previous statements (elsewhere) in support of this API.

Things looked good at first read above.  Obviously I forgot about regex and .tld support.  And what other details might I have missed?  What other subtle interactions of include and exclude or other behavior will cause painful backwards compatibility issues for my users?

More importantly: this just puts me as an extension developer in a sad and frustrated place.  If I adopt this, I become just a wrapper around some very-high-level API which I have little or no control over.  I expect to find myself stuck in more "sorry, Mozilla decided X and gave me no choise, so you don't get that feature added/bug fixed" style interactions with my users.

Why not give me primitives to accomplish what I want, and also be useful for all other extensions?  Surely user scripts are not the only place where some sort of sandboxed/scoped execution of javascript is useful?
(Assignee)

Comment 7

a year ago
(In reply to Anthony Lieuallen from comment #6)
> So I've slept on it a while, and I have to retract my previous statements
> (elsewhere) in support of this API.

Hi Anthony,
I'm really sorry to hear that, especially given that the patches based on the userScript API design are in a stage that allows to actually "test drive" the API, verify what works and what doesn't, and what would still be missing.

> 
> Things looked good at first read above.  Obviously I forgot about regex and
> .tld support.  And what other details might I have missed?  What other
> subtle interactions of include and exclude or other behavior will cause
> painful backwards compatibility issues for my users?
> 
> More importantly: this just puts me as an extension developer in a sad and
> frustrated place.  If I adopt this, I become just a wrapper around some
> very-high-level API which I have little or no control over.  I expect to
> find myself stuck in more "sorry, Mozilla decided X and gave me no choise,
> so you don't get that feature added/bug fixed" style interactions with my
> users.
> 
> Why not give me primitives to accomplish what I want, and also be useful for
> all other extensions?  Surely user scripts are not the only place where some
> sort of sandboxed/scoped execution of javascript is useful?

Providing direct access to the Sandbox constructor is something that has been discussed and it is not a viable option.

The proposed API design aims to provide as much freedom as possible to the extensions (e.g. by allowing the extensions to define their own custom API methods to the userScripts sandbox), as well as to provide a more easy and consistent way to ensure that these scripts are actually going to be able to run when they are expected to (which is particularly tricky for ones that should run on "document_start" if the user script is injected on the fly using an API like tabs.executeScript, because that API cannot guarantee that the script will actually be able to be executed on its target at during the desired phase of the target document loading).

About "being locked-in by adopting the userScripts API", we tried to design the userScripts API in a way that should not limit what an extension can or cannot do on its own when it adopts this API:

Given that the userScripts.register API is basically providing most of the options already supported by the contentScripts.register API (plus the additional ones which are only available to the userScripts.register API), and that the extension can inject its own custom API methods to make available to the userScript sandboxes, if something is currently possible using one of the other WebExtensions APIs, then it should also be possible for an extension that combines it with the userScripts API.

On the other specific issues already mentioned:

- tld: this seem (at least at a first glance) to be a missing WebExtensions APIs feature more than just something missing specifically from the userScripts API design (but if there is any additional relationship between the tld feature request and this API design, it would be great to better define what is the relationship between them, so that it can be taken into account)

- "differences in the include/exclude properties behaviors": thanks to the details collected by sdaniele3 (thanks a lot! these details are really helpful) we can get a better sense of the issue, its impact and what can be reasonable workarounded by the extension itself (and which part of it can't, or can't be workarounded in a performance sensible way)

About these two issues it would also be interesting to learn what Greasemonkey is currently doing to workaround them (if any, and eventually what are the issues related to the workarounds that are currently used, e.g. do these workarounds present performance issues or do not provide a consistent behavior? e.g. like the tabs.executeScript workaround described above), so that we can keep it into account and verify if something in the userScripts API design could be preventing it or making it unreasonably harder.
(In reply to Anthony Lieuallen from comment #6)

> Why not give me primitives to accomplish what I want, and also be useful for
> all other extensions?  Surely user scripts are not the only place where some
> sort of sandboxed/scoped execution of javascript is useful?

This is a very vague request.  As has been said, we're never going to expose primitives such as direct access to sandbox apis.  

I read two things into this:

- you want to do your own matching (e.g. as you possibly could with tabs.executeScript)
- you want a sandbox that does not require attaching to a content page

I think the first is a reasonable consideration and could be a followup.  The second I'm lacking any concrete current use case that couldn't be handled other ways, but am open to what that would be.

Overall I think we've tried to be very responsive to your needs without compromising the intention/goals of webextensions.  Yes, you will never get access to low level primitives again, but there have been a lot of requests for changes and additions to webrequest apis that we do consider and approve.

Yes, ultimately you will be "stuck" with whatever APIs get approved for addition.  Having userScripts or not doesn't change that.

Comment 9

a year ago
> About these two issues it would also be interesting to learn what Greasemonkey is currently doing to workaround them

In the current codebase Greasemonkey does all matching itself [1] and if a script matches a URL the content is executed using `tabs.executeScript`. The biggest problem is the timing issue (as previously discussed). Another, potential, issue may be performance. Unless I'm mistaken Firefox does URL matching for content scripts in C code rather than Javascript. I don't quite understand the relationship between Firefox C code and Firefox Javascript code so I may be off base, but I'd think that the C code would perform better?

As for workarounds using the `(content|user)Script` API interfaces (testing on Beta), I've made some proof of concepts. The one that I've gotten to work the best is probably a performance nightmare. Rather than restate what I did I'm just going to link the GitHub comment. As another point, it's an attempt to work around the "OR" issue, regex issue, and tld issue. https://github.com/greasemonkey/greasemonkey/issues/2663#issuecomment-366484164

I would also like to point out that even if the userScript registration API supported the above features for matching it would also be great to manually query currently running scripts. The reason is that many managers like to list "current number of running scripts on the tab" or "which scripts are currently running on the tab". Without such an interface the userscript manager would have to self-implement pattern matching and that would result in inconsistencies and, probably, performance issues.

I'd envision the API to be something like `browser.userScripts.runningOn(tabId)` which would return a list of "scriptMetadata" (that was passed into `.register`) objects.

> tld: this seem (at least at a first glance) to be a missing WebExtensions APIs feature more than just something missing specifically from the userScripts API design (but if there is any additional relationship ...

I don't think there's any "additional" relationship. But it would need to be applicable in the (content|user)Script match/include patterns.

[1] Greasemonkey has to include it's own MatchPattern implementation which is not ideal. This may be somewhat outside the scope of /this/ particular issue, but it would be great if some interface could be created to access Firefox's implementation. This doesn't have to be direct access but an API like `browser.<something>.matchpattern.matches(url, [pattern, pattern])` would be sufficient. This would eliminate any inconsistencies. Bug 1395278 asked for something similar, but was shot down. The reasoning doesn't make much sense to me (too broad?). Although, this may not be required depending on how the userScript API turns out.
See Also: → bug 1315558
(Assignee)

Updated

a year ago
Depends on: 1445909
Just thought of another subtle issue that is missing from the designs above.

In GM 3 users could control which order their installed user scripts would run.  In some circumstances, script A will change the page in a way that script B can't understand, but not vice versa.  So the user chooses to have script B run first.

Assume GM adopts the user script API designed above.  How do we provide this feature to our users?

Comment 11

a year ago
> Users can turn Greasemonkey on and off at runtime.  If I turn it off then scripts don't run until I turn it back on.  I do this by interacting with the extension through the browser action's popup.

I believe scripts are run in the order they're registered. Therefore the simplest would be to unregister and re-register. Downside is that every time you save a script (for development or whatever) you have to unregister everything to ensure the order. Not particularly difficult though.

Comment 12

a year ago
Gah, bad copy / paste.

Replace the quote with

> Assume GM adopts the user script API designed above.  How do we provide this feature to our users?

Updated

a year ago
Depends on: 1447361
(Assignee)

Comment 13

a year ago
(In reply to Anthony Lieuallen from comment #10)
> Just thought of another subtle issue that is missing from the designs above.
> 
> In GM 3 users could control which order their installed user scripts would
> run.  In some circumstances, script A will change the page in a way that
> script B can't understand, but not vice versa.  So the user chooses to have
> script B run first.
> 
> Assume GM adopts the user script API designed above.  How do we provide this
> feature to our users?

Hi Anthony,
thanks for pointing out this issue and describe the reasons behind it, it is definitely helpful.

I confirm sdaniele3's comment 11, the scripts are executed in the order they're registered, and so currently the only way to change the execution order of two registered scripts (content scripts or user scripts) is to unregister them and then register them again in the correct order.

Making the API able to change the order of the registered scripts is something that is going to require additional changes to some of the "low level" internals, I'm going to take a look at the needed changes and I'll file a separate bugzilla issue to evaluate it with the rest of the team.

Comment 14

a year ago
> Making the API able to change the order of the registered scripts is something that is going
> to require additional changes to some of the "low level" internals, I'm going to take a look
> at the needed changes and I'll file a separate bugzilla issue to evaluate it with the rest
> of the team.

Changing the internals to support ordering may be less important if the following (quoted from
bug 1445909) is implemented.

(Luca Greco)
> To be able to temporarily disable a registered script we can evaluate to define a script API
> method (e.g. `script.setEnabled(true/false)`) which could prevent the script from being
> executed without unregister it completely.

The reasons a script may need to be unregistered / reregistered are, generally, the following.
A script is installed, a script is updated / saved (in an editor), a script is disabled / enabled
or (assuming ordering is maintained) when the script execution order changes. Along those lines
the most common use case would be, from my experience, enabling and disabling a script. If the
overhead of unregistering and reregistering all scripts when a single script is disabled can
be eliminated then the benefit may not be worth the time / complexity of internally supporting
script order. If the extension only has to unregister / reregister all scripts with infrequent
operations (install / save / update / order change) then I think it's a viable option.
(Assignee)

Updated

a year ago
Duplicate of this bug: 1353468
(Assignee)

Updated

11 months ago
See Also: → bug 1458947

Updated

11 months ago
Blocks: 1459029
Summary: Implement a userScripts API → [META] Implement a userScripts API

Updated

10 months ago
Depends on: 1378459

Updated

9 months ago
Product: Toolkit → WebExtensions
Depends on: 1470310
(Assignee)

Updated

9 months ago
Depends on: 1470466
No longer blocks: 1459029
Blocks: 1459029

Comment 16

8 months ago
Is it be possible to allow the "userScripts" permission to be used as an optional permission?
(I'm the developer of Gesturey a mouse gesture extension and I want to support custom user scripts on certain gestures. Since this is a very advanced feature which only a minority of users will use, I don't want to declare this as a core permission.)

Updated

7 months ago
Depends on: 1481653
(Assignee)

Updated

6 months ago
Depends on: 1491024

Updated

6 months ago
Depends on: 1491036

Updated

6 months ago
Depends on: 1491051

Updated

6 months ago
Depends on: 1491272

Updated

6 months ago
Depends on: 1491274

Updated

5 months ago
Depends on: 1498739
(Assignee)

Updated

5 months ago
Depends on: 1498343
(Assignee)

Updated

5 months ago
Duplicate of this bug: 1423966

Comment 18

4 months ago
In case it is not already part of the API ....

Would it be possible to add/attach a CSS registering option to the same API?

Similar to the "content_scripts" in manifest.json

  "content_scripts": [
    {
      "matches": ["://*.mozilla.org/*"],
      "js": ["script.js"],
      "css": ["style.css"]
    },


While user CSS is not a script, a method to permanently register/unregister CSS in the exact way a userscript is registered, would be great.

It is possible to inject CSS as script via userScript API, but it would be better for it to have its own sub-section.

I assume it should not be complicated to attach a method to register both script & css within the same API!!?
(Assignee)

Comment 19

4 months ago
(In reply to erosman from comment #18)
> In case it is not already part of the API ....
> 
> Would it be possible to add/attach a CSS registering option to the same API?
> ...
> While user CSS is not a script, a method to permanently register/unregister
> CSS in the exact way a userscript is registered, would be great.

This is already covered by contentScripts.register (https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/contentScripts/register).

Comment 20

4 months ago
AFA I can see, ScriptOptions parameters for the userScripts.register() are identical (CSS omitted) to contentScripts.register(). That is great and helps and simplifies extension coding immensely.

Please continue in keeping them identical.
(Assignee)

Updated

4 months ago
Depends on: 1509339

Comment 21

4 months ago
Sorry if this is the wrong place to ask for, but is it possible to run a user script in a specific tab like the currently active one? (perhaps similar to tabs.executeScript())

An example use case would be an user script that can be applied to the current tab/website on browser action click or another trigger. In my specific case I want users to be able to run scripts on certain mouse gestures.

Updated

3 months ago
Depends on: 1514809

Updated

3 months ago
Depends on: 1516356

Comment 22

3 months ago
FYI I want to re-iterate that I want a way to turn x-rays off.
You need to log in before you can comment on or make changes to this bug.