Enable Greasemonkey to be a WebExtension

REOPENED
Unassigned
(NeedInfo from)

Status

()

Toolkit
WebExtensions: Compatibility
REOPENED
4 months ago
11 days ago

People

(Reporter: andym, Unassigned, NeedInfo)

Tracking

(Depends on: 2 bugs)

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: triaged[awe:{e4a8a97b-f2ed-450b-b12d-ee082ba24781}])

(Reporter)

Description

4 months ago
Greasemonkey does quite a lot of things and this is to track those things.
(Reporter)

Updated

4 months ago
Depends on: 1215065
Status: NEW → RESOLVED
Last Resolved: 4 months ago
Resolution: --- → DUPLICATE
Duplicate of bug: 1232177
(Reporter)

Comment 2

4 months ago
Please don't duplicate, this is a tracker to help the Greasemonkey author track their move to a WebExtension.
Status: RESOLVED → REOPENED
Resolution: DUPLICATE → ---
OK, but I don't see why we need two bugs for that.

Comment 4

4 months ago
I'm going to dump a quick summary of things here for now, let me know if you'd like individual bugs for separate topics.  That said, Greasemonkey has a 10+ year history of accumulated features, and users accustomed to those features.  My goal would be to keep as many of them as possible.

One related group:

1) The ability to download and store "files" including arbitrary content (some will be text, some will be e.g. images).
2) The ability to load those "files" contents (i.e. load the user script's source, for evaluation).
3) A way to efficiently serve these "files" by URL, this means things like using a <style> or <img> tag that content can then load like any other.  (Today, we implement a nsIProtocolHandler for this.)
4) The ability to launch an editor, to change the content of (at least one of) these files.  This is IMO one of Greasemonkey's points over Tampermonkey today: Tampermonkey's inbuilt editor is unpleasant in a variety of ways, but with Greasemonkey users are free to select whatever text editor they prefer.  (Also: some sort of embeddable Scratchpad with it's JS editing capabilities would be nice.)

Part 1 and 2 are probably easy with something like the File API, but parts 3 and 4 are not so obviously accomplishable.
(Reporter)

Comment 5

4 months ago
For 1) and 2) it doesn't sound like you need to have the files actually touch the file system, but rather have an efficient place to store and load blobs from. Something like storage.local won't work for that, but we've been investigating using indexedDB. 

There's more details in bug 1323414 and then bug 1331618 for storing up 20gb at a time.

And an example for that here: https://github.com/mdn/webextensions-examples/pull/171

Comment 6

4 months ago
Turns out point 4 from comment #4 is ~identical to `extension.getURL()` except, for a user script installed into Greasemonkey, rather than for a part of its own XPI.


I'm also starting to look into the details of the sandbox environment.  Greasemonkey has two paths: one simply injects a script, with no special nature, directly into content -- by requesting a Cu.Sandbox with the content principal and wantXrays=false.  The other uses an _expanded_ principal, wrapping the content, and wantXrays=true.  This mode also injects some number of privileged APIs.

It looks like content scripts give us exportFunction and cloneInto and wrappedJSObject, which gets us most of the way to those features, but we won't be able to control which mode (xrays or not? expanded principal or not?) we're in?
(Reporter)

Comment 7

4 months ago
Kris should be able to help on comment #6
Flags: needinfo?(kmaglione+bmo)

Comment 8

4 months ago
Whoops, above I said "point four from comment #4" and meant "point 3".  Point four is still something I'm very interested in, but worried that WebExtensions by design completely prevents any kind of file system interaction, so I could never use the editor I'm already familiar and comfortable with for authoring (installed and active) user scripts.


Two other interesting areas have to do with detecting/intercepting normal browser events.

For injection: We use an observer on document-element-inserted to run user scripts before any content.  Two examples are for redirections (e.g. http->https) or for influencing the behavior of inline scripts (be it fixing compatibility or otherwise) before they execute.  We also want the document to exist (e.g. inject styles now, to prevent flicker), but nothing else.  IOW we want exactly document-element-inserted.

Similarly the "Intercept HTTP requests" talks about observing and canceling, but is there any way to resume a canceled request?  We do this today if:
1) Observe and cancel the request because it ends in .user.js
2) Re-start it internally, discover that the response is actually HTML (not a user script), cancel this
3) Resume the original request (and the browser's navigation to that page)

We mostly migrated away from our nsIContentPolicy implementation for e10s, but found that it was the only way to detect loading of file:// -- no amount of intercepting HTTP requests works when loading a local file (e.g. I saved this old version of this user script, because I don't like the change made in a more recent update).


Storage:  https://developer.mozilla.org/en-US/Add-ons/WebExtensions/API/Storage/StorageArea
Implements a get and set, but no list of previously set keys?

This is also suboptimal for storage.sync; Ideally we'd have one key per sync-able object, and be able to observe appearance during sync.  As is we'll be forced to on every (even minor) change load and serialize all the possible data into one value.. and I still don't know how to observe (besides polling?) when that value has changed.  Oh!
https://developer.mozilla.org/en-US/Add-ons/WebExtensions/API/storage/sync
But Firefox doesn't even support storage.sync, only Chrome does?


Clipboard: https://wiki.greasespot.net/GM_setClipboard
We expose GM_setClipboard(String), but WebExtensions seems to have only "trigger the copy command", over the current selection.


For execution:
It seems either I have to write some executes-everywhere content script which coordinates many calls to eval(), or there's also tabs.executeScript().  But I can't tell how I'd inject our special API methods to either.  If I figure that out though, we'll also (AFAICT) be executing the user script in the full privileged "content script" scope... so how do I isolate user scripts from each other?


Phew.

Comment 9

4 months ago
FYI https://groups.google.com/d/topic/greasemonkey-dev/K6IyDUWnTQc/discussion

I've started a design doc and a discussion in GM's developer group for this topic.



Also, via other channels, it's been explained to me that storage.sync is coming in Firefox 52 (stable is 51.0.1 now), so the docs are "out of date" when they say Firefox does not support it.  And I noticed that

https://developer.mozilla.org/en-US/Add-ons/WebExtensions/API/storage/onChanged

Does exist.  Still no `list()` call though?

Updated

2 months ago
Depends on: 1353468

Comment 10

2 months ago
Adding dependency on bug 1332273, which is a lite version of bug 1323433.
Depends on: 1332273

Comment 11

2 months ago
I don't know how bug 1332273 relates, at least in its current state.  My plans and ideas have shifted a bit, recently.  See especially https://github.com/greasemonkey/greasemonkey/issues/2484 .

Pretty sure we'll use IndexedDB for storing values, which offers synchronous access.  The hard bits that are left:

* Dynamic content script execution, at document_start time.
* Efficiently serving dynamic content efficiently (e.g. images downloaded with the script, inserted into content -- IndexedDB and blobs might help except I think they won't cross the extension/content boundary).
* Generally knowing how/having confidence that my user scripts as content scripts (I assume this will be the solution) work, but don't get privileges I don't intend them to.  How do I tell one content script from another (in a message listener in a background script)?

Comment 12

2 months ago
(In reply to Anthony Lieuallen from comment #11)
> I don't know how bug 1332273 relates, at least in its current state.

Bug 1332273 is about offering a way to WebExtensions to declare options that are synchronously available to content scripts of an add-on.

If that is implemented, then you can dynamically execute user scripts at document_start time as follows:
1. Declare a static content script in manifest.json at document_start.
2. In that content script, synchronously retrieve the set of relevant user scripts for the current document (e.g. using APIs introduced by bug 1332273).
3. Run the script:
   - If using "@grant none" using window.eval (this runs code in the page's context - bug 1288284 confirms that this is the desired behavior).
   - If using privileged APIs, create a sandbox (not yet possible, see bug 1353468).

Furthermore, you can use it to implement synchronous getters such as GM_getValue. The API proposed in bug 1332273 offers read-only access to content scripts, so GM_setValue has to go through the background page and you need to synchronize GM_getValue and GM_setValue (so that after calling GM_setValue('x', 1), GM_getValue('x') returns 1 immediately).


> My plans and ideas have shifted a bit, recently.  See especially
> https://github.com/greasemonkey/greasemonkey/issues/2484 .
> 
> Pretty sure we'll use IndexedDB for storing values, which offers synchronous
> access.

- IndexedDB in a content script uses the database of the page's origin, not of your extension's.
- IndexedDB is an async API; sync IndexedDB was eventually removed from the spec due to the lack of implementations.
- Currently there are no supported APIs for synchronous communication between the background page and the content script.
In your Github ticket, the8472 says that sync XHR+webRequest can be used. Note that sync XHR is deprecated, and webRequest+sync XHR is quite a hack that should be avoided if possible for performance reasons.

Your ideas in the Github issue are sane though: Where possible, use async APIs.


>  The hard bits that are left:
> 
> * Dynamic content script execution, at document_start time.

See above, the top of my reply.


> * Efficiently serving dynamic content efficiently (e.g. images downloaded
> with the script, inserted into content -- IndexedDB and blobs might help
> except I think they won't cross the extension/content boundary).

I think that it is worthwhile to allow WebExtensions to create blob:-URLs for a specific origin. This can also be beneficial to add-ons such as Decentraleyes. I may create a bugzilla ticket once I flesh out this idea.


> * Generally knowing how/having confidence that my user scripts as content
> scripts (I assume this will be the solution) work, but don't get privileges
> I don't intend them to.  How do I tell one content script from another (in a
> message listener in a background script)?

Content scripts of a single extension are all executed in a single execution environment. There is no isolation.
See bug 1353468 for the request to support proper sandboxes.

Comment 13

2 months ago
> In that content script, synchronously retrieve the set of relevant user scripts for the current document (e.g. using APIs introduced by bug 1332273).

But the body of the script would need to be included.  Single user scripts (not to mention some set of install scripts) easily blow past the 4kbyte limit discussed there.

> - IndexedDB is an async API; sync IndexedDB was eventually removed from the spec due to the lack of implementations.

Oh, crud.  Maybe https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API (and all child pages) shouldn't list all the objects named `IDBFooSync` in the side bar?  So far I only saw that they exist and never clicked on one yet to see the warnings.

> - IndexedDB in a content script uses the database of the page's origin, not of your extension's.

Oh!  Back to the drawing board.  (Never got around to testing those ideas ..)  If I can retrieve a blob handle in background and A) pass that to e.g. `tabs.executeScript() and B) pass that into the src of an <img> via a content script, that still might be good enough.

> ... allow WebExtensions to create blob:-URLs for a specific origin ...

Can that origin be "*://*" ?  I want to create e.g. blobs for images that might be put into the DOM of any page.

Comment 14

a month ago
I don't know how to mark blockers (or if doing so would be appropriate), but:

* I'm very interested in bug 1352217.
* I just filed bug 1356568 for behavior I'd like to see w.r.t. blob URLs.
You need to log in before you can comment on or make changes to this bug.