Bug 1396172 (webext-tb)

[meta] Add support for WebExtensions

NEW
Unassigned

Status

enhancement
2 years ago
4 days ago

People

(Reporter: public, Unassigned)

Tracking

(Depends on 14 bugs, {meta})

unspecified

Thunderbird Tracking Flags

(thunderbird59+ affected, thunderbird60 affected)

Details

Attachments

(1 attachment, 3 obsolete attachments)

Reporter

Description

2 years ago
User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0
Build ID: 20170802111520

Steps to reproduce:

Tried to install a few WE:
https://addons.mozilla.org/en-US/firefox/addon/ublock-origin/
https://addons.mozilla.org/en-US/firefox/addon/scroll_anywhere/



Actual results:

I could not install them.


Expected results:

Some Firefox are really useful for Thunderbird, but Firefox 57 will only run WebExtensions. Can you please add support for WebExtensions in Thunderbird?
Reporter

Comment 1

2 years ago
Only then can uBlock Origin be supported for Thunderbird: https://github.com/gorhill/uBlock/issues/117

Updated

2 years ago
Component: Untriaged → General

Comment 2

2 years ago
See bug 1346918 for tests we have switched off due to Webextensions not being supported in TB. I think that's a little out of date, so the switched-off tests can be found more easily here:
http://searchfox.org/mozilla-central/search?q=appname+%3D%3D+%22thunderbird%22&case=false&regexp=false&path=
http://searchfox.org/mozilla-central/search?q=MOZ_APP_NAME+%3D%3D+%22thunderbird%22&case=false&regexp=false&path=
See Also: → TBWebExtFail
I was sure I had already filed a bug for this, but it seems not. Anyway, confirming we want to at least enable the basics so that we can benefit from some of the features it brings. WebExtensions most likely will not work out of the box for Thunderbird, and to properly support some of the UI facing features we're going to have to invest quite some work (and copy/paste a lot of code, sadly).

I'm going to start this out with a WiP patch that makes WX not break on the content policy we have set. This might be enough to re-enable the tests.

As to what is required to make the UI features work, we'll need to create implementations of some of the ext- scripts in browser/
https://dxr.mozilla.org/comm-central/search?q=path%3Abrowser%2Fcomponents%2Fextensions+path%3Aext-&redirect=false

This shouldn't be too hard for browser action buttons. The tabs/windows implementation will be a lot of work, and it is unclear how we want to map tabs (either content tabs in our tabmail implementation, or message readers, or both. Probably not xul tabs, except maybe to the extent that special tabs can be created and closed. Content scripts should not be able to run there). There are also some APIs we probably don't want to implement or can't (devtools, omnibox, settings/url overrides). Then of course we can start creating new APIs to suit mail features, but that should be coordinated and APIs should not be accepted lightly.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Posted patch WiP patch for basic features - v1 (obsolete) β€” β€” Splinter Review
JΓΆrg, when you are back at work and have some spare time, maybe you can do a try run with this patch and those wx tests enabled again. I suspect at least bug 
1267612 might work again this way.

Comment 5

2 years ago
(BMO only inserts a link if the bug and the number are on the same line: bug 1267612).
Spare time? Never. That said, I can easily enable the disabled tests (which can be found with the searches from comment #2) and see what still fails.

Comment 6

2 years ago
OK, running a few disable tests by hand:

mozilla/mach xpcshell-test toolkit/mozapps/extensions/test/xpcshell/test_webextension_install.js
fails with
TEST_STATUS: Thread-1 test_implicit_id FAIL [test_implicit_id : 25] Add-on signing is enabled - false == true
Hmm, not strictly a w/e failure. We could submit a patch to enable the test but skip that check.

mozilla/mach xpcshell-test toolkit/components/extensions/test/xpcshell/test_csp_custom_policies.js
Passes. Nice.

mozilla/mach xpcshell-test toolkit/mozapps/extensions/test/xpcshell/test_webextension_icons.js
Passes.

toolkit/mozapps/extensions/test/xpcshell/test_webextension.js still fails.
toolkit/mozapps/extensions/test/xpcshell/test_webextension_embedded.js still fails.

mozilla/mach xpcshell-test toolkit/mozapps/extensions/test/xpcshell/test_webextension_install_syntax_error.js
Passes.

toolkit/mozapps/extensions/test/xpcshell/test_ext_management.js hangs.

I'll get this landed and create a patch to re-enable some tests.

Comment 7

2 years ago
Posted patch 1396172-re-enable-tests.patch (obsolete) β€” β€” Splinter Review
These are the tests we can enable.

toolkit/components/extensions/test/xpcshell/test_csp_custom_policies.js is different since there the entire directory is disabled:
http://searchfox.org/mozilla-central/rev/01970ed92d74f82d4e94a1e4365892bbcc593889/toolkit/components/extensions/test/xpcshell/xpcshell.ini#4

So in order to re-enable it, we'd have to re-enable the entire directory and then disable everything that fails.

Here's a patch for the easy stuff in toolkit/mozapps/extensions/test/xpcshell.
Attachment #8918600 - Flags: feedback?(philipp)
Attachment #8918600 - Flags: feedback?(kmaglione+bmo)
Attachment #8918600 - Flags: feedback?(kmaglione+bmo) → feedback+

Comment 8

2 years ago
Comment on attachment 8918600 [details] [diff] [review]
1396172-re-enable-tests.patch

Thanks Kris. I need r+ to get that landed ;-)
Any idea why I need to disable some sub-tests?
Attachment #8918600 - Flags: review?(kmaglione+bmo)

Comment 9

2 years ago
Here we have a try with both patches and also re-enabling tests in toolkit/components/extensions/test/xpcshell:
http://searchfox.org/mozilla-central/rev/01970ed92d74f82d4e94a1e4365892bbcc593889/toolkit/components/extensions/test/xpcshell/xpcshell.ini#4
test_csp_custom_policies.js should pass but there will be many tests that don't pass there.

Try:
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=1b752dbabc80a2955e3a970a138d2a949eef95e2

Comment 10

2 years ago
Attachment #8918594 - Attachment is obsolete: true
Attachment #8918621 - Flags: review+
Comment on attachment 8918600 [details] [diff] [review]
1396172-re-enable-tests.patch

I think the pattern that is more common is:

add_task({
  skip_if: () => mozinfo.appname == "thunderbird"
}, async function test_the_thing() {
...
});

This way the test doesn't even show in the list and you don't need to load appconstants. Untested though.
Attachment #8918600 - Flags: feedback?(philipp) → feedback+

Comment 13

2 years ago
Pushed by mozilla@jorgk.com:
https://hg.mozilla.org/comm-central/rev/c91a3bb11073
Add support for WebExtensions: Part 1: Content Policy. r=jorgk
Keywords: checkin-needed

Comment 14

2 years ago
(In reply to Jorg K (GMT+2) from comment #9)
> Try:
> https://treeherder.mozilla.org/#/jobs?repo=try-comm-
> central&revision=1b752dbabc80a2955e3a970a138d2a949eef95e2
That didn't work at all due to some infra problem, so here again:
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=f97a372cb31aca6f4ace33b3031532f9f5799a7f

Updated

2 years ago
Attachment #8918621 - Attachment description: Part 1: Content policy → Part 1: Content policy [landed comment #13]

Comment 15

2 years ago
Re-enabling tests in toolkit/components/extensions/test/xpcshell doesn't appear to be a good idea since 55 of them time out and another 5 fail:
https://archive.mozilla.org/pub/thunderbird/try-builds/mozilla@jorgk.com-f97a372cb31aca6f4ace33b3031532f9f5799a7f/try-comm-central-linux64/try-comm-central_ubuntu64_vm_test-xpcshell-bm54-tests1-linux64-build9.txt.gz

So we need to invest more work so see why those fail. For now, we can only enable three tests as per attachment 1396172-re-enable-tests.patch.

Kris, perhaps you can give us some details of what we need to do to get TB "WebExtension" ready. As a start, we don't want to define our own interfaces, but only run "normal" FF W/E like uBlock.
Attachment #8918600 - Flags: review?(kmaglione+bmo) → review+
(In reply to Jorg K (GMT+2) from comment #8)
> Comment on attachment 8918600 [details] [diff] [review]
> 1396172-re-enable-tests.patch
> 
> Thanks Kris. I need r+ to get that landed ;-)
> Any idea why I need to disable some sub-tests?

Sorry for the delay. I was sick most of last week; didn't get much done.

I'm not sure why the permissions tests don't work. How were they failing?

The other tests depend on the new signing system, which as far as I know Thunderbird still doesn't implement.

Comment 17

2 years ago
Posted file 1396172-failures.txt (obsolete) β€”
Kris, thanks for the comments. I'm not sure what you're referring to with
> The other tests depend on the new signing system ...
You're referring to test_implicit_id[_temp] or the tests in toolkit/components/extensions/test/xpcshell which mostly time out or fail?

My patch disables four tests: test_implicit_id, test_implicit_id_temp, test_permissions_prompt and test_permissions_prompt_cancel. Assuming that you were referring to the former two, I enabled the two permission tests again to check how they fail.

mozilla/mach xpcshell-test toolkit/mozapps/extensions/test/xpcshell/test_webextension_install.js fails with:
 0:05.90 TEST_STATUS: Thread-1 test_permissions_prompt FAIL [test_permissions_prompt : 522] Permission handler was invoked - "undefined" != "undefined"

Full log enclosed. Do you think those two can be made to pass?
Flags: needinfo?(kmaglione+bmo)
Requesting tracking as without having basic support for WebExtensions, add-ons will now not be able to have preference displays, as the legacy preference bindings have been removed from toolkit (bug 1414406).
Blocks: 1418914
(In reply to Mark Banner (:standard8) from comment #18)
> Requesting tracking as without having basic support for WebExtensions,
> add-ons will now not be able to have preference displays, as the legacy
> preference bindings have been removed from toolkit (bug 1414406).

Ah I thought this was just about the inline options feature, not the very legacy way of add-ons showing preferences. To be able to provide compat for existing add-ons using inline options we might need to resurrect those bindings in c-c and hope that the underlying xbl stuff is not removed before 59.

With the patch landed in comment 13, it is possible to install WebExtensions in Thunderbird actually. What is not working is any kind of API that is in browser/ which also includes the tabs API, and we don't have any mail-specific APIs.

I don't see that we will have the energy to provide this, and transition our add-ons in time for 59, so I fear we'll have to do everything possible to keep legacy add-ons running.

JΓΆrg, as you approved tracking for 59, what is the scope you expect for this bug?

Comment 20

2 years ago
(In reply to Philipp Kewisch [:Fallen]  from comment #19)
> JΓΆrg, as you approved tracking for 59, what is the scope you expect for this
> bug?
Hmm, I'm perhaps just ignorant about the difference of "tracking ?" and "tracking +". I guess it's an item we need to keep an eye on for TB 59. I don't have a scope in mind.

Previously tracking flags were mostly cosmetic, we have shipped ESR versions without tracked bugs being fixed. This time we won't have that option since if we're not fixing the add-on issues I've just summaries on Maildev "Add-ons under threat in TB", TB 59 will just be unusable. Also note bug 1418914.
(In reply to Philipp Kewisch [:Fallen]  from comment #19)
> (In reply to Mark Banner (:standard8) from comment #18)
> > Requesting tracking as without having basic support for WebExtensions,
> > add-ons will now not be able to have preference displays, as the legacy
> > preference bindings have been removed from toolkit (bug 1414406).
> 
> Ah I thought this was just about the inline options feature, not the very
> legacy way of add-ons showing preferences. To be able to provide compat for
> existing add-ons using inline options we might need to resurrect those
> bindings in c-c and hope that the underlying xbl stuff is not removed before
> 59.

When I said that, I hadn't actually realised this bug had already landed, and it was just the tests to fix up.

It turns out that it is actually bug 1418914 that is preventing from a WebExtension being used for in-line preferences.

Resurrecting the xbl stuff might be an option, but I suspect isn't sustainable, and may be hard to hook into the add-ons manager (especially now the support for the install.rdf options is also removed, so you can't install an add-on declaring the old-style inline preferences).

> With the patch landed in comment 13, it is possible to install WebExtensions
> in Thunderbird actually. What is not working is any kind of API that is in
> browser/ which also includes the tabs API, and we don't have any
> mail-specific APIs.

We don't need mail specific APIs. We can use hybrid extensions (aka embedded WebExtensions), https://developer.mozilla.org/Add-ons/WebExtensions/Embedded_WebExtensions. It is a little more complicated in some areas, but it is possible to do, and a good transition stage if the long-term furture might be for WebExtensions.
Alias: webext-tb
Flags: needinfo?(kmaglione+bmo)
Keywords: leave-openmeta
Magnus, I'd suggest we create a WebExtensions component in Thunderbird. Thoughts?
Flags: needinfo?(mkmelin+mozilla)
Sure, could be useful.
Flags: needinfo?(mkmelin+mozilla)

Updated

a year ago
Blocks: webext

Updated

11 months ago
Depends on: 1468667

Updated

11 months ago
Depends on: 1469238

Comment 24

10 months ago
(In reply to Mark Banner (:standard8) (away 13 - 17 Aug) from comment #21)
> We don't need mail specific APIs.

Of course not directly. So far even the "Browser API" isn't completed.

But is there any chance that there will be "mail specific" APIs in the future? I think there should be at least an *optional* possibility to have some API for the most common Addon types.

IMHO the current "API", which (like in pre-WebExtension Firefox versions) are just the internal interfaces which tend to change regularly, is pretty horrible. Especially for beginners or everyone without deep insight in the backend code.
(In reply to Manuel Reimer from comment #24)
> (In reply to Mark Banner (:standard8) (away 13 - 17 Aug) from comment #21)
> > We don't need mail specific APIs.
> 
> Of course not directly. So far even the "Browser API" isn't completed.

My comment was taken slightly out of context - the comment was with respect to being able to do various things easily in Thunderbird 60, so that extensions could easily upgrade and get working again (e.g. preferences). However, other workarounds were made and there was enough for extensions without needing WebExtension parts (and I could never get it to work reliably with a hybrid version anyway...).

Off-topic, but the browser API is never going to be "completed" - there will always be more features to add, or different extension points that people want.

> But is there any chance that there will be "mail specific" APIs in the
> future? I think there should be at least an *optional* possibility to have
> some API for the most common Addon types.

Well if you look at the dependencies of this bug, you'll see there's already a couple mentioned...


IMO it is unclear what is left in this bug to do - I believe the basic WebExtension support is there now, maybe this should be closed, and a new bug set up to track the implementation of various web extension APIs.
This is actually the meta-but for WebExtensions. Given the amount of comments and patches landed we could consider creating a new one though.
Summary: Add support for WebExtensions → [meta] Add support for WebExtensions

Updated

9 months ago
No longer blocks: 1418914
Depends on: 1418914

Updated

9 months ago
Attachment #8922244 - Attachment is obsolete: true

Comment 27

9 months ago
Comment on attachment 8918600 [details] [diff] [review]
1396172-re-enable-tests.patch

Never landed, a more comprehensive patch landed in bug 1468667.
Attachment #8918600 - Attachment is obsolete: true
Depends on: 1487008
Depends on: 1487017
Component: General → Add-Ons: Extensions API
Depends on: 1499617
Here's a rough list I wrote the other day of the next stuff I'm working on. Everything under mailTab now has some more meat on the bones in bug 1499617; the accounts and messages I'll be working on in bug 1488176.

https://gist.github.com/darktrojan/d5102d4505298edc4535528ea211f6c6
Depends on: 1503421
Depends on: 1503423
Depends on: 1504475
Depends on: 1520338
Depends on: 1520367
Depends on: 1531591
Depends on: 1531593
Depends on: 1531317
Depends on: 1531597
Depends on: 1538590

Updated

2 months ago
Depends on: 1530122

Comment 29

2 months ago

The bug 1525274 is a blocker for my add-on (I seem to not have the permissions to set it as a blocker for this bug)

Depends on: 1525274
Depends on: 1542643
Depends on: 1529785
Reporter

Comment 30

a month ago

(In reply to public from comment #1)

Only then can uBlock Origin be supported for Thunderbird:
https://github.com/gorhill/uBlock/issues/117

uBlock Origin issue merged to https://github.com/gorhill/uBlock/issues/3698

Depends on: 1546915
You need to log in before you can comment on or make changes to this bug.