1396172 - (webext-tb) [meta] Add support for WebExtensions

Status: RESOLVED FIXED

(Thunderbird :: Add-Ons: Extensions API, enhancement)

Description

[David Hedlund]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0
Build ID: 20170802111520

Steps to reproduce:

Tried to install a few WE:
https://addons.mozilla.org/en-US/firefox/addon/ublock-origin/
https://addons.mozilla.org/en-US/firefox/addon/scroll_anywhere/



Actual results:

I could not install them.


Expected results:

Some Firefox are really useful for Thunderbird, but Firefox 57 will only run WebExtensions. Can you please add support for WebExtensions in Thunderbird?

Comment 1

[David Hedlund]

Only then can uBlock Origin be supported for Thunderbird: https://github.com/gorhill/uBlock/issues/117

Comment 2

[Jorg K (CEST = GMT+2)]

See bug 1346918 for tests we have switched off due to Webextensions not being supported in TB. I think that's a little out of date, so the switched-off tests can be found more easily here:
http://searchfox.org/mozilla-central/search?q=appname+%3D%3D+%22thunderbird%22&case=false&regexp=false&path=
http://searchfox.org/mozilla-central/search?q=MOZ_APP_NAME+%3D%3D+%22thunderbird%22&case=false&regexp=false&path=

Comment 3

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

I was sure I had already filed a bug for this, but it seems not. Anyway, confirming we want to at least enable the basics so that we can benefit from some of the features it brings. WebExtensions most likely will not work out of the box for Thunderbird, and to properly support some of the UI facing features we're going to have to invest quite some work (and copy/paste a lot of code, sadly).

I'm going to start this out with a WiP patch that makes WX not break on the content policy we have set. This might be enough to re-enable the tests.

As to what is required to make the UI features work, we'll need to create implementations of some of the ext- scripts in browser/
https://dxr.mozilla.org/comm-central/search?q=path%3Abrowser%2Fcomponents%2Fextensions+path%3Aext-&redirect=false

This shouldn't be too hard for browser action buttons. The tabs/windows implementation will be a lot of work, and it is unclear how we want to map tabs (either content tabs in our tabmail implementation, or message readers, or both. Probably not xul tabs, except maybe to the extent that special tabs can be created and closed. Content scripts should not be able to run there). There are also some APIs we probably don't want to implement or can't (devtools, omnibox, settings/url overrides). Then of course we can start creating new APIs to suit mail features, but that should be coordinated and APIs should not be accepted lightly.

Comment 4

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Attachment: WiP patch for basic features - v1

Jörg, when you are back at work and have some spare time, maybe you can do a try run with this patch and those wx tests enabled again. I suspect at least bug 
1267612 might work again this way.

Comment 5

[Jorg K (CEST = GMT+2)]

(BMO only inserts a link if the bug and the number are on the same line: bug 1267612).
Spare time? Never. That said, I can easily enable the disabled tests (which can be found with the searches from comment #2) and see what still fails.

Comment 6

[Jorg K (CEST = GMT+2)]

OK, running a few disable tests by hand:

mozilla/mach xpcshell-test toolkit/mozapps/extensions/test/xpcshell/test_webextension_install.js
fails with
TEST_STATUS: Thread-1 test_implicit_id FAIL [test_implicit_id : 25] Add-on signing is enabled - false == true
Hmm, not strictly a w/e failure. We could submit a patch to enable the test but skip that check.

mozilla/mach xpcshell-test toolkit/components/extensions/test/xpcshell/test_csp_custom_policies.js
Passes. Nice.

mozilla/mach xpcshell-test toolkit/mozapps/extensions/test/xpcshell/test_webextension_icons.js
Passes.

toolkit/mozapps/extensions/test/xpcshell/test_webextension.js still fails.
toolkit/mozapps/extensions/test/xpcshell/test_webextension_embedded.js still fails.

mozilla/mach xpcshell-test toolkit/mozapps/extensions/test/xpcshell/test_webextension_install_syntax_error.js
Passes.

toolkit/mozapps/extensions/test/xpcshell/test_ext_management.js hangs.

I'll get this landed and create a patch to re-enable some tests.

Comment 7

[Jorg K (CEST = GMT+2)]

Attachment: 1396172-re-enable-tests.patch

These are the tests we can enable.

toolkit/components/extensions/test/xpcshell/test_csp_custom_policies.js is different since there the entire directory is disabled:
http://searchfox.org/mozilla-central/rev/01970ed92d74f82d4e94a1e4365892bbcc593889/toolkit/components/extensions/test/xpcshell/xpcshell.ini#4

So in order to re-enable it, we'd have to re-enable the entire directory and then disable everything that fails.

Here's a patch for the easy stuff in toolkit/mozapps/extensions/test/xpcshell.

Comment 8

[Jorg K (CEST = GMT+2)]

Comment on attachment 8918600 [details] [diff] [review]
1396172-re-enable-tests.patch

Thanks Kris. I need r+ to get that landed ;-)
Any idea why I need to disable some sub-tests?

Comment 9

[Jorg K (CEST = GMT+2)]

Here we have a try with both patches and also re-enabling tests in toolkit/components/extensions/test/xpcshell:
http://searchfox.org/mozilla-central/rev/01970ed92d74f82d4e94a1e4365892bbcc593889/toolkit/components/extensions/test/xpcshell/xpcshell.ini#4
test_csp_custom_policies.js should pass but there will be many tests that don't pass there.

Try:
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=1b752dbabc80a2955e3a970a138d2a949eef95e2

Comment 10

[Jorg K (CEST = GMT+2)]

Attachment: Part 1: Content policy [landed comment #13]

Comment 11

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Comment on attachment 8918600 [details] [diff] [review]
1396172-re-enable-tests.patch

I think the pattern that is more common is:

add_task({
  skip_if: () => mozinfo.appname == "thunderbird"
}, async function test_the_thing() {
...
});

This way the test doesn't even show in the list and you don't need to load appconstants. Untested though.

Comment 12

[Jorg K (CEST = GMT+2)]

We have a few variations:
http://searchfox.org/mozilla-central/source/toolkit/components/telemetry/tests/unit/test_TelemetryEnvironment.js#1639
http://searchfox.org/mozilla-central/source/toolkit/mozapps/extensions/test/xpcshell/test_reload.js#35

BTW, I added a commit message to Philipp's patch and I'm going to land it soon.

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/c91a3bb11073
Add support for WebExtensions: Part 1: Content Policy. r=jorgk

Comment 14

[Jorg K (CEST = GMT+2)]

(In reply to Jorg K (GMT+2) from comment #9)
> Try:
> https://treeherder.mozilla.org/#/jobs?repo=try-comm-
> central&revision=1b752dbabc80a2955e3a970a138d2a949eef95e2
That didn't work at all due to some infra problem, so here again:
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=f97a372cb31aca6f4ace33b3031532f9f5799a7f

Comment 15

[Jorg K (CEST = GMT+2)]

Re-enabling tests in toolkit/components/extensions/test/xpcshell doesn't appear to be a good idea since 55 of them time out and another 5 fail:
https://archive.mozilla.org/pub/thunderbird/try-builds/[e-mail]-f97a372cb31aca6f4ace33b3031532f9f5799a7f/try-comm-central-linux64/try-comm-central_ubuntu64_vm_test-xpcshell-bm54-tests1-linux64-build9.txt.gz

So we need to invest more work so see why those fail. For now, we can only enable three tests as per attachment 1396172-re-enable-tests.patch.

Kris, perhaps you can give us some details of what we need to do to get TB "WebExtension" ready. As a start, we don't want to define our own interfaces, but only run "normal" FF W/E like uBlock.

Comment 16

[Kris Maglione [:kmag]]

(In reply to Jorg K (GMT+2) from comment #8)
> Comment on attachment 8918600 [details] [diff] [review]
> 1396172-re-enable-tests.patch
> 
> Thanks Kris. I need r+ to get that landed ;-)
> Any idea why I need to disable some sub-tests?

Sorry for the delay. I was sick most of last week; didn't get much done.

I'm not sure why the permissions tests don't work. How were they failing?

The other tests depend on the new signing system, which as far as I know Thunderbird still doesn't implement.

Comment 17

[Jorg K (CEST = GMT+2)]

Attachment: 1396172-failures.txt

Kris, thanks for the comments. I'm not sure what you're referring to with
> The other tests depend on the new signing system ...
You're referring to test_implicit_id[_temp] or the tests in toolkit/components/extensions/test/xpcshell which mostly time out or fail?

My patch disables four tests: test_implicit_id, test_implicit_id_temp, test_permissions_prompt and test_permissions_prompt_cancel. Assuming that you were referring to the former two, I enabled the two permission tests again to check how they fail.

mozilla/mach xpcshell-test toolkit/mozapps/extensions/test/xpcshell/test_webextension_install.js fails with:
 0:05.90 TEST_STATUS: Thread-1 test_permissions_prompt FAIL [test_permissions_prompt : 522] Permission handler was invoked - "undefined" != "undefined"

Full log enclosed. Do you think those two can be made to pass?

Comment 18

[Mark Banner (:standard8)]

Requesting tracking as without having basic support for WebExtensions, add-ons will now not be able to have preference displays, as the legacy preference bindings have been removed from toolkit (bug 1414406).

Comment 19

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

(In reply to Mark Banner (:standard8) from comment #18)
> Requesting tracking as without having basic support for WebExtensions,
> add-ons will now not be able to have preference displays, as the legacy
> preference bindings have been removed from toolkit (bug 1414406).

Ah I thought this was just about the inline options feature, not the very legacy way of add-ons showing preferences. To be able to provide compat for existing add-ons using inline options we might need to resurrect those bindings in c-c and hope that the underlying xbl stuff is not removed before 59.

With the patch landed in comment 13, it is possible to install WebExtensions in Thunderbird actually. What is not working is any kind of API that is in browser/ which also includes the tabs API, and we don't have any mail-specific APIs.

I don't see that we will have the energy to provide this, and transition our add-ons in time for 59, so I fear we'll have to do everything possible to keep legacy add-ons running.

Jörg, as you approved tracking for 59, what is the scope you expect for this bug?

Comment 20

[Jorg K (CEST = GMT+2)]

(In reply to Philipp Kewisch [:Fallen]  from comment #19)
> Jörg, as you approved tracking for 59, what is the scope you expect for this
> bug?
Hmm, I'm perhaps just ignorant about the difference of "tracking ?" and "tracking +". I guess it's an item we need to keep an eye on for TB 59. I don't have a scope in mind.

Previously tracking flags were mostly cosmetic, we have shipped ESR versions without tracked bugs being fixed. This time we won't have that option since if we're not fixing the add-on issues I've just summaries on Maildev "Add-ons under threat in TB", TB 59 will just be unusable. Also note bug 1418914.

Comment 21

[Mark Banner (:standard8)]

(In reply to Philipp Kewisch [:Fallen]  from comment #19)
> (In reply to Mark Banner (:standard8) from comment #18)
> > Requesting tracking as without having basic support for WebExtensions,
> > add-ons will now not be able to have preference displays, as the legacy
> > preference bindings have been removed from toolkit (bug 1414406).
> 
> Ah I thought this was just about the inline options feature, not the very
> legacy way of add-ons showing preferences. To be able to provide compat for
> existing add-ons using inline options we might need to resurrect those
> bindings in c-c and hope that the underlying xbl stuff is not removed before
> 59.

When I said that, I hadn't actually realised this bug had already landed, and it was just the tests to fix up.

It turns out that it is actually bug 1418914 that is preventing from a WebExtension being used for in-line preferences.

Resurrecting the xbl stuff might be an option, but I suspect isn't sustainable, and may be hard to hook into the add-ons manager (especially now the support for the install.rdf options is also removed, so you can't install an add-on declaring the old-style inline preferences).

> With the patch landed in comment 13, it is possible to install WebExtensions
> in Thunderbird actually. What is not working is any kind of API that is in
> browser/ which also includes the tabs API, and we don't have any
> mail-specific APIs.

We don't need mail specific APIs. We can use hybrid extensions (aka embedded WebExtensions), https://developer.mozilla.org/Add-ons/WebExtensions/Embedded_WebExtensions. It is a little more complicated in some areas, but it is possible to do, and a good transition stage if the long-term furture might be for WebExtensions.

Comment 22

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Magnus, I'd suggest we create a WebExtensions component in Thunderbird. Thoughts?

Comment 23

[Magnus Melin [:mkmelin]]

Sure, could be useful.

Comment 24

[Manuel Reimer]

(In reply to Mark Banner (:standard8) (away 13 - 17 Aug) from comment #21)
> We don't need mail specific APIs.

Of course not directly. So far even the "Browser API" isn't completed.

But is there any chance that there will be "mail specific" APIs in the future? I think there should be at least an *optional* possibility to have some API for the most common Addon types.

IMHO the current "API", which (like in pre-WebExtension Firefox versions) are just the internal interfaces which tend to change regularly, is pretty horrible. Especially for beginners or everyone without deep insight in the backend code.

Comment 25

[Mark Banner (:standard8)]

(In reply to Manuel Reimer from comment #24)
> (In reply to Mark Banner (:standard8) (away 13 - 17 Aug) from comment #21)
> > We don't need mail specific APIs.
> 
> Of course not directly. So far even the "Browser API" isn't completed.

My comment was taken slightly out of context - the comment was with respect to being able to do various things easily in Thunderbird 60, so that extensions could easily upgrade and get working again (e.g. preferences). However, other workarounds were made and there was enough for extensions without needing WebExtension parts (and I could never get it to work reliably with a hybrid version anyway...).

Off-topic, but the browser API is never going to be "completed" - there will always be more features to add, or different extension points that people want.

> But is there any chance that there will be "mail specific" APIs in the
> future? I think there should be at least an *optional* possibility to have
> some API for the most common Addon types.

Well if you look at the dependencies of this bug, you'll see there's already a couple mentioned...


IMO it is unclear what is left in this bug to do - I believe the basic WebExtension support is there now, maybe this should be closed, and a new bug set up to track the implementation of various web extension APIs.

Comment 26

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

This is actually the meta-but for WebExtensions. Given the amount of comments and patches landed we could consider creating a new one though.

Comment 27

[Jorg K (CEST = GMT+2)]

Comment on attachment 8918600 [details] [diff] [review]
1396172-re-enable-tests.patch

Never landed, a more comprehensive patch landed in bug 1468667.

Comment 28

[Geoff Lankow (:darktrojan)]

Here's a rough list I wrote the other day of the next stuff I'm working on. Everything under mailTab now has some more meat on the bones in bug 1499617; the accounts and messages I'll be working on in bug 1488176.

https://gist.github.com/darktrojan/d5102d4505298edc4535528ea211f6c6

Comment 29

[plieser]

The bug 1525274 is a blocker for my add-on (I seem to not have the permissions to set it as a blocker for this bug)

Comment 30

[David Hedlund]

(In reply to public from comment #1)

Only then can uBlock Origin be supported for Thunderbird:
https://github.com/gorhill/uBlock/issues/117

uBlock Origin issue merged to https://github.com/gorhill/uBlock/issues/3698

Comment 31

[michaelpopepublic]

Not quite sure where to put this... but are there any plans to expose the email address in browser.accounts.list()?

I'm writing an extension which synchronizes email client tags (in a way some imap servers don't allow), and namespacing the synced data by email address would be helpful.

Not quite sure if this is the right place to put a comment like this though.

Comment 32

[Geoff Lankow (:darktrojan)]

Not directly, no, because an account can have more than one email address associated with it. Bug 1531593 is the one to follow, but as you can see, there's no progress yet.

Comment 33

[Geoff Lankow (:darktrojan)]

From this point on I am not using this bug. It was intended as a meta-bug for feature requests, but it has become a magnet for anything to do with WebExtensions, which is unnecessary because that's what the "Add-Ons: Extensions API" category is all about.

Comment 34

[David Hedlund]

Status from https://developer.thunderbird.net/add-ons/about-add-ons#extensions

• MailExtension: The only valid extension type in Thunderbird 74 and later.
• Legacy WebExtensions: Deprecated in Thunderbird 74.
• Legacy Bootstrap Extension: Deprecated in Thunderbird 68.
• Legacy Overlay Extension: Deprecated in Thunderbird 68.

Current releases and release notes:

• Thunderbird 68: https://archive.mozilla.org/pub/thunderbird/releases/68.0/, https://www.thunderbird.net/en-US/thunderbird/68.0/releasenotes/
• Thunderbird 74 beta 1: https://archive.mozilla.org/pub/thunderbird/releases/74.0b1/, https://www.thunderbird.net/en-US/thunderbird/74.0beta/releasenotes/

Release schedule:

• Thunderbird tracks the ESR schedule. Firefox ESR 78.3 is expected to be released in 2020-09-22. - https://wiki.mozilla.org/Release_Management/Calendar