1490598 - Enable automatic language pack upload to ATN via Taskcluster.

Status: RESOLVED FIXED

(Thunderbird :: Add-Ons: General, defect)

Description

[franz.trischberger]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0
Build ID: 20180830143136

Steps to reproduce:

I tried to install Deutsch Language Pack.


Actual results:

It does not work (install button deactivated) as the Language Pack is only compatible with 52.0 and not the most recent 60.0 Release.


Expected results:

The Deutsch Language Pack should be available for 60.0, too (which was released more than a month ago).

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

This affects all language packs by Mozilla on addons.thunderbird.net. The German one can be found at https://addons.thunderbird.net/en-US/thunderbird/addon/deutsch-de-language-pack-thund/

Andreas, do you know whom to talk to to upload the new language packs.

Franz, for now you can download the language packs from http://ftp.mozilla.org/pub/thunderbird/releases/60.0/win64/xpi/ de.xpi is German. Right-click and save the links.

Comment 2

[franz.trischberger]

Sebastian: Thx for the reply.
I tried to get it working with the linked de.xpi. Still all in English. Tried it.xpi, same issue.
They get copied to default_profile/extensions and renamed (looks like in firefox) but apparently not used.
It is a different issue, I think. Do you know if it is already reported?
How can I debug this issue?

Comment 3

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

You still need to manually activate the language pack:

1. Go to Thunderbird's Options.
2. Go to "Advanced" > "General"
3. Open the "Config Editor" and confirm the warning.
4. Right-click the list and choose "New" > "String".
5. Use as name intl.locale.requested and set the value to de
6. Close Thunderbird and launch it again.

Comment 4

[franz.trischberger]

Thank you very much! That did the trick.
And sorry for the dumb questions. Never had to use the advanced config editor before.

Comment 5

[Andrei Hajdukewycz [:sancus]]

This is a little weird. I've fixed the issue preventing web extensions from uploading to addons.thunderbird.net, but they can't be approved right now because addons-server tries to sign them, and we don't have a working signing server nor do I think we can even have one.

I don't know if I can just disable signing in the code or if that will produce language packs that won't enable by default or what.

Comment 6

[Andrei Hajdukewycz [:sancus]]

I got this to upload by disabling signing, but addons-server only recognizes web extensions as compatible with Firefox by default, so you have to manually force compatibility for Thunderbird on the versions page (https://addons.thunderbird.net/en-US/developers/addon/deutsch-de-language-pack-thund/versions/2468594) otherwise it won't be installable for Thunderbird and in fact the page will break because it tries to redirect you to AMO, since the addon is only compatible with Firefox, creating a circular redirect.

Comment 7

[Andrei Hajdukewycz [:sancus]]

Also, all the versions have a buildid string in them, which seems less than ideal. "Version 60.0buildid20180731173940" is what is displayed after upload, rather than Version 60.0.

Comment 8

[franz.trischberger]

Thx Andrei! The language pack is available now and auto update from 52 to 60 worked.
Unfortunately the GUI is still in English. I have to manually set intl.locale.requested in the advanced config editor.

Comment 9

[Andrei Hajdukewycz [:sancus]]

Is it normal to have to manually enable these? I don't really know how they're supposed to work.

Comment 10

[franz.trischberger]

Before the 60.0 switch it usually was enough to install and enable the language pack addon.

Comment 11

[Philipp Kewisch [:Fallen][📱📆]]

So for Firefox, there is some releng magic that creates and uploads the language packs from automation, hence the 60.0buildid20180731173940 version number. We may have to do something like this as well. ni? rjl to figure this out.

It would also be great if on addons.thunderbird.net we could assume that WebExtensions are (only) compatible with Thunderbird by default.

Comment 12

[Andrei Hajdukewycz [:sancus]]

(In reply to Philipp Kewisch [:Fallen] [:📆] from comment #11)
> It would also be great if on addons.thunderbird.net we could assume that
> WebExtensions are (only) compatible with Thunderbird by default.

If we do this, it does mean that you will be able to upload any Firefox web extension, it will be auto approved and available for download/install, but probably won't work. Maybe that's not a big deal, but it is a regression from legacy extensions, because Web Extensions don't have any application compatibility information beyond "gecko". I dunno if this is intentional, if we're supposed to be guessing the application compatibility based on the permissions and chrome_resources used by the extension, but it seems like it'd be a lot easier if we could just require addiong Thunderbird to the application list for all TB Web Extensions manifests. Maybe that goes against the design intention though, I don't really understand how this was supposed to work.

Comment 13

[Rob Lemley [:rjl]]

I took a real quick look, and there are some taskcluster tasks defined for Firefox that deal with signing and uploading langpack addons:
release-sign-and-push-langpacks
release-beetmover-signed-langpacks
release-beetmover-signed-langpacks-checksums

Those would need to be ported over to Thunderbird. Additionally, new scopes will need to be created within Taskcluster to support uploading to addons.thunderbird.net.

Comment 15

[Wayne Mery (:wsmwk)]

Counting up some of what I am guessing are the most used languages, I get 600,000 users. We failed to track this for release. So we need to prioritize fixing this. Rob, how long might this take and can it go to the top of your list?

Comment 16

[Rob Lemley [:rjl]]

I don't have a good sense of how long this will take unfortunately. A lot depends on ATN and what it's capabilities are, and at the moment it's a black box for me. I'll try to pull together what needs to be done. I'll likely need input/help from :sancus.
In the short-term, can we set up a knowledgebase article or wiki page with the alternate install instructions described earlier in this bug to grab the XPI from the ftp site and install that way? (See comment 1 and comment 3) I realize it's not ideal, but we can at least get people going that way.

Comment 17

[Jorg K (CEST = GMT+2)]

(In reply to Wayne Mery (:wsmwk) from comment #15)
> Counting up some of what I am guessing are the most used languages, I get
> 600,000 users.
Wayne, can you please clarify this. What are these language packs used for? Everyone can download a localised version of TB with the language pack already installed. In TB 60 it's quite cumbersome to activate an installed language pack or switch between them. Somehow I'm missing the urgency of this.

Comment 18

[Wayne Mery (:wsmwk)]

I'm not an intl user, so I have no clue to answer either question of what and urgency. But there are two bug reporters who can be asked, plus whoever is in SUMO such as https://support.mozilla.org/en-US/questions/1240358

Perhaps there is a lot of overlap, or many users don't care. Cut 600k in half and it still a high percentage of users, so this at least deserves a strong assessment by us.

Comment 19

[Jorg K (CEST = GMT+2)]

Even if we fix this, those 600.000 users will then run into bug 1437480. They'll need to set pref "intl.locale.requested". Exposing this in the UI has only arrived in TB 63 in bug 1481852 which we can't backport.

Comment 20

[Michal Stanke (Mozilla.cz) [:mstanke][:MikkCZ]]

The language packs are often used for:
- portable version of TB
- Linux distros if they do not provide langpack as a package in their repository
- if someone installs English version of TB and consider installing different version not-safe or too difficult
- or if they simply do not know it is possible (usually other sw allows to switch languages seamlessly compared to TB).

Comment 21

[Philipp Kewisch [:Fallen][📱📆]]

(In reply to Rob Lemley [:rjl] from comment #16)
> I don't have a good sense of how long this will take unfortunately. A lot
> depends on ATN and what it's capabilities are, and at the moment it's a
> black box for me.

Firefox has a job that does this for AMO, so you should be able to copy a lot from there. Essentially there is a user that does the upload via the API, and the secret is stored with the others. Hope it helps!

Comment 22

[Wayne Mery (:wsmwk)]

> Firefox has a job that does this for AMO,  

Back to Rob :)

Comment 23

[Rob Lemley [:rjl]]

I will see what I can come up with.

Comment 24

[Rob Lemley [:rjl]]

Attachment: tb_langpack_ATN_taskcluster.patch

There is work being done on the Firefox langpack addons process in bug
1566298, so this is an opportune time to get these working on Thunderbird,
uploading to addons.thunderbird.net.

First pass at porting the Taskcluster kind files from Firefox.

Comment 25

[Justin Wood (:Callek)]

Comment on attachment 9078323 [details] [diff] [review]
tb_langpack_ATN_taskcluster.patch

Note: This reflects the more-recent work I'm doing here -- however it does not yet work (I need to get autograph support for a new signing user that can sign addons, AND I need to modify the AMO submission user to be @mozilla.com -- you may want to chat with me before implementing :-)

Comment 26

[Wayne Mery (:wsmwk)]

Thanks Rob. Great to see this making progress. Intl issues sometimes don't get enough attention.

Comment 27

[Magnus Melin [:mkmelin] (away, back Aug 4)]

Lang packs are here though: http://ftp.mozilla.org/pub/thunderbird/releases/60.8.0/linux-x86_64/xpi/

Comment 28

[Onno Ekker [:nONoNonO UTC+1]]

Did someone manually upload German (Deutch) language pack for TB60? All others I've checked are still compatible with TB52. If this bug can't be easily fixed, I'm willing to add all language packs to ATN, if someone gives me the rights to upload them…

Comment 29

[Onno Ekker [:nONoNonO UTC+1]]

Now that all the language packs that are available for TB 68 have been manually uploaded and are available on ATN (thank you sancus!), maybe also multilingual can be enabled? I can confirm that if I toggle intl.multilingual.enabled and intl.multilingual.downloadEnabled to true, I can search for new languages in TB68, I can download them and I can switch to them!

Comment 30

[Magnus Melin [:mkmelin] (away, back Aug 4)]

It was enabled in bug 1590206.

Comment 31

[Michal Stanke (Mozilla.cz) [:mstanke][:MikkCZ]]

Not sure if the languages pack upload are broken again, or never worked completely, however at least the Czech language pack is missing on ATN, see https://addons.thunderbird.net/thunderbird/language-tools/. IIRC it's supposed to be here https://addons.mozilla.org/thunderbird/addon/czech-cz-language-pack-thunder/.

Czech is definitely the only one, Slovak is missing too. Both are available on FTP https://archive.mozilla.org/pub/thunderbird/releases/68.8.0/linux-x86_64/xpi/.

Comment 32

[Rob Lemley [:rjl]]

Getting this out of my queue until ATN is able to support this automatically.

Comment 33

[zetka]

Still the same situation - Thunderbird 78. In the options there is no possibility to add another language. When trying to search for new language, nothing appears. I am interested in Czech version.

Comment 34

[Rob Lemley [:rjl]]

Until they are uploaded to ATN, you can download them from https://archive.mozilla.org/pub/thunderbird/releases/78.2.1/linux-x86_64/xpi/. (78.2.1 is the latest at time of comment).

Comment 35

[Dave Miller [:justdave]]

If someone can catch me up real quick here, it looks (from comment 24) like this actually got implemented at one point, but didn't actually work? Or did that not actually get committed? Is it worth building on top of that and debugging it or should I start over copying what Firefox does now and trying to port that over?

Comment 36

[Rob Lemley [:rjl]]

(In reply to Dave Miller [:justdave] ([e-mail]) from comment #35)

If someone can catch me up real quick here, it looks (from comment 24) like this actually got implemented at one point, but didn't actually work? Or did that not actually get committed? Is it worth building on top of that and debugging it or should I start over copying what Firefox does now and trying to port that over?

I would start with what Firefox has today. I don't think the work I had based that patch on got finished, or else has changed significantly since I did that.

Comment 38

[Wayne Mery (:wsmwk)]

FYI over the next couple weeks will get significantly more interest from users for this because automatic updates will be resuming, hopefully in large numbers.

Comment 39

[Wayne Mery (:wsmwk)]

FYI over the next couple weeks there will be significantly more interest from users because automatic updates will be resuming, hopefully in large numbers.

Comment 41

[u617804]

Hi I just updated TB in Debian Buster to 78.3.1 (that's the version that's now in Debian Buster's package repository). That installed also
the package thunderbird-l10n-de 1:78.3.1-2~deb10u2
for German language support. But TB is in English, and German language is not selectable under preferences menu.

Then I went to TB Addons, there was written the installed German language pack would not be compatible with TB 78. Then I clicked "manage"->add from file and selected /usr/share/thunderbird/extensions/[e-mail] from the Debian package.
Now I have German language in TB 78, it says updated last 2020-10-07 so today. And the new OpenPGP strings are translated, so it is in the Debian package, but was not updated automatically.

There is no need to download the xpi file in this case, it is in the Debian package correct, but the language is not updated automatically what I would have assumed.

Comment 42

[Dave Miller [:justdave]]

We indeed don't have them on the addons server yet (soon!) but if they're pushing it out as a package anyway then the failure to use it sort of sounds like a package bug? If it's something in TB it might be worth filing a separate bug for that, but I'd have the Debian maintainers make sure it's not a bug in the packaging first.

Comment 43

[Andrei Hajdukewycz [:sancus]]

The language packs that exist on ATN have been updated to their 78 versions respectively. However, the taskcluster automation is not yet finished, so I'm changing this bug to refer to that since it has some relevant history etc.

There is also some cleanup of Language Pack listings necessary. Since that is a website issue, it will be tracked here: https://github.com/thundernest/addons-server/issues/152

Comment 44

[Dave Miller [:justdave]]

(In reply to Andrei Hajdukewycz [:sancus] from comment #43)

The language packs that exist on ATN have been updated to their 78 versions respectively.

That turned out to be that only the ones that already had listings on ATN had gotten updated. I just went through and created listings for all the ones Mozilla builds that weren't there and uploaded them, so they should all be there for real now.

Comment 45

[Magnus Melin [:mkmelin] (away, back Aug 4)]

Did the automation get finished?

Comment 46

[Rob Lemley [:rjl]]

Attachment: WIP: Bug 1490598 - Taskgraph config for langpack upload to ATN.

Comment 47

[Rob Lemley [:rjl]]

Attachment: WIP: Bug 1490598 - Add comm-*-addon client configurations.

Comment 48

[Andrei Hajdukewycz [:sancus]]

The script currently used for this is here: https://gist.github.com/Sancus/58c7a4c5174eab3b513f7a9c876a17b5

It contains the code that handles the annoying parts of interacting with the ATN upload API(the token authorization, basically).

Comment 49

[Rob Lemley [:rjl]]

Attachment: Bug 1490598 - ATN Uploader docker image. r=#thunderbird-build-system-reviewers

Comment 50

[Rob Lemley [:rjl]]

Attachment: Bug 1490598 - Upload langpacks to ATN on release. r=#thunderbird-build-system-reviewers

Depends on D180494

Comment 51

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/81a318158885
ATN Uploader docker image. r=dandarnell
https://hg.mozilla.org/comm-central/rev/192ec4650383
Upload langpacks to ATN on release. r=dandarnell

Comment 52

[Rob Lemley [:rjl]]

Comment on attachment 9338372 [details]
Bug 1490598 - Upload langpacks to ATN on release. r=#thunderbird-build-system-reviewers

[Approval Request Comment]
Upload langpacks to ATN automagically on release. Both patches.

Comment 53

[Daniel Darnell [:dandarnell]]

Thunderbird 115.0b5:
https://hg.mozilla.org/releases/comm-beta/rev/2a13fab4e11a

Comment 54

[Daniel Darnell [:dandarnell]]

https://hg.mozilla.org/releases/comm-beta/rev/7b74680c9b00

Comment 55

[Wayne Mery (:wsmwk)]

Comment on attachment 9338371 [details]
Bug 1490598 - ATN Uploader docker image. r=#thunderbird-build-system-reviewers

[Triage Comment]
Approved for beta - assuming you also want this

Comment 56

[Wayne Mery (:wsmwk)]

Comment on attachment 9338372 [details]
Bug 1490598 - Upload langpacks to ATN on release. r=#thunderbird-build-system-reviewers

[Triage Comment]
Approved for beta