Last Comment Bug 485617 - Update and maintain AMO's Dictionary & Language Pack page with Mozilla's 1.9.0 FTP site
: Update and maintain AMO's Dictionary & Language Pack page with Mozilla's 1.9....
Status: RESOLVED FIXED
[ReviewTeam]
:
Product: addons.mozilla.org Graveyard
Classification: Graveyard
Component: Dictionaries (show other bugs)
: unspecified
: All All
: P5 enhancement
: Q4 2012
Assigned To: Kris Maglione [:kmag]
:
Mentors:
: 441515 451824 708289 (view as bug list)
Depends on: 832060 969327
Blocks: 1083689 451824 632886
  Show dependency treegraph
 
Reported: 2009-03-27 15:09 PDT by Seth Bindernagel
Modified: 2016-02-04 14:49 PST (History)
30 users (show)
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments

Description Seth Bindernagel 2009-03-27 15:09:34 PDT
Is there a way to get the language packs created by RelEng for our current release of Firefox 3 that are listed on this ftp site 

http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla1.9.0-l10n/

in sync with the AMO site that currently lists the language packs available as add-ons

https://addons.mozilla.org/en-US/firefox/browse/type:3

I would like all the "official version" language packs created by the RelEng team to be shown on this AMO page.  

The AMO page typically lists only the language packs for locales that are not being built for official release by our RelEng team.  However, users who want to change their UI (using Locale Switcher add-on) might actually like to switch between some of our more prominent, official localizations and not just the add-ons presently listed on AMO.

Keep in mind that the AMO site would have to sync with only the official release and not any forthcoming Firefox release under development.  When a new Firefox release takes place, the 1.9.0 ftp site with language packs that would be syncing with AMO would have to change to 1.9.1.  Make sense?

In chatting with some members of RelEng, it seems that AMO could do this possibly with a scraping script, and in the future could upload langpacks as part of the nightly/release process.  

We might have to make sure that the AMO page distinguished between the "official version lang packs" and the "other lang packs" (presently listed on the existing site).  As a reference point, we sort of do this on the all.html page with the Official Versions section and the Beta Versions section.
Comment 1 Seth Bindernagel 2009-03-27 15:30:23 PDT
Similar bug:

https://bugzilla.mozilla.org/show_bug.cgi?id=484863
Comment 2 Nick Thomas [:nthomas] 2009-03-27 17:14:46 PDT
The first url in comment #1 is actually nightly builds. Did you mean the likes of 
 http://releases.mozilla.org/pub/mozilla.org/firefox/releases/3.0.8/win32/xpi/
Comment 3 Seth Bindernagel 2009-03-27 17:26:06 PDT
Some other interesting information, I looked at the addons that might benefit from this site enhancement:

Locale Switcher:  3,870  weekly downloads/399,249 total downloads 
Quick Locale Switcher:  9,972  weekly downloads/811,945 total downloads 

From what I can tell, Quick Locale Switcher may use the FTP site.  The figures above provide some indication about the appetite for more visible language packs.
Comment 4 Seth Bindernagel 2009-03-27 17:26:39 PDT
(In reply to comment #2)
>  http://releases.mozilla.org/pub/mozilla.org/firefox/releases/3.0.8/win32/xpi/

Yes.
Comment 5 Tomer Cohen :tomer 2009-03-27 17:49:26 PDT
Please note that we should the updateUrl to the XPI files, which is something I
think we should do also for the XPIs on the FTP. Maybe direct these to the AMO
langpack?

It should also support minor updates without version bump, for cases where
there is some changes required.
Comment 6 Sebastian Hengst [:aryx][:archaeopteryx] 2009-03-29 03:00:48 PDT
Dupe of bug 451824.
Comment 7 Axel Hecht [pto-Aug-30][:Pike] 2009-03-29 03:52:51 PDT
Not really a dupe, but bug 451824 depends on this one.

Note that we do have GUIDs for the language packs, but we don't have good revisions.
Comment 8 Nick Nguyen [:osunick] 2009-04-02 18:08:50 PDT
So it sounds like you want the following:

Show both official and unofficial language packs on this page.  Official packs should have some sort of badge denoting their 'official-ness', and are synced every 24 hours with the ftp site.  Also, we should feature both locale switcher add-ons at the top like we do with categories of firefox extensions.

Does that sound right?
Comment 9 Seth Bindernagel 2009-04-02 18:24:58 PDT
(In reply to comment #8)
> Show both official and unofficial language packs on this page.  Official packs
> should have some sort of badge denoting their 'official-ness', and are synced
> every 24 hours with the ftp site.  

Yes, I think something like that.

> Also, we should feature both locale switcher
> add-ons at the top like we do with categories of firefox extensions.

I think this would be helpful.

Some other questions that have come up:
1)  Who maintains the "official" lang packs?  Would there need to be some automation that makes sure the version min and max is current and other stuff like that?
2)  Would these "official" language packs auto pass through the sandbox?
3)  I heard something mentioned about a writable API that might allow automatic upload of language packs?

Not sure if there are other questions or issues.  Some had mentioned XML parsing errors if there were string changes in minor updates (from say, 3.0.7 to 3.0.8).  But, I think the automatic update of language packs and the compatibility check prompting users to install the new version would answer that issue.  No?
Comment 10 Axel Hecht [pto-Aug-30][:Pike] 2009-04-03 06:42:59 PDT
We currently have no restrictions in compatibility for language packs on a stable branch, so those themselves don't answer the question. We need an explicit policy on what to do here.

Same goes for the version numbers of the language packs themselves. Currently, we use the gecko platform version, which means that, changed or not, the language pack gets a new version number on each minor update. That'd be annoying to users, I guess.

I'm not sure if we can come up with a good versioning scheme. On hg, using the numeric id of the revision might work, but we need to make sure that that's consistent at least across plain clones that don't do local edits. On CVS, I have no idea, which tempts me to not fix this bug for the builds we do from CVS.

Maintance of the language packs should probably be with the owners of the localization, with the release automation "peering" with them in uploading and updating the compat info. I guess, nothing fleshed out.
Comment 11 Seth Bindernagel 2009-04-09 17:01:49 PDT
On version and maintenance of language packs, I believe we can start by making the minVersion of the auto-generated language packs the most recent version of Firefox.  This should limit any possible conflicts between version of Firefox and language pack since both would have to match.

As an intermediary starting point, we could generate a static html page that explains the official language packs and offers them for installation.  The existing AMO Dictionary & Language Pack page could direct people looking for official-version language packs to this page. The static html page also would offer users a download for the most recent version of Firefox if they didn't have it.  (As mentioned, users would need both for the lang pack to work.)  We may even choose to feature Quick Locale Switcher and/or Locale Switcher to illustrate the use of these language packs.

Then, the static html page would update with the auto-generated language packs from RelEng for each new minor release of Firefox -- probably within hours and not minutes, unless minutes is just as easy.  As users update their version of Firefox, the add-on compatibility check could prompt them to install the new add-on because the old language pack would no longer be compatible.  If the user does not install the update, then he or she cannot use the older language pack add-on.  In this scenario, the add-on manager would present the "add-on is not compatible/check for updates" dialogue.  (Apologies, I am not sure of that exact wording that is presented by the add-on manager.)
Comment 12 Fred Wenzel [:wenzel] 2009-06-10 03:37:28 PDT
*** Bug 441515 has been marked as a duplicate of this bug. ***
Comment 13 Wil Clouser [:clouserw] 2009-12-01 13:25:07 PST
This is becoming more possible.  We're doing something similar with personas now (syncing every 24 hours) so the concept is valid and somewhat proven.  :)
Comment 14 Jan Killian 2010-09-22 23:12:12 PDT
This is now the only blocker for https://bugzilla.mozilla.org/show_bug.cgi?id=451824. Any chance to get this through?
Comment 15 Wil Clouser [:clouserw] 2010-09-22 23:21:40 PDT
Closer than ever.  You'll want to watch bug 586701 - 586704 and their dependencies.  Once those are fixed the possibility is there, but we'd need someone on the other end to interact with the new API.  At the earliest, this would be next year.
Comment 16 Toni Hermoso Pulido 2011-09-10 05:27:21 PDT
So, where are we regarding this? Is there anything a profile like me could help? I would love to offer langpacks for my locale from AMO.
Comment 17 Wil Clouser [:clouserw] 2011-09-19 01:00:50 PDT
The API exists.  We're connecting builder.amo to it first and it appears to mostly work.  The last time this came up we had a couple options:

1) A build script on the platform side that uploads the .xpi's to the AMO API every time the strings change.  I'm not super excited about the ever growing list of hundreds of versions with this solution.  We'd have to add delete versions a week or two old with the script as well.

2) Adjust the <updateurl> in the add-on itself to point to a simple script on the build side.  This would make Firefox check that for updates and bypass AMO completely.  That would be a simple solution to this bug but you'd lose out on the benefits of AMO - which in a case like this are fairly minimal I imagine.
Comment 18 Eduardo Trápani 2011-12-07 09:44:55 PST
Any news on this?  I would also love to provide my locale from AMO.

I've tried to upload the official language packs, but they have to go through the revision process (though I'm asking them to consider fast-tracking them: Bug 708289) and that takes time.
Comment 19 Kris Maglione [:kmag] 2012-11-21 14:59:18 PST
Talked to Jorge about this yesterday. We're thinking about putting together a script to fetch the language packs from the release server and auto-upload/approve them. I'd like to have this done this quarter, since it should remove some burden from the review team, aside from its usefulness to users.

I'm currently planning on doing this only for the release branch, and possibly adding a beta branch language packs for betas. Since the official language packs only target one major app version and AMO currently only makes the most recent version it knows about available for updates and on listing pages, rather than the most recent compatible version, I think that's the only option for the time being.
Comment 20 Andy McKay [:andym] 2012-12-19 15:41:56 PST
Would an API in Addons that could be accessed by external users work? Then we don't have the logic inside zamboni, but externally by just providing an API others can use?
Comment 21 Axel Hecht [pto-Aug-30][:Pike] 2012-12-19 15:49:31 PST
(In reply to Andy McKay [:andym] from comment #20)
> Would an API in Addons that could be accessed by external users work? Then
> we don't have the logic inside zamboni, but externally by just providing an
> API others can use?

I guess there are two canonical pieces of infrastructure that can "do the work" on top of an API, one is AMO, one is releng.

It'd be great if we can get this done. Perfect has been the enemy of this bug for almost 4 years, and it may be tempting to put another year on top. That'd be sad.
Comment 22 Kris Maglione [:kmag] 2012-12-19 16:16:41 PST
(In reply to Andy McKay [:andym] from comment #20)
> Would an API in Addons that could be accessed by external users work? Then
> we don't have the logic inside zamboni, but externally by just providing an
> API others can use?

This could feasibly be done externally mostly with APIs that already exist, but it would require some sort of hackery to get the add-ons approved. We'd have to either re-instate the trusted flag, or programmatically approve the add-ons from a different privileged account. I'm not particularly fond of either idea, and would prefer the process to happen with the oversight of a reviewer and as close to the AMO side as possible.
Comment 23 Andy McKay [:andym] 2012-12-19 16:18:32 PST
You could take the logic you've written and make it an API that is protected by a particular token, same as we do with builder. That would mean the amount of code in zamboni is minimized.
Comment 24 Kris Maglione [:kmag] 2012-12-19 16:33:31 PST
I'm not especially averse to the process being initiated from the releng side, but I doubt that it would reduce the amount of code needed in Zamboni to support it. We'd also still need the admin panel either way, even if not the update form.

In any case, there are 57 of these language packs on AMO at the moment, which are either hopelessly outdated or are consuming review team resources after every release, so we need a working solution sooner rather than later.
Comment 25 Eduardo Trápani 2012-12-20 05:10:15 PST
> Talked to Jorge about this yesterday. We're thinking about putting together
> a script to fetch the language packs from the release server and
> auto-upload/approve them. I'd like to have this done this quarter, since it
> should remove some burden from the review team, aside from its usefulness to
> users.

Great!  As Axel said, please don't try to do it perfectly, a working solution right now would help a lot.  It is currently a burden for localizers too, among other things it's hard for us to have the right timing for new releases because we never know how long the review process is going to take.  If you implement that, then the language packs could go live together with the new Firefox version without additional effort from us or the review team.
Comment 26 Kris Maglione [:kmag] 2013-01-10 19:43:31 PST
The server portion of this is live. We're currently contacting the maintainers of the listings which use the IDs of the official language packs for permission to take over maintainership. Hopefully we can run the first import next week.
Comment 27 Kris Maglione [:kmag] 2013-01-17 15:07:25 PST
*** Bug 708289 has been marked as a duplicate of this bug. ***
Comment 28 Kris Maglione [:kmag] 2013-01-17 15:08:09 PST
https://addons.mozilla.org/language-tools/
Comment 29 Kris Maglione [:kmag] 2013-01-17 15:20:56 PST
https://github.com/mozilla/zamboni/pull/533
Comment 30 Wil Clouser [:clouserw] 2013-01-17 15:27:36 PST
This bug is almost 4 years old.  Thanks Kris!
Comment 31 Eduardo Trápani 2013-02-13 10:03:37 PST
When will the language packs be available?  Right now https://addons.mozilla.org/en-US/firefox/addon/esperanto-language-pack/ only offers version 15 ...
Comment 32 Kris Maglione [:kmag] 2013-02-14 10:53:21 PST
(In reply to Eduardo Trápani from comment #31)
> When will the language packs be available?  Right now
> https://addons.mozilla.org/en-US/firefox/addon/esperanto-language-pack/ only
> offers version 15 ...

Only langpacks with Mozilla as an author will be auto-updated. As far as I know, we never received permission to add ourselves as an author for that one.
Comment 33 Eduardo Trápani 2013-02-14 11:56:12 PST
Kris, Jorge Villalobos (Add-ons Developer Relations Lead) sent a message on 2013.01.01 with the subject "[Action required] Introducing automatic updates for language packs on AMO".  The message said "In order for this feature to work, we need to add our account as an owner for your add-on.".  We agreed and the answer was "We'll take the listing, then. You don't need to do anything else.".

Is there anything else we should have done?  What can we do to let you add yourselves as the author of our language pack?
Comment 34 Jorge Villalobos [:jorgev] 2013-02-14 12:14:59 PST
Eduardo, it looks like I dropped the ball on this one and didn't include you on the list of people who had accepted to automatically update their add-on. We will add out account to your listing shortly, which should be enough for it to be automatically updated for Firefox 19.
Comment 35 Jorge Villalobos [:jorgev] 2013-02-14 12:15:32 PST
(In reply to Jorge Villalobos [:jorgev] from comment #34)
> Eduardo, it looks like I dropped the ball on this one and didn't include you
> on the list of people who had accepted to automatically update their add-on.
> We will add out account to your listing shortly, which should be enough for
> it to be automatically updated for Firefox 19.

... "We will add *our* account" ...
Comment 36 Toni Hermoso Pulido 2013-02-19 13:34:57 PST
Hi,

for sake of tracking, it looks like we are missing 'ca' as well for 19.0
Comment 37 Kris Maglione [:kmag] 2013-02-19 18:09:24 PST
(In reply to Toni Hermoso Pulido from comment #36)
> for sake of tracking, it looks like we are missing 'ca' as well for 19.0

Yes, see bug 832060. We still have to repair and update those langpacks manually until their builds are fixed. :(
Comment 38 Sebastian Hengst [:aryx][:archaeopteryx] 2013-02-22 10:58:52 PST
What will happen to language packs for Thunderbird & SeaMonkey?
https://addons.mozilla.org/en-US/seamonkey/language-tools/ and https://addons.mozilla.org/en-US/thunderbird/language-tools/ don't list the official language packs.
Comment 39 Sebastian Hengst [:aryx][:archaeopteryx] 2013-02-22 11:00:17 PST
*** Bug 451824 has been marked as a duplicate of this bug. ***
Comment 40 Kris Maglione [:kmag] 2013-02-22 11:04:54 PST
(In reply to Archaeopteryx [:aryx] from comment #38)
> What will happen to language packs for Thunderbird & SeaMonkey?
> https://addons.mozilla.org/en-US/seamonkey/language-tools/ and
> https://addons.mozilla.org/en-US/thunderbird/language-tools/ don't list the
> official language packs.

We technically have the ability to auto-update those too, but it requires contacting the developers of the owners of already-listed language packs for permission to add Mozilla as an author.
Comment 41 Kris Maglione [:kmag] 2013-02-22 15:33:24 PST
And when I say we, I can get you the list of add-ons if you want to do it. I don't think it's very likely to happen otherwise.
Comment 42 Toni Hermoso Pulido 2013-04-03 00:25:27 PDT
Still pending langpacks, even when no problem, as commented in bug 832060
Comment 43 Axel Hecht [pto-Aug-30][:Pike] 2013-04-03 03:49:31 PDT
Please file a new bug on that.
Comment 44 Kris Maglione [:kmag] 2013-04-03 10:14:28 PDT
No need to file a new bug. Bug 832060 covers this.
Comment 45 Toni Hermoso Pulido 2013-08-14 05:14:57 PDT
Ey guys, don't know if I should reopen or fill another bug, but langpacks are not being built: 
Ex.:

https://addons.mozilla.org/en-US/firefox/addon/fran%C3%A7ais-language-pack/

and all the rest!
Comment 46 Tim Babych 2013-08-14 05:19:03 PDT
I confirm for Ukrainian.
Comment 47 Kris Maglione [:kmag] 2013-08-14 11:16:38 PDT
Thanks for reporting. We're aware of the problem. The updater has been failing since the AMO servers were migrated and we're looking into fixing it.
Comment 48 Kris Maglione [:kmag] 2013-08-20 13:10:53 PDT
The language packs have been updated with today's push. Please feel free to file a new bug if the update fails again.
Comment 49 Carlos [nohamelin] 2014-09-26 14:20:28 PDT
(In reply to Kris Maglione [:kmag] from comment #40)
> (In reply to Archaeopteryx [:aryx] from comment #38)
> > What will happen to language packs for Thunderbird & SeaMonkey?
> > https://addons.mozilla.org/en-US/seamonkey/language-tools/ and
> > https://addons.mozilla.org/en-US/thunderbird/language-tools/ don't list the
> > official language packs.
> 
> We technically have the ability to auto-update those too, but it requires
> contacting the developers of the owners of already-listed language packs for
> permission to add Mozilla as an author.

About that, I filled bug 1072515 for Thunderbird. Seamonkey will need a separated bug.

Note You need to log in before you can comment on or make changes to this bug.