Update and maintain AMO's Dictionary & Language Pack page with Mozilla's 1.9.0 FTP site

RESOLVED FIXED in Q4 2012

Status

addons.mozilla.org Graveyard
Dictionaries
P5
enhancement
RESOLVED FIXED
8 years ago
a year ago

People

(Reporter: Seth Bindernagel, Assigned: kmag)

Tracking

(Depends on: 1 bug, Blocks: 1 bug)

unspecified
Q4 2012
Dependency tree / graph

Details

(Whiteboard: [ReviewTeam])

(Reporter)

Description

8 years ago
Is there a way to get the language packs created by RelEng for our current release of Firefox 3 that are listed on this ftp site 

http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla1.9.0-l10n/

in sync with the AMO site that currently lists the language packs available as add-ons

https://addons.mozilla.org/en-US/firefox/browse/type:3

I would like all the "official version" language packs created by the RelEng team to be shown on this AMO page.  

The AMO page typically lists only the language packs for locales that are not being built for official release by our RelEng team.  However, users who want to change their UI (using Locale Switcher add-on) might actually like to switch between some of our more prominent, official localizations and not just the add-ons presently listed on AMO.

Keep in mind that the AMO site would have to sync with only the official release and not any forthcoming Firefox release under development.  When a new Firefox release takes place, the 1.9.0 ftp site with language packs that would be syncing with AMO would have to change to 1.9.1.  Make sense?

In chatting with some members of RelEng, it seems that AMO could do this possibly with a scraping script, and in the future could upload langpacks as part of the nightly/release process.  

We might have to make sure that the AMO page distinguished between the "official version lang packs" and the "other lang packs" (presently listed on the existing site).  As a reference point, we sort of do this on the all.html page with the Official Versions section and the Beta Versions section.
(Reporter)

Comment 1

8 years ago
Similar bug:

https://bugzilla.mozilla.org/show_bug.cgi?id=484863
The first url in comment #1 is actually nightly builds. Did you mean the likes of 
 http://releases.mozilla.org/pub/mozilla.org/firefox/releases/3.0.8/win32/xpi/
(Reporter)

Comment 3

8 years ago
Some other interesting information, I looked at the addons that might benefit from this site enhancement:

Locale Switcher:  3,870  weekly downloads/399,249 total downloads 
Quick Locale Switcher:  9,972  weekly downloads/811,945 total downloads 

From what I can tell, Quick Locale Switcher may use the FTP site.  The figures above provide some indication about the appetite for more visible language packs.
(Reporter)

Comment 4

8 years ago
(In reply to comment #2)
>  http://releases.mozilla.org/pub/mozilla.org/firefox/releases/3.0.8/win32/xpi/

Yes.

Comment 5

8 years ago
Please note that we should the updateUrl to the XPI files, which is something I
think we should do also for the XPIs on the FTP. Maybe direct these to the AMO
langpack?

It should also support minor updates without version bump, for cases where
there is some changes required.
Dupe of bug 451824.
Not really a dupe, but bug 451824 depends on this one.

Note that we do have GUIDs for the language packs, but we don't have good revisions.
Blocks: 451824
So it sounds like you want the following:

Show both official and unofficial language packs on this page.  Official packs should have some sort of badge denoting their 'official-ness', and are synced every 24 hours with the ftp site.  Also, we should feature both locale switcher add-ons at the top like we do with categories of firefox extensions.

Does that sound right?
(Reporter)

Comment 9

8 years ago
(In reply to comment #8)
> Show both official and unofficial language packs on this page.  Official packs
> should have some sort of badge denoting their 'official-ness', and are synced
> every 24 hours with the ftp site.  

Yes, I think something like that.

> Also, we should feature both locale switcher
> add-ons at the top like we do with categories of firefox extensions.

I think this would be helpful.

Some other questions that have come up:
1)  Who maintains the "official" lang packs?  Would there need to be some automation that makes sure the version min and max is current and other stuff like that?
2)  Would these "official" language packs auto pass through the sandbox?
3)  I heard something mentioned about a writable API that might allow automatic upload of language packs?

Not sure if there are other questions or issues.  Some had mentioned XML parsing errors if there were string changes in minor updates (from say, 3.0.7 to 3.0.8).  But, I think the automatic update of language packs and the compatibility check prompting users to install the new version would answer that issue.  No?
We currently have no restrictions in compatibility for language packs on a stable branch, so those themselves don't answer the question. We need an explicit policy on what to do here.

Same goes for the version numbers of the language packs themselves. Currently, we use the gecko platform version, which means that, changed or not, the language pack gets a new version number on each minor update. That'd be annoying to users, I guess.

I'm not sure if we can come up with a good versioning scheme. On hg, using the numeric id of the revision might work, but we need to make sure that that's consistent at least across plain clones that don't do local edits. On CVS, I have no idea, which tempts me to not fix this bug for the builds we do from CVS.

Maintance of the language packs should probably be with the owners of the localization, with the release automation "peering" with them in uploading and updating the compat info. I guess, nothing fleshed out.
(Reporter)

Comment 11

8 years ago
On version and maintenance of language packs, I believe we can start by making the minVersion of the auto-generated language packs the most recent version of Firefox.  This should limit any possible conflicts between version of Firefox and language pack since both would have to match.

As an intermediary starting point, we could generate a static html page that explains the official language packs and offers them for installation.  The existing AMO Dictionary & Language Pack page could direct people looking for official-version language packs to this page. The static html page also would offer users a download for the most recent version of Firefox if they didn't have it.  (As mentioned, users would need both for the lang pack to work.)  We may even choose to feature Quick Locale Switcher and/or Locale Switcher to illustrate the use of these language packs.

Then, the static html page would update with the auto-generated language packs from RelEng for each new minor release of Firefox -- probably within hours and not minutes, unless minutes is just as easy.  As users update their version of Firefox, the add-on compatibility check could prompt them to install the new add-on because the old language pack would no longer be compatible.  If the user does not install the update, then he or she cannot use the older language pack add-on.  In this scenario, the add-on manager would present the "add-on is not compatible/check for updates" dialogue.  (Apologies, I am not sure of that exact wording that is presented by the add-on manager.)

Updated

8 years ago
Duplicate of this bug: 441515
This is becoming more possible.  We're doing something similar with personas now (syncing every 24 hours) so the concept is valid and somewhat proven.  :)
Severity: normal → enhancement
Priority: -- → P5
Target Milestone: --- → Future

Comment 14

7 years ago
This is now the only blocker for https://bugzilla.mozilla.org/show_bug.cgi?id=451824. Any chance to get this through?
Closer than ever.  You'll want to watch bug 586701 - 586704 and their dependencies.  Once those are fixed the possibility is there, but we'd need someone on the other end to interact with the new API.  At the earliest, this would be next year.

Updated

6 years ago
Blocks: 632886

Comment 16

6 years ago
So, where are we regarding this? Is there anything a profile like me could help? I would love to offer langpacks for my locale from AMO.
The API exists.  We're connecting builder.amo to it first and it appears to mostly work.  The last time this came up we had a couple options:

1) A build script on the platform side that uploads the .xpi's to the AMO API every time the strings change.  I'm not super excited about the ever growing list of hundreds of versions with this solution.  We'd have to add delete versions a week or two old with the script as well.

2) Adjust the <updateurl> in the add-on itself to point to a simple script on the build side.  This would make Firefox check that for updates and bypass AMO completely.  That would be a simple solution to this bug but you'd lose out on the benefits of AMO - which in a case like this are fairly minimal I imagine.

Comment 18

6 years ago
Any news on this?  I would also love to provide my locale from AMO.

I've tried to upload the official language packs, but they have to go through the revision process (though I'm asking them to consider fast-tracking them: Bug 708289) and that takes time.
(Assignee)

Comment 19

5 years ago
Talked to Jorge about this yesterday. We're thinking about putting together a script to fetch the language packs from the release server and auto-upload/approve them. I'd like to have this done this quarter, since it should remove some burden from the review team, aside from its usefulness to users.

I'm currently planning on doing this only for the release branch, and possibly adding a beta branch language packs for betas. Since the official language packs only target one major app version and AMO currently only makes the most recent version it knows about available for updates and on listing pages, rather than the most recent compatible version, I think that's the only option for the time being.
OS: Mac OS X → All
Whiteboard: [ReviewTeam]
Target Milestone: Future → Q4 2012
Assignee: nobody → kmaglione+bmo

Comment 20

4 years ago
Would an API in Addons that could be accessed by external users work? Then we don't have the logic inside zamboni, but externally by just providing an API others can use?
(In reply to Andy McKay [:andym] from comment #20)
> Would an API in Addons that could be accessed by external users work? Then
> we don't have the logic inside zamboni, but externally by just providing an
> API others can use?

I guess there are two canonical pieces of infrastructure that can "do the work" on top of an API, one is AMO, one is releng.

It'd be great if we can get this done. Perfect has been the enemy of this bug for almost 4 years, and it may be tempting to put another year on top. That'd be sad.
(Assignee)

Comment 22

4 years ago
(In reply to Andy McKay [:andym] from comment #20)
> Would an API in Addons that could be accessed by external users work? Then
> we don't have the logic inside zamboni, but externally by just providing an
> API others can use?

This could feasibly be done externally mostly with APIs that already exist, but it would require some sort of hackery to get the add-ons approved. We'd have to either re-instate the trusted flag, or programmatically approve the add-ons from a different privileged account. I'm not particularly fond of either idea, and would prefer the process to happen with the oversight of a reviewer and as close to the AMO side as possible.

Comment 23

4 years ago
You could take the logic you've written and make it an API that is protected by a particular token, same as we do with builder. That would mean the amount of code in zamboni is minimized.
(Assignee)

Comment 24

4 years ago
I'm not especially averse to the process being initiated from the releng side, but I doubt that it would reduce the amount of code needed in Zamboni to support it. We'd also still need the admin panel either way, even if not the update form.

In any case, there are 57 of these language packs on AMO at the moment, which are either hopelessly outdated or are consuming review team resources after every release, so we need a working solution sooner rather than later.

Comment 25

4 years ago
> Talked to Jorge about this yesterday. We're thinking about putting together
> a script to fetch the language packs from the release server and
> auto-upload/approve them. I'd like to have this done this quarter, since it
> should remove some burden from the review team, aside from its usefulness to
> users.

Great!  As Axel said, please don't try to do it perfectly, a working solution right now would help a lot.  It is currently a burden for localizers too, among other things it's hard for us to have the right timing for new releases because we never know how long the review process is going to take.  If you implement that, then the language packs could go live together with the new Firefox version without additional effort from us or the review team.
(Assignee)

Comment 26

4 years ago
The server portion of this is live. We're currently contacting the maintainers of the listings which use the IDs of the official language packs for permission to take over maintainership. Hopefully we can run the first import next week.
(Assignee)

Updated

4 years ago
Duplicate of this bug: 708289
(Assignee)

Comment 28

4 years ago
https://addons.mozilla.org/language-tools/
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
(Assignee)

Comment 29

4 years ago
https://github.com/mozilla/zamboni/pull/533
(Assignee)

Updated

4 years ago
Depends on: 832060
This bug is almost 4 years old.  Thanks Kris!

Comment 31

4 years ago
When will the language packs be available?  Right now https://addons.mozilla.org/en-US/firefox/addon/esperanto-language-pack/ only offers version 15 ...
(Assignee)

Comment 32

4 years ago
(In reply to Eduardo Trápani from comment #31)
> When will the language packs be available?  Right now
> https://addons.mozilla.org/en-US/firefox/addon/esperanto-language-pack/ only
> offers version 15 ...

Only langpacks with Mozilla as an author will be auto-updated. As far as I know, we never received permission to add ourselves as an author for that one.

Comment 33

4 years ago
Kris, Jorge Villalobos (Add-ons Developer Relations Lead) sent a message on 2013.01.01 with the subject "[Action required] Introducing automatic updates for language packs on AMO".  The message said "In order for this feature to work, we need to add our account as an owner for your add-on.".  We agreed and the answer was "We'll take the listing, then. You don't need to do anything else.".

Is there anything else we should have done?  What can we do to let you add yourselves as the author of our language pack?
Eduardo, it looks like I dropped the ball on this one and didn't include you on the list of people who had accepted to automatically update their add-on. We will add out account to your listing shortly, which should be enough for it to be automatically updated for Firefox 19.
(In reply to Jorge Villalobos [:jorgev] from comment #34)
> Eduardo, it looks like I dropped the ball on this one and didn't include you
> on the list of people who had accepted to automatically update their add-on.
> We will add out account to your listing shortly, which should be enough for
> it to be automatically updated for Firefox 19.

... "We will add *our* account" ...

Comment 36

4 years ago
Hi,

for sake of tracking, it looks like we are missing 'ca' as well for 19.0
(Assignee)

Comment 37

4 years ago
(In reply to Toni Hermoso Pulido from comment #36)
> for sake of tracking, it looks like we are missing 'ca' as well for 19.0

Yes, see bug 832060. We still have to repair and update those langpacks manually until their builds are fixed. :(
What will happen to language packs for Thunderbird & SeaMonkey?
https://addons.mozilla.org/en-US/seamonkey/language-tools/ and https://addons.mozilla.org/en-US/thunderbird/language-tools/ don't list the official language packs.
Duplicate of this bug: 451824
(Assignee)

Comment 40

4 years ago
(In reply to Archaeopteryx [:aryx] from comment #38)
> What will happen to language packs for Thunderbird & SeaMonkey?
> https://addons.mozilla.org/en-US/seamonkey/language-tools/ and
> https://addons.mozilla.org/en-US/thunderbird/language-tools/ don't list the
> official language packs.

We technically have the ability to auto-update those too, but it requires contacting the developers of the owners of already-listed language packs for permission to add Mozilla as an author.
(Assignee)

Comment 41

4 years ago
And when I say we, I can get you the list of add-ons if you want to do it. I don't think it's very likely to happen otherwise.

Comment 42

4 years ago
Still pending langpacks, even when no problem, as commented in bug 832060
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Please file a new bug on that.
Status: REOPENED → RESOLVED
Last Resolved: 4 years ago4 years ago
Resolution: --- → FIXED
(Assignee)

Comment 44

4 years ago
No need to file a new bug. Bug 832060 covers this.

Comment 45

4 years ago
Ey guys, don't know if I should reopen or fill another bug, but langpacks are not being built: 
Ex.:

https://addons.mozilla.org/en-US/firefox/addon/fran%C3%A7ais-language-pack/

and all the rest!

Comment 46

4 years ago
I confirm for Ukrainian.
(Assignee)

Comment 47

4 years ago
Thanks for reporting. We're aware of the problem. The updater has been failing since the AMO servers were migrated and we're looking into fixing it.
(Assignee)

Comment 48

4 years ago
The language packs have been updated with today's push. Please feel free to file a new bug if the update fails again.

Updated

3 years ago
Depends on: 969327

Comment 49

3 years ago
(In reply to Kris Maglione [:kmag] from comment #40)
> (In reply to Archaeopteryx [:aryx] from comment #38)
> > What will happen to language packs for Thunderbird & SeaMonkey?
> > https://addons.mozilla.org/en-US/seamonkey/language-tools/ and
> > https://addons.mozilla.org/en-US/thunderbird/language-tools/ don't list the
> > official language packs.
> 
> We technically have the ability to auto-update those too, but it requires
> contacting the developers of the owners of already-listed language packs for
> permission to add Mozilla as an author.

About that, I filled bug 1072515 for Thunderbird. Seamonkey will need a separated bug.

Updated

2 years ago
Blocks: 1083689
Product: addons.mozilla.org → addons.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.