346278 - Allow building localized lightning langpacks

Status: RESOLVED FIXED

(Calendar :: Build Config, defect, P1)

Description

[Thorsten 'nightrat' Fritz]

Building of localized lightning extension should be available.

Bug 344602 is fixed.

Comment 1

[Matthew (lilmatt) Willis]

CCing rob_strong so he can tell us the Right Way to localize extensions so they work nice with the Add-ons Manager.

Comment 2

[Robert Strong (they/them - no direct email)]

The choices are to either build the extension with all available locales or build the extension with one locale and build language packs for the available languages. The language packs could then have the extension as a dependency.

Bug 285848 will provide automatic discovery of language packs for an extension.
Bug 345610 shows additional work planned for extension dependencies.

Comment 3

[Matthew (lilmatt) Willis]

Taking this.

Comment 4

[Matthew (lilmatt) Willis]

Waiting for all localization to complete before shipping Lightning sounds like a trade-off we don't want to make.  Given our discussion in the calendar weekly meeting, I'll work on the lightning+langpack scenario.

Comment 5

[Matthew (lilmatt) Willis]

Attachment: rev0 - Allows building of langpacks for lightning similar to sunbirds

make -C calendar/locales langpack-ab-CD
should make a langpack xpi for lightning in your dist/install directory.

Comment 6

[Stefan Sitter [:ssitter]]

Comment on attachment 237689 [details] [diff] [review]
rev0 - Allows building of langpacks for lightning similar to sunbirds

I only did a quick test until now and didn't look at makefile changes in detail.

1. Building Sunbird, going to SbObjDir/calendar/locales and run make langpack-ab-CD produces lightning langpack instead of sunbird langpack. This should be fixed.

2. Building Thunderbird, going to LtnObjDir/calendar/locales and run make langpack-ab-CD produces lightning langpacks. But there is a problem with the lightning files. en-US/lightning files were packaged in all langpacks instead of correct one.

 +++ making chrome /cygdrive/y/obj/ltn-test/calendar/locales  
     => ../../dist/xpi-stage/locale-de/chrome/lightning-de.jar
 +++ updating chrome ../../dist/xpi-stage/locale-de/chrome/../chrome.manifest
   zip warning: ../../../locale-de/chrome/lightning-de.jar not found or empty
   adding: locale/en-US/lightning/lightning.dtd (stored 0%)
   adding: locale/en-US/lightning/lightning.properties (stored 0%)

3. I did not get the langpack to work. Either there were missing entities or ltnGetString() failed or some other errors. Maybe related to point 2.

4. But I saw that error messages in console were translated. The langpack contains toolkit locals too (in 'ab-CD.jar'). I think this causes problems with toolkit locals from host application. Our normal lightning.xpi ships 'calendar-en-US.jar' and 'lightning-en-US.jar' only.

5. Package naming: I think the package name should use the Lightning version in name not the Thunderbird version. This makes it more clear where it belongs to. For example lightning-3.0a1.ab-CD.langpack.xpi should be lightning-0.1+.ab-CD.langpack.xpi instead.

6. install.rdf: em:version="3.0a1" I think this should be the Lightning version too.

7. install.rdf: <em:minVersion>3.0a1 / <em:maxVersion>3.0a1 I think we want the supported Thunderbird version here instead of the one build in. For example when building from branch this should be 1.5.0.* to 2.0.* instead of 2.0a1 to 2.0a1.

8. Makefile changes seems to assume that not-Sunbird means Thunderbird in every case. I think it should be prepared in a way so that it is easy to extend e.g. SeaMonkey and Firefox later.

Comment 7

[Matthew (lilmatt) Willis]

Comment on attachment 237689 [details] [diff] [review]
rev0 - Allows building of langpacks for lightning similar to sunbirds

Removing r2 request.

Ya. This needs a bit more work. Not going to make 0.3 using this approach.

Comment 8

[Matthew (lilmatt) Willis]

Clarified summary. This is about _langpacks_, not just having a localized lightning (which for 0.3, we'll probably do with a big ol' extension)

Moved to Ln 0.5

Comment 9

[Matthew (lilmatt) Willis]

This is not going to make 0.5.

Comment 10

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Milestone 0.7 Consolidation. Filter on PinkyANDBrain to get rid of bugspam.

Comment 11

[Stefan Sitter [:ssitter]]

We now can build Lightning with all locales included (Bug 352546). I think this bug is therefore no longer required.

Comment 12

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

(In reply to comment #6)
> 1. Building Sunbird, going to SbObjDir/calendar/locales and run make
> langpack-ab-CD produces lightning langpack instead of sunbird langpack. This
> should be fixed.
Is this still the case? If so, we should file/find a separate bug.

> 2. Building Thunderbird, going to LtnObjDir/calendar/locales and run make
> langpack-ab-CD produces lightning langpacks. But there is a problem with the
> lightning files. en-US/lightning files were packaged in all langpacks instead
> of correct one.
Just out of curiosity, what happens? Although we don't really need lightning langpacks, I think we might want to at least leave the possibility, in case a new locale wants to test changes without building a full lightning? I don't really mind if we don't have a ltn langpack though, but maybe we should spit out an error that building langpacks is not supported.

> 8. Makefile changes seems to assume that not-Sunbird means Thunderbird in every
> case. I think it should be prepared in a way so that it is easy to extend e.g.
> SeaMonkey and Firefox later.
How much of this is true?

Comment 13

[Simon Paquet [:sipaq]]

Based on our discussions at FOSDEM, this should definitely block 1.0 because we need daily localized Lightning builds to enable localization testing for localizers once we no longer have daily localized Sunbird builds.

Comment 14

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

So, lets get some traction on this bug. We have multiple possibilities:

A) Build lightning in one language (i.e each localized language)
B) Build real langpacks

What are the advantages?
Which is easier to accomplish?
What does a langpack look like?
  - i.e: is it just an extension with only a locale definition?

Comment 15

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Gozer, any ideas on this from a build perspective?

Comment 16

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Attachment: [checked in] rev1 - Allow repackaging lightning into separate locales

This patch should be a nice start. The process is as follows:

$ make AB_CD=de repack-l10n

* Removes any existing xpi-stage/lightning-de
* Copies xpi-stage/lightning to xpi-stage/lightning-de
* Removes xpi-stage/lightning-de/chrome/(calendar|lightning)-en-US.jar
* Makes all calendar locales for de, putting them in xpi-stage/lightning-de
* Recreates install.rdf with $(L10NBASEDIR)/de/defines.inc included
* Processes $(L10NBASEDIR)/de/lightning-l10n.js into xpi-stage/lightning-de/$(PREF_DIR)/lightning-l10n.js

I hope I didn't miss anything.

Comment 17

[Axel Hecht [:Pike]]

(In reply to comment #14)
> So, lets get some traction on this bug. We have multiple possibilities:
> 
> A) Build lightning in one language (i.e each localized language)
> B) Build real langpacks

C) ship a single multi-locale lightning

> What are the advantages?
> Which is easier to accomplish?
> What does a langpack look like?
>   - i.e: is it just an extension with only a locale definition?

A) and B) are easy to do from a project management and build perspective, but they're hard to ship to end-users. If you'd go into the addons manager in TB and search for lighting, you'd get a dozen plus matching addons.

C) is tougher to build, but is the best user experience in terms of discoverability. I don't have an overview how much of a download size increase a single locale for lightning is, though.

My personal take would be to re-fix 352546, building lightning with all locales. That has been backed out in comm-central, AFAICT. With l10n-merge, doing that should be fairly good even for nightly builds.

Comment 18

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

(In reply to comment #17)


> C) ship a single multi-locale lightning
I think we should do this for releases only. Maybe we can create a target repack-all that creates this multilang lightning.xpi as in bug 352546. We just don't want to do this for nightlies, since if one locale is missing a file, then the whole build will break.

> My personal take would be to re-fix 352546, building lightning with all
> locales. That has been backed out in comm-central, AFAICT. With l10n-merge,
> doing that should be fairly good even for nightly builds.
What exactly does l10n-merge do? Not sure I understood it correctly, but does it make sure there are no missing files and extends missing entities? In this case we could consider making the nightlies multi-language too.

Comment 19

[Axel Hecht [:Pike]]

(In reply to comment #18)
> What exactly does l10n-merge do? Not sure I understood it correctly, but does
> it make sure there are no missing files and extends missing entities? In this
> case we could consider making the nightlies multi-language too.

That's exactly what l10n-merge does. We're going to use that to create Firefox l10n builds for a nightly update channel, too, so yes, we're pretty confident that the builds created with l10n-merge don't fall down on missing strings or files.

Comment 20

[Stefan Sitter [:ssitter]]

(In reply to comment #17)
> C) is tougher to build, but is the best user experience in terms 
> of discoverability. I don't have an overview how much of a 
> download size increase a single locale for lightning is, though.

Based on my experience with the previous win32 releases the download size increase is almost 100 % (~1 MByte for en-US Lightning, ~2 MByte with all locales.)

In my opinion for releases all locales should be included in one package.

Comment 21

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

(In reply to comment #18)
> > C) ship a single multi-locale lightning
> I think we should do this for releases only. Maybe we can create a target
> repack-all that creates this multilang lightning.xpi as in bug 352546. We just
> don't want to do this for nightlies, since if one locale is missing a file,
> then the whole build will break.
Also, if we want each locale builder to notify that the locale is orange/red, we need separate tasks. So we definitely need to create a per-locale .xpi. Personally I'm fine with including all locales in the nightlies with l10n-merge, since 2 MB is not too much to download in times of broadband.

Comment 22

[Philippe M. Chiasson (:gozer)]

Comment on attachment 385706 [details] [diff] [review]
[checked in] rev1 - Allow repackaging lightning into separate locales

Looks good, and tested with the 'de' locale and produced a german lightning XPI. Good enough for me!

Comment 23

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Comment on attachment 385706 [details] [diff] [review]
[checked in] rev1 - Allow repackaging lightning into separate locales

Patch pushed to comm-central <http://hg.mozilla.org/comm-central/rev/7cc483a6abe2>

Now we hopefully only need the buildbot changes.

Comment 24

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Attachment: [checked in] Fix regression - v1

For some reason, since repack-l10n is the first target, calling make in obj/calendar/lightning only does the repack. Putting an empty all:: target before it fixes. Not sure if this is the right approach, gozer maybe you know why and how to fix better?

Comment 25

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Comment on attachment 387855 [details] [diff] [review]
[checked in] Fix regression - v1

r=gozer via irc. We chose to just put the l10n rules below rules.mk

Comment 26

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Comment on attachment 387855 [details] [diff] [review]
[checked in] Fix regression - v1

pushed as rev f028d60fbcfc

Comment 27

[Simon Paquet [:sipaq]]

What is still missing here, so that we can announce this to localizers?

Comment 28

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

This bug still needs the buildbot changes so that the l10n builders actually execute the above noted make call.

Comment 29

[Philippe M. Chiasson (:gozer)]

(In reply to comment #28)
> This bug still needs the buildbot changes so that the l10n builders actually
> execute the above noted make call.

Taking this one over, as it's time for some buildbot foo

Comment 30

[Mark Banner (:standard8)]

Adding dev-doc-needed keyword - localisers need to know how to do this.

Comment 31

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Developer documentation has been started at https://developer.mozilla.org/en/Calendar/Localization I'd like to see some more information on how to build Sunbird and some more linkage. Sipaq, maybe you can complete this page?

Comment 32

[Simon Paquet [:sipaq]]

(In reply to comment #31)
> Developer documentation has been started at
> https://developer.mozilla.org/en/Calendar/Localization I'd like to see some
> more information on how to build Sunbird and some more linkage. Sipaq, maybe
> you can complete this page?

I've done some work on that page. I've adapted the information found on https://developer.mozilla.org/en/Thunderbird_Localization, moved the relevant bits over to https://developer.mozilla.org/en/Calendar/Localization and added some tags.

Information on how to build a localized Sunbird build is still missing though.

Comment 33

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Gozer, any possibility you could get to this soon? It would really be good for localizers to get localized builds without needing to build themselves, especially to shorten the release process for lightning 1.0b1. This means they can take a look without needing to wait for a release candidate.

Comment 34

[Simon Paquet [:sipaq]]

(In reply to comment #33)
> Gozer, any possibility you could get to this soon? It would really be good for
> localizers to get localized builds without needing to build themselves,
> especially to shorten the release process for lightning 1.0b1. This means they
> can take a look without needing to wait for a release candidate.

Gozer: PING!

Comment 35

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Sorry for not updating. I've started to look into this myself now and talked to gozer. I've successfully set up a buildbot, but still am having trouble completing a build. It will probably take me quite a bit longer than it would gozer, but I'm working on it.

Comment 36

[Michael Duelli]

There seems to be something wrong with (at least) German locales with Lightning 1.0beta1 release candidate.

The first three menu entries (File, Edit, View) seem to have empty strings and are not shown as a consequence.

Comment 37

[Cédric Corazza]

(In reply to comment #36)
This looks like bug 536138.

Comment 38

[Simon Paquet [:sipaq]]

Philipp, Gozer,

how can we get more traction here? This is really *the* major showstopper for Lightning localizers, now that Sunbird has been discontinued.

Comment 39

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

sipaq, I'm actively working on this bug, its about 30-40% done. I do hope to have this fixed in the next two weeks.

Comment 40

[Hubert Gajewski]

(In reply to comment #38)
> This is really *the* major showstopper for
> Lightning localizers, now that Sunbird has been discontinued.

There is temporary workaround for this issue: http://l10n.mozilla.org/~hubert/tools/compare-locales/comm-1.9.2/

Comment 41

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Attachment: [checked in] Add more packaging code

This patch moves the repack stuff into a separate file and adds some more packaging code. I've addded a mock application.ini to the lightning.xpi root, this makes it easier to parse the build ids and revisions using the build system.

Comment 42

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Attachment: Extension Factory for Buildbot

With this factory (and some changes to master.cfg), I was able to repackage a downloaded build into a -de build.

I'm not too happy about the inheritance here though. In CCBaseExtensionRepackFactory, no matter which order I call the __init__ functions, I get an error. The solution I have right now is to copy the setters for baseWorkDir/objdir/extensionPath, but this duplicates code and looks ugly. Maybe you have an idea how this could be fixed.

I'm not asking for review on this patch yet, since I need to make postUploadCmd do the right thing.

For the config.py part, something like this is needed:

BRANCHES['comm-1.9.2-lightning']['extensions'] = {
    'lightning': {
        'subdir': "calendar/lightning",
        'l10n': True
    }
}

and of course master.cfg needs to be changed to instanciate the factory. I'll upload patches for those when all the pending changes to master.cfg are through and I have feedback here.

Comment 43

[Philippe M. Chiasson (:gozer)]

Comment on attachment 476674 [details] [diff] [review]
[checked in] Add more packaging code

Only question I have is the 

-builtin(include, mozilla/build/autoconf/acwinpaths.m4)dnl

In aclocal.m4

I am assuming that's an accidental inclusion in the patch. Otherwise, looks good to me.

Comment 44

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Comment on attachment 476674 [details] [diff] [review]
[checked in] Add more packaging code

Yes, the aclocal stuff was temporary and not part of the patch. I've taken that out and then pushed this patch to comm-central

http://hg.mozilla.org/comm-central/rev/5d2e14c3261e

Comment 45

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

gozer, any chance you could take a look at the factory soon? :)

Comment 46

[Philippe M. Chiasson (:gozer)]

Comment on attachment 476675 [details] [diff] [review]
Extension Factory for Buildbot

Generally looking good.  I agree that the inheritance stuff is tricky, and I am not
versed in Python enough to know what the tidy solution is to the _init_ problem.


>class BaseExtensionRepackFactory(BaseRepackFactory):
>
>    def __init__(self, extensionPath, enUSBinaryURL, objdir='', baseWorkDir='build',
>    [...]
>
>        env.update({'EN_US_BINARY_URL': enUSBinaryURL})

Does this environment setting really need to be global to the factory and not set for
only the step that needs it ?

>        BaseRepackFactory.__init__(self, mozillaDir=mozillaDir, env=env, **kwargs)
> [...]
>
>    def updateEnUS(self):
> [...]
>        self.addStep(ShellCommand,
>         command=['hg', 'update', '-r', WithProperties('%(moz_revision)s')],

As everywhere else, I'd make sure that's an hg up -C

>         haltOnFailure=True,
>         workdir='%s/%s' % (self.baseWorkDir, self.origSrcDir)
>        )
> [...]
>
>class CCBaseExtensionRepackFactory(BaseExtensionRepackFactory, CCBaseRepackFactory):
> [...]
>
>    def updateEnUS(self):
> [...]
>        self.addStep(ShellCommand,
>         command=['hg', 'update', '-r', WithProperties('%(moz_revision)s')],

hg up -C comment as previously

>         haltOnFailure=True,
>         workdir='%s/%s' % (self.baseWorkDir, self.mozillaSrcDir)
>        )

Comment 47

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Buildbot master contains local code that uses this factory (with slight changes due to buildbot 0.7)

Backported packaging code in comm-1.9.2 rev ae66d580d810

Comment 48

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

platform.ini is not created during repack builds, added code to recreate it from the bundled application ini, see

https://hg.mozilla.org/releases/comm-1.9.2/rev/bf07c1da4ced

NOTE: This needs to be forward-ported as soon as everything works.

Comment 49

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

After 22 tries, we have a success! Forward ported patches:

http://hg.mozilla.org/comm-central/4b7dee4ee6d7

Comment 50

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Build log of succeeded build:

http://build.mozillamessaging.com/buildbot/calendar/builders/Linux%20comm-1.9.2%20lightning%20l10n%20extension%20build/builds/22

Output of succeeded build:

http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/nightly/latest-comm-1.9.2-l10n/

All thats left now is getting scheduling right. I'm currently using the NightlyL10n scheduler, but I have the feeling it will not work.

Thunderbird's master.cfg looks totally different and they seem to use the TriggerableL10n scheduler. I'll ask gozer about this.

Comment 51

[Stefan Sitter [:ssitter]]

(In reply to comment #50)
> Output of succeeded build:
> http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/nightly/latest-comm-1.9.2-l10n/

The file is just named lightning-de.xpi but is only valid for Linux. Either this should be indicated in the package name or the file should be located in a linux-xpi package like the English nightly builds.

Comment 52

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

(In reply to comment #51)
> (In reply to comment #50)
> The file is just named lightning-de.xpi but is only valid for Linux. Either
> this should be indicated in the package name or the file should be located in a
> linux-xpi package like the English nightly builds.
Yes, fixed. Sorry for the confusion and thanks!


I've just committed a packager fix so that trunk will also work, which uses omnijar and a flat file format for the chrome jars.

http://hg.mozilla.org/comm-central/rev/eb1b068abaa3
http://hg.mozilla.org/releases/comm-1.9.2/rev/181ed9ad67da

Comment 53

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

This is mostly fixed now! There are a few small tweaks to do, I've filed/pushed bug 633262 so we don't get those ugly failures after post_upload.py almost completes. I'll take care of things as they come and file new bugs to actually commit the buildbot changes that are currently only local on the master.

Right now we will provide nightly l10n builds, our build machinery isn't strong enough to also provide dep builds. Maybe we can get a new 32 bit linux machine, or possibly move some of the build jobs to the faster 64 bit box and cross compile.

I will blog about l10n builds this soon.

Comment 54

[Stefan Sitter [:ssitter]]

There is one big problem with the current packages. They will not work in the en-US Thunderbird nightly builds although the localization is complete and error free.

Reason: The chrome.manifest file still contains references to the en-US files. But this files are not shipped in the package. But the en-US nightly Thunderbird build will default to them and fails during loading.

Removing the following lines from chrome.manifest solves the problem:
locale lightning en-US jar:chrome/lightning-en-US.jar!/locale/en-US/lightning/
locale calendar en-US jar:chrome/calendar-en-US.jar!/locale/en-US/calendar/

Comment 55

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Good reason to reopen this one, I'll take care.

Comment 56

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

This should fix the en-US issue:

http://hg.mozilla.org/comm-central/rev/40969e8de0a8
http://hg.mozilla.org/releases/comm-1.9.2/rev/f95b28fad878

Comment 57

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

That seemed to work, closing again.