Allow building localized lightning langpacks

RESOLVED FIXED

Status

Calendar
Build Config
P1
major
RESOLVED FIXED
11 years ago
6 years ago

People

(Reporter: Thorsten 'nightrat' Fritz, Assigned: Fallen)

Tracking

({dev-doc-needed})

unspecified
dev-doc-needed
Bug Flags:
blocking-calendar1.0 +

Details

(Whiteboard: [needed beta][no l10n impact])

Attachments

(4 attachments, 1 obsolete attachment)

(Reporter)

Description

11 years ago
Building of localized lightning extension should be available.

Bug 344602 is fixed.
CCing rob_strong so he can tell us the Right Way to localize extensions so they work nice with the Add-ons Manager.
The choices are to either build the extension with all available locales or build the extension with one locale and build language packs for the available languages. The language packs could then have the extension as a dependency.

Bug 285848 will provide automatic discovery of language packs for an extension.
Bug 345610 shows additional work planned for extension dependencies.
Taking this.
Assignee: nobody → mattwillis
OS: Linux → All
Hardware: PC → All
Waiting for all localization to complete before shipping Lightning sounds like a trade-off we don't want to make.  Given our discussion in the calendar weekly meeting, I'll work on the lightning+langpack scenario.

Status: NEW → ASSIGNED
Created attachment 237689 [details] [diff] [review]
rev0 - Allows building of langpacks for lightning similar to sunbirds

make -C calendar/locales langpack-ab-CD
should make a langpack xpi for lightning in your dist/install directory.
Attachment #237689 - Flags: second-review?(dmose)
Attachment #237689 - Flags: first-review?(ssitter)

Comment 6

11 years ago
Comment on attachment 237689 [details] [diff] [review]
rev0 - Allows building of langpacks for lightning similar to sunbirds

I only did a quick test until now and didn't look at makefile changes in detail.

1. Building Sunbird, going to SbObjDir/calendar/locales and run make langpack-ab-CD produces lightning langpack instead of sunbird langpack. This should be fixed.

2. Building Thunderbird, going to LtnObjDir/calendar/locales and run make langpack-ab-CD produces lightning langpacks. But there is a problem with the lightning files. en-US/lightning files were packaged in all langpacks instead of correct one.

 +++ making chrome /cygdrive/y/obj/ltn-test/calendar/locales  
     => ../../dist/xpi-stage/locale-de/chrome/lightning-de.jar
 +++ updating chrome ../../dist/xpi-stage/locale-de/chrome/../chrome.manifest
   zip warning: ../../../locale-de/chrome/lightning-de.jar not found or empty
   adding: locale/en-US/lightning/lightning.dtd (stored 0%)
   adding: locale/en-US/lightning/lightning.properties (stored 0%)

3. I did not get the langpack to work. Either there were missing entities or ltnGetString() failed or some other errors. Maybe related to point 2.

4. But I saw that error messages in console were translated. The langpack contains toolkit locals too (in 'ab-CD.jar'). I think this causes problems with toolkit locals from host application. Our normal lightning.xpi ships 'calendar-en-US.jar' and 'lightning-en-US.jar' only.

5. Package naming: I think the package name should use the Lightning version in name not the Thunderbird version. This makes it more clear where it belongs to. For example lightning-3.0a1.ab-CD.langpack.xpi should be lightning-0.1+.ab-CD.langpack.xpi instead.

6. install.rdf: em:version="3.0a1" I think this should be the Lightning version too.

7. install.rdf: <em:minVersion>3.0a1 / <em:maxVersion>3.0a1 I think we want the supported Thunderbird version here instead of the one build in. For example when building from branch this should be 1.5.0.* to 2.0.* instead of 2.0a1 to 2.0a1.

8. Makefile changes seems to assume that not-Sunbird means Thunderbird in every case. I think it should be prepared in a way so that it is easy to extend e.g. SeaMonkey and Firefox later.
Attachment #237689 - Flags: first-review?(ssitter) → first-review-
Comment on attachment 237689 [details] [diff] [review]
rev0 - Allows building of langpacks for lightning similar to sunbirds

Removing r2 request.

Ya. This needs a bit more work. Not going to make 0.3 using this approach.
Attachment #237689 - Flags: second-review?(dmose)
Clarified summary. This is about _langpacks_, not just having a localized lightning (which for 0.3, we'll probably do with a big ol' extension)

Moved to Ln 0.5
Summary: Allow building localized lightning versions → Allow building localized lightning langpacks
Target Milestone: --- → Lightning 0.5
This is not going to make 0.5.
Assignee: lilmatt → nobody
Status: ASSIGNED → NEW
Target Milestone: Lightning 0.5 → Lightning 0.7
(Assignee)

Comment 10

10 years ago
Milestone 0.7 Consolidation. Filter on PinkyANDBrain to get rid of bugspam.
Target Milestone: Lightning 0.7 → Sunbird 0.7
Target Milestone: 0.7 → ---
Version: Trunk → unspecified

Comment 11

9 years ago
We now can build Lightning with all locales included (Bug 352546). I think this bug is therefore no longer required.
(Assignee)

Comment 12

9 years ago
(In reply to comment #6)
> 1. Building Sunbird, going to SbObjDir/calendar/locales and run make
> langpack-ab-CD produces lightning langpack instead of sunbird langpack. This
> should be fixed.
Is this still the case? If so, we should file/find a separate bug.

> 2. Building Thunderbird, going to LtnObjDir/calendar/locales and run make
> langpack-ab-CD produces lightning langpacks. But there is a problem with the
> lightning files. en-US/lightning files were packaged in all langpacks instead
> of correct one.
Just out of curiosity, what happens? Although we don't really need lightning langpacks, I think we might want to at least leave the possibility, in case a new locale wants to test changes without building a full lightning? I don't really mind if we don't have a ltn langpack though, but maybe we should spit out an error that building langpacks is not supported.

> 8. Makefile changes seems to assume that not-Sunbird means Thunderbird in every
> case. I think it should be prepared in a way so that it is easy to extend e.g.
> SeaMonkey and Firefox later.
How much of this is true?
Based on our discussions at FOSDEM, this should definitely block 1.0 because we need daily localized Lightning builds to enable localization testing for localizers once we no longer have daily localized Sunbird builds.
Component: Lightning Only → Build Config
Flags: blocking-calendar1.0?
QA Contact: lightning → build
(Assignee)

Comment 14

8 years ago
So, lets get some traction on this bug. We have multiple possibilities:

A) Build lightning in one language (i.e each localized language)
B) Build real langpacks

What are the advantages?
Which is easier to accomplish?
What does a langpack look like?
  - i.e: is it just an extension with only a locale definition?
(Assignee)

Comment 15

8 years ago
Gozer, any ideas on this from a build perspective?
Flags: blocking-calendar1.0? → blocking-calendar1.0+
Whiteboard: [not needed beta][no l10n impact]
(Assignee)

Comment 16

8 years ago
Created attachment 385706 [details] [diff] [review]
[checked in] rev1 - Allow repackaging lightning into separate locales

This patch should be a nice start. The process is as follows:

$ make AB_CD=de repack-l10n

* Removes any existing xpi-stage/lightning-de
* Copies xpi-stage/lightning to xpi-stage/lightning-de
* Removes xpi-stage/lightning-de/chrome/(calendar|lightning)-en-US.jar
* Makes all calendar locales for de, putting them in xpi-stage/lightning-de
* Recreates install.rdf with $(L10NBASEDIR)/de/defines.inc included
* Processes $(L10NBASEDIR)/de/lightning-l10n.js into xpi-stage/lightning-de/$(PREF_DIR)/lightning-l10n.js

I hope I didn't miss anything.
Assignee: nobody → philipp
Attachment #237689 - Attachment is obsolete: true
Status: NEW → ASSIGNED
Attachment #385706 - Flags: review?(gozer)
(In reply to comment #14)
> So, lets get some traction on this bug. We have multiple possibilities:
> 
> A) Build lightning in one language (i.e each localized language)
> B) Build real langpacks

C) ship a single multi-locale lightning

> What are the advantages?
> Which is easier to accomplish?
> What does a langpack look like?
>   - i.e: is it just an extension with only a locale definition?

A) and B) are easy to do from a project management and build perspective, but they're hard to ship to end-users. If you'd go into the addons manager in TB and search for lighting, you'd get a dozen plus matching addons.

C) is tougher to build, but is the best user experience in terms of discoverability. I don't have an overview how much of a download size increase a single locale for lightning is, though.

My personal take would be to re-fix 352546, building lightning with all locales. That has been backed out in comm-central, AFAICT. With l10n-merge, doing that should be fairly good even for nightly builds.
(Assignee)

Comment 18

8 years ago
(In reply to comment #17)


> C) ship a single multi-locale lightning
I think we should do this for releases only. Maybe we can create a target repack-all that creates this multilang lightning.xpi as in bug 352546. We just don't want to do this for nightlies, since if one locale is missing a file, then the whole build will break.

> My personal take would be to re-fix 352546, building lightning with all
> locales. That has been backed out in comm-central, AFAICT. With l10n-merge,
> doing that should be fairly good even for nightly builds.
What exactly does l10n-merge do? Not sure I understood it correctly, but does it make sure there are no missing files and extends missing entities? In this case we could consider making the nightlies multi-language too.
(In reply to comment #18)
> What exactly does l10n-merge do? Not sure I understood it correctly, but does
> it make sure there are no missing files and extends missing entities? In this
> case we could consider making the nightlies multi-language too.

That's exactly what l10n-merge does. We're going to use that to create Firefox l10n builds for a nightly update channel, too, so yes, we're pretty confident that the builds created with l10n-merge don't fall down on missing strings or files.

Comment 20

8 years ago
(In reply to comment #17)
> C) is tougher to build, but is the best user experience in terms 
> of discoverability. I don't have an overview how much of a 
> download size increase a single locale for lightning is, though.

Based on my experience with the previous win32 releases the download size increase is almost 100 % (~1 MByte for en-US Lightning, ~2 MByte with all locales.)

In my opinion for releases all locales should be included in one package.
(Assignee)

Comment 21

8 years ago
(In reply to comment #18)
> > C) ship a single multi-locale lightning
> I think we should do this for releases only. Maybe we can create a target
> repack-all that creates this multilang lightning.xpi as in bug 352546. We just
> don't want to do this for nightlies, since if one locale is missing a file,
> then the whole build will break.
Also, if we want each locale builder to notify that the locale is orange/red, we need separate tasks. So we definitely need to create a per-locale .xpi. Personally I'm fine with including all locales in the nightlies with l10n-merge, since 2 MB is not too much to download in times of broadband.
Attachment #385706 - Flags: review?(gozer) → review+
Comment on attachment 385706 [details] [diff] [review]
[checked in] rev1 - Allow repackaging lightning into separate locales

Looks good, and tested with the 'de' locale and produced a german lightning XPI. Good enough for me!
(Assignee)

Comment 23

8 years ago
Comment on attachment 385706 [details] [diff] [review]
[checked in] rev1 - Allow repackaging lightning into separate locales

Patch pushed to comm-central <http://hg.mozilla.org/comm-central/rev/7cc483a6abe2>

Now we hopefully only need the buildbot changes.
Attachment #385706 - Attachment description: rev1 - Allow repackaging lightning into separate locales → [checked in] rev1 - Allow repackaging lightning into separate locales
(Assignee)

Comment 24

8 years ago
Created attachment 387855 [details] [diff] [review]
[checked in] Fix regression - v1

For some reason, since repack-l10n is the first target, calling make in obj/calendar/lightning only does the repack. Putting an empty all:: target before it fixes. Not sure if this is the right approach, gozer maybe you know why and how to fix better?
Attachment #387855 - Flags: review?(gozer)
(Assignee)

Comment 25

8 years ago
Comment on attachment 387855 [details] [diff] [review]
[checked in] Fix regression - v1

r=gozer via irc. We chose to just put the l10n rules below rules.mk
Attachment #387855 - Flags: review?(gozer) → review+
(Assignee)

Comment 26

8 years ago
Comment on attachment 387855 [details] [diff] [review]
[checked in] Fix regression - v1

pushed as rev f028d60fbcfc
Attachment #387855 - Attachment description: Fix regression - v1 → [checked in] Fix regression - v1
What is still missing here, so that we can announce this to localizers?
(Assignee)

Comment 28

8 years ago
This bug still needs the buildbot changes so that the l10n builders actually execute the above noted make call.
(In reply to comment #28)
> This bug still needs the buildbot changes so that the l10n builders actually
> execute the above noted make call.

Taking this one over, as it's time for some buildbot foo
Assignee: philipp → gozer
Adding dev-doc-needed keyword - localisers need to know how to do this.
Keywords: dev-doc-needed
(Assignee)

Comment 31

8 years ago
Developer documentation has been started at https://developer.mozilla.org/en/Calendar/Localization I'd like to see some more information on how to build Sunbird and some more linkage. Sipaq, maybe you can complete this page?
(In reply to comment #31)
> Developer documentation has been started at
> https://developer.mozilla.org/en/Calendar/Localization I'd like to see some
> more information on how to build Sunbird and some more linkage. Sipaq, maybe
> you can complete this page?

I've done some work on that page. I've adapted the information found on https://developer.mozilla.org/en/Thunderbird_Localization, moved the relevant bits over to https://developer.mozilla.org/en/Calendar/Localization and added some tags.

Information on how to build a localized Sunbird build is still missing though.
(Assignee)

Comment 33

8 years ago
Gozer, any possibility you could get to this soon? It would really be good for localizers to get localized builds without needing to build themselves, especially to shorten the release process for lightning 1.0b1. This means they can take a look without needing to wait for a release candidate.
(In reply to comment #33)
> Gozer, any possibility you could get to this soon? It would really be good for
> localizers to get localized builds without needing to build themselves,
> especially to shorten the release process for lightning 1.0b1. This means they
> can take a look without needing to wait for a release candidate.

Gozer: PING!
(Assignee)

Comment 35

8 years ago
Sorry for not updating. I've started to look into this myself now and talked to gozer. I've successfully set up a buildbot, but still am having trouble completing a build. It will probably take me quite a bit longer than it would gozer, but I'm working on it.

Comment 36

8 years ago
There seems to be something wrong with (at least) German locales with Lightning 1.0beta1 release candidate.

The first three menu entries (File, Edit, View) seem to have empty strings and are not shown as a consequence.

Comment 37

8 years ago
(In reply to comment #36)
This looks like bug 536138.
(Assignee)

Updated

7 years ago
Whiteboard: [not needed beta][no l10n impact] → [needed beta][no l10n impact]
Philipp, Gozer,

how can we get more traction here? This is really *the* major showstopper for Lightning localizers, now that Sunbird has been discontinued.
Severity: normal → major
Priority: -- → P1
(Assignee)

Comment 39

7 years ago
sipaq, I'm actively working on this bug, its about 30-40% done. I do hope to have this fixed in the next two weeks.
Assignee: gozer → philipp

Comment 40

7 years ago
(In reply to comment #38)
> This is really *the* major showstopper for
> Lightning localizers, now that Sunbird has been discontinued.

There is temporary workaround for this issue: http://l10n.mozilla.org/~hubert/tools/compare-locales/comm-1.9.2/
(Assignee)

Comment 41

7 years ago
Created attachment 476674 [details] [diff] [review]
[checked in] Add more packaging code

This patch moves the repack stuff into a separate file and adds some more packaging code. I've addded a mock application.ini to the lightning.xpi root, this makes it easier to parse the build ids and revisions using the build system.
Attachment #476674 - Flags: review?(gozer)
(Assignee)

Comment 42

7 years ago
Created attachment 476675 [details] [diff] [review]
Extension Factory for Buildbot

With this factory (and some changes to master.cfg), I was able to repackage a downloaded build into a -de build.

I'm not too happy about the inheritance here though. In CCBaseExtensionRepackFactory, no matter which order I call the __init__ functions, I get an error. The solution I have right now is to copy the setters for baseWorkDir/objdir/extensionPath, but this duplicates code and looks ugly. Maybe you have an idea how this could be fixed.

I'm not asking for review on this patch yet, since I need to make postUploadCmd do the right thing.

For the config.py part, something like this is needed:

BRANCHES['comm-1.9.2-lightning']['extensions'] = {
    'lightning': {
        'subdir': "calendar/lightning",
        'l10n': True
    }
}

and of course master.cfg needs to be changed to instanciate the factory. I'll upload patches for those when all the pending changes to master.cfg are through and I have feedback here.
Attachment #476675 - Flags: feedback?(gozer)
Comment on attachment 476674 [details] [diff] [review]
[checked in] Add more packaging code

Only question I have is the 

-builtin(include, mozilla/build/autoconf/acwinpaths.m4)dnl

In aclocal.m4

I am assuming that's an accidental inclusion in the patch. Otherwise, looks good to me.
Attachment #476674 - Flags: review?(gozer) → review+
(Assignee)

Comment 44

7 years ago
Comment on attachment 476674 [details] [diff] [review]
[checked in] Add more packaging code

Yes, the aclocal stuff was temporary and not part of the patch. I've taken that out and then pushed this patch to comm-central

http://hg.mozilla.org/comm-central/rev/5d2e14c3261e
Attachment #476674 - Attachment description: Add more packaging code → [checked in] Add more packaging code
(Assignee)

Comment 45

7 years ago
gozer, any chance you could take a look at the factory soon? :)
(Assignee)

Updated

7 years ago
Whiteboard: [needed beta][no l10n impact] → [needed beta][no l10n impact][needs feedback]
Comment on attachment 476675 [details] [diff] [review]
Extension Factory for Buildbot

Generally looking good.  I agree that the inheritance stuff is tricky, and I am not
versed in Python enough to know what the tidy solution is to the _init_ problem.


>class BaseExtensionRepackFactory(BaseRepackFactory):
>
>    def __init__(self, extensionPath, enUSBinaryURL, objdir='', baseWorkDir='build',
>    [...]
>
>        env.update({'EN_US_BINARY_URL': enUSBinaryURL})

Does this environment setting really need to be global to the factory and not set for
only the step that needs it ?

>        BaseRepackFactory.__init__(self, mozillaDir=mozillaDir, env=env, **kwargs)
> [...]
>
>    def updateEnUS(self):
> [...]
>        self.addStep(ShellCommand,
>         command=['hg', 'update', '-r', WithProperties('%(moz_revision)s')],

As everywhere else, I'd make sure that's an hg up -C

>         haltOnFailure=True,
>         workdir='%s/%s' % (self.baseWorkDir, self.origSrcDir)
>        )
> [...]
>
>class CCBaseExtensionRepackFactory(BaseExtensionRepackFactory, CCBaseRepackFactory):
> [...]
>
>    def updateEnUS(self):
> [...]
>        self.addStep(ShellCommand,
>         command=['hg', 'update', '-r', WithProperties('%(moz_revision)s')],

hg up -C comment as previously

>         haltOnFailure=True,
>         workdir='%s/%s' % (self.baseWorkDir, self.mozillaSrcDir)
>        )
Attachment #476675 - Flags: feedback?(gozer) → feedback+
Whiteboard: [needed beta][no l10n impact][needs feedback] → [needed beta][no l10n impact]
(Assignee)

Comment 47

6 years ago
Buildbot master contains local code that uses this factory (with slight changes due to buildbot 0.7)

Backported packaging code in comm-1.9.2 rev ae66d580d810
(Assignee)

Comment 48

6 years ago
platform.ini is not created during repack builds, added code to recreate it from the bundled application ini, see

https://hg.mozilla.org/releases/comm-1.9.2/rev/bf07c1da4ced

NOTE: This needs to be forward-ported as soon as everything works.
(Assignee)

Comment 49

6 years ago
After 22 tries, we have a success! Forward ported patches:

http://hg.mozilla.org/comm-central/4b7dee4ee6d7
(Assignee)

Comment 50

6 years ago
Build log of succeeded build:

http://build.mozillamessaging.com/buildbot/calendar/builders/Linux%20comm-1.9.2%20lightning%20l10n%20extension%20build/builds/22

Output of succeeded build:

http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/nightly/latest-comm-1.9.2-l10n/

All thats left now is getting scheduling right. I'm currently using the NightlyL10n scheduler, but I have the feeling it will not work.

Thunderbird's master.cfg looks totally different and they seem to use the TriggerableL10n scheduler. I'll ask gozer about this.

Comment 51

6 years ago
(In reply to comment #50)
> Output of succeeded build:
> http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/nightly/latest-comm-1.9.2-l10n/

The file is just named lightning-de.xpi but is only valid for Linux. Either this should be indicated in the package name or the file should be located in a linux-xpi package like the English nightly builds.
(Assignee)

Comment 52

6 years ago
(In reply to comment #51)
> (In reply to comment #50)
> The file is just named lightning-de.xpi but is only valid for Linux. Either
> this should be indicated in the package name or the file should be located in a
> linux-xpi package like the English nightly builds.
Yes, fixed. Sorry for the confusion and thanks!


I've just committed a packager fix so that trunk will also work, which uses omnijar and a flat file format for the chrome jars.

http://hg.mozilla.org/comm-central/rev/eb1b068abaa3
http://hg.mozilla.org/releases/comm-1.9.2/rev/181ed9ad67da
(Assignee)

Comment 53

6 years ago
This is mostly fixed now! There are a few small tweaks to do, I've filed/pushed bug 633262 so we don't get those ugly failures after post_upload.py almost completes. I'll take care of things as they come and file new bugs to actually commit the buildbot changes that are currently only local on the master.

Right now we will provide nightly l10n builds, our build machinery isn't strong enough to also provide dep builds. Maybe we can get a new 32 bit linux machine, or possibly move some of the build jobs to the faster 64 bit box and cross compile.

I will blog about l10n builds this soon.
Status: ASSIGNED → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED

Comment 54

6 years ago
There is one big problem with the current packages. They will not work in the en-US Thunderbird nightly builds although the localization is complete and error free.

Reason: The chrome.manifest file still contains references to the en-US files. But this files are not shipped in the package. But the en-US nightly Thunderbird build will default to them and fails during loading.

Removing the following lines from chrome.manifest solves the problem:
locale lightning en-US jar:chrome/lightning-en-US.jar!/locale/en-US/lightning/
locale calendar en-US jar:chrome/calendar-en-US.jar!/locale/en-US/calendar/
(Assignee)

Comment 55

6 years ago
Good reason to reopen this one, I'll take care.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 56

6 years ago
This should fix the en-US issue:

http://hg.mozilla.org/comm-central/rev/40969e8de0a8
http://hg.mozilla.org/releases/comm-1.9.2/rev/f95b28fad878
(Assignee)

Comment 57

6 years ago
That seemed to work, closing again.
Status: REOPENED → RESOLVED
Last Resolved: 6 years ago6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.