Last Comment Bug 519684 - Create multi-locale builds for fennec on maemo for nightlies
: Create multi-locale builds for fennec on maemo for nightlies
Status: RESOLVED FIXED
[fennec l10n][l10n]
:
Product: Release Engineering
Classification: Other
Component: Other (show other bugs)
: other
: ARM Maemo
: P1 normal (vote)
: ---
Assigned To: Armen Zambrano [:armenzg] - Engineering productivity
:
Mentors:
Depends on: 519468 519682 519696 524820 525470 525811 526233 531695
Blocks: 513761 525301 525327 526154
  Show dependency treegraph
 
Reported: 2009-09-30 05:23 PDT by Axel Hecht [:Pike]
Modified: 2013-08-12 21:54 PDT (History)
7 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
bash script that downloads the en-US Fennec builds plus all the locales specified in mobile-browser/locales/all-locales and it generates a Fennec multi-locale build (1.93 KB, text/plain)
2009-10-06 14:34 PDT, Armen Zambrano [:armenzg] - Engineering productivity
no flags Details
[log] after applying patch I still can't do "make chrome-%" (6.02 KB, text/plain)
2009-10-13 14:34 PDT, Armen Zambrano [:armenzg] - Engineering productivity
no flags Details
[WIP] this shows how to generate a multi-locale build for Fennec with all the steps (1.59 KB, text/plain)
2009-10-14 13:29 PDT, Armen Zambrano [:armenzg] - Engineering productivity
no flags Details
[WIP](patch 1 of 2) add --with-l10n-base to mozconfig and pass multiLocale and L10nRepoPath variables to the Maemo's nightly factory (1.83 KB, patch)
2009-10-16 14:02 PDT, Armen Zambrano [:armenzg] - Engineering productivity
no flags Details | Diff | Splinter Review
[WIP][buildbotcustom](patch 2 of 2) before packaging we iterate through a list of locales to check them out, run compare-locales and add the chrome for the locale (6.67 KB, patch)
2009-10-16 14:04 PDT, Armen Zambrano [:armenzg] - Engineering productivity
no flags Details | Diff | Splinter Review
[WIP][Approach 2] make target that takes from a file like maemo-locales which locales to add to a multi-locale build (2.13 KB, patch)
2009-10-16 14:05 PDT, Armen Zambrano [:armenzg] - Engineering productivity
no flags Details | Diff | Splinter Review
[WIP] (patch 1 of 2) add --with-l10n-base to mozconfig and use a MultiNightlyL10n scheduler (8.70 KB, patch)
2009-10-21 09:25 PDT, Armen Zambrano [:armenzg] - Engineering productivity
no flags Details | Diff | Splinter Review
[WIP][buildbotcustom](patch 2 of 2) allow to submit BuildSet with list of locales as a property and use it in MaemoBuildFactory (11.59 KB, patch)
2009-10-21 09:28 PDT, Armen Zambrano [:armenzg] - Engineering productivity
no flags Details | Diff | Splinter Review
[WIP] (patch 1 of 2) add --with-l10n-base to mozconfig and use a MultiNightlyL10n scheduler (21.77 KB, patch)
2009-10-22 18:09 PDT, Armen Zambrano [:armenzg] - Engineering productivity
no flags Details | Diff | Splinter Review
[WIP][buildbotcustom](patch 2 of 2) allow to submit BuildSet with list of locales as a property and use it in MaemoBuildFactory (13.39 KB, patch)
2009-10-22 18:12 PDT, Armen Zambrano [:armenzg] - Engineering productivity
no flags Details | Diff | Splinter Review
[WIP] (patch 1 of 2) add --with-l10n-base to mozconfig and use a MultiNightlyL10n scheduler (21.87 KB, patch)
2009-10-23 14:29 PDT, Armen Zambrano [:armenzg] - Engineering productivity
no flags Details | Diff | Splinter Review
[WIP][buildbotcustom](patch 2 of 2) allow to submit BuildSet with list of locales as a property and use it in MaemoBuildFactory (16.79 KB, patch)
2009-10-23 14:29 PDT, Armen Zambrano [:armenzg] - Engineering productivity
no flags Details | Diff | Splinter Review
(patch 1 of 2) use a MultiNightlyL10n scheduler (21.09 KB, patch)
2009-10-26 09:15 PDT, Armen Zambrano [:armenzg] - Engineering productivity
coop: review+
aki: review+
Details | Diff | Splinter Review
[buildbotcustom](patch 2 of 2) allow to submit BuildSet with list of locales as a property and use it in MaemoBuildFactory (17.44 KB, patch)
2009-10-26 09:16 PDT, Armen Zambrano [:armenzg] - Engineering productivity
coop: review+
aki: review+
Details | Diff | Splinter Review
(patch 1 of 2) use a MultiNightlyL10n scheduler (21.15 KB, patch)
2009-10-28 07:19 PDT, Armen Zambrano [:armenzg] - Engineering productivity
l10n: review+
Details | Diff | Splinter Review
[buildbotcustom](patch 2 of 2) allow to submit BuildSet with list of locales as a property and use it in MaemoBuildFactory (17.53 KB, patch)
2009-10-28 07:20 PDT, Armen Zambrano [:armenzg] - Engineering productivity
coop: review+
l10n: review+
coop: checked‑in+
Details | Diff | Splinter Review
[log] make chrome-da failed (17.07 KB, patch)
2009-10-28 09:38 PDT, Armen Zambrano [:armenzg] - Engineering productivity
no flags Details | Diff | Splinter Review
[buildbot-configs] (patch 1 of 2) enable multi-locale builds for Fennec on staging and on production (44.77 KB, patch)
2009-10-28 12:37 PDT, Armen Zambrano [:armenzg] - Engineering productivity
coop: review+
coop: review+
Details | Diff | Splinter Review

Description Axel Hecht [:Pike] 2009-09-30 05:23:35 PDT
The release and nightly builds for fennec on maemo are supposed to be multi-locale builds. The on-push builds probably shouldn't.

Bug 519682 will add the required makefile foo.

From a build automation side, there are some tricky things.

- the list of locales will be controled by a new file, similar to all-locales. I expect the name to be maemo-locales, living in locales inside mobile-browser.

- For each locale, there need to be at least three+ steps:
-- get the locale repository (1.9.2 or l10n central, depending on toolkit)
--- update locale to the right revision
--- update the build properties with the revision for that locale
-- compare-locales (with merge, on by default, switchable off for release)
-- make -C mobile/locales chrome-ab-CD

The make calls should happen in-between the compile and the package steps. Not sure where the source steps should be.

Nightly builds should warn on failure, release builds should abort-and-fail on failure.

The builds should report on Mozilla-l10 in addition to where they report now.

We should have TinderboxPrint for at least locale failures.
Comment 1 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-06 09:01:53 PDT
Here is how I generated it:
1) get en-US's linux gnueabi tar ball
2) get locale's linux gnueabi tar ball
3) untar en-US
4) untar locale.jar and locale.manifest

wget ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mobile-1.9.2/fennec-1.0b4.en-US.linux-gnueabi-arm.tar.bz2
wget ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mobile-1.9.2-l10n/fennec-1.0b4.es-ES.linux-gnueabi-arm.tar.bz2
tar xjf fennec-1.0b4.en-US.linux-gnueabi-arm.tar.bz2
tar xjf fennec-1.0b4.es-ES.linux-gnueabi-arm.tar.bz2 fennec/chrome/es-ES.jar fennec/chrome/es-ES.manifest
tar -cjf fennec-1.0b4.multi.linux-gnueabi-arm.tar.bz2 fennec
tar -cf fennec-1.0b4.multi.linux-gnueabi-arm.tar fennec

I copied the tar ball into the device and the multi-locale worked correctly except the bookmarks.

The infrastructure with this approach would look like this:
- Linux gnueabi arm tar ball is generated (normal en-US build as we curently do)
- The locales are triggered and generate their tar balls
- The multi-locale is generated after the locales have finished

(In reply to comment #0)
> - For each locale, there need to be at least three+ steps:
> -- get the locale repository (1.9.2 or l10n central, depending on toolkit)
> --- update locale to the right revision
> --- update the build properties with the revision for that locale
> -- compare-locales (with merge, on by default, switchable off for release)
> -- make -C mobile/locales chrome-ab-CD
> 
> The make calls should happen in-between the compile and the package steps. Not
> sure where the source steps should be.
This was my initial thought until I came up with this other way of doing it (thanks mfinkle). If I am understanding correctly the way you are mentioning removes the parallelization and ties everything to one slave.

Do you think the way that I described would work?

I also thought of triggering this multi-locale builder after the last locale finishes. I have not yet figure out how to do it but for the time being we could have a normal nightly scheduler an hour later than the locales are normally finished.
Comment 2 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-06 14:34:47 PDT
Created attachment 404920 [details]
bash script that downloads the en-US Fennec builds plus all the locales specified in mobile-browser/locales/all-locales and it generates a Fennec multi-locale build
Comment 3 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-06 14:50:37 PDT
Comment on attachment 404920 [details]
bash script that downloads the en-US Fennec builds plus all the locales specified in mobile-browser/locales/all-locales and it generates a Fennec multi-locale build

This script generates a multi-locale build of Fennec and it is currently set up in my people account with a cronjob at 3AM PDT.
http://people.mozilla.com/~armenzg/fennec/
The script takes around 3 minutes with the current number of locales

The locales that were successfully added are:
 ar be ca cs de es-AR es-ES fa fr fy-NL
 gl he id it ja ja-JP-mac ko nl pl pt-BR
 ru sk sl tr vi zh-CN 

The benefit of this method is not having to checkout anything and we take advantage of the locale tar balls that have been generated in parallel.

The problems I currently have are:
* The version of Fennec cannot be easily (trivial way) obtained as checking a file like version.txt on Firefox. The version can be obtained from this file http://mxr.mozilla.org/mobile-browser/source/confvars.sh#43 which contains the version for WinMo and Maemo targets
* I don't know a way of figuring out when all locales have finished being processed. Maybe a hack on buildbot to trigger other builder when the queue is emptied?

TODO:
This script should not allow locales that were not created after the en-US tar ball to be added. A date comparison should do.

Planning:
I believe the best approach for me is to work a little with the patches in bug 519682 until I get some feedback from any of you.
Comment 4 Mark Finkle (:mfinkle) (use needinfo?) 2009-10-06 18:26:36 PDT
(In reply to comment #3)

> * The version of Fennec cannot be easily (trivial way) obtained as checking a
> file like version.txt on Firefox. The version can be obtained from this file
> http://mxr.mozilla.org/mobile-browser/source/confvars.sh#43 which contains the
> version for WinMo and Maemo targets

fennec/application.ini has the version of release. It might be easier to parse than confvars.sh
Comment 5 Axel Hecht [:Pike] 2009-10-07 07:26:45 PDT
Is there any sense in me commenting? You basically did everything counter the spec in comment 0.

Parallizing multi-locale builds is per-se not possible.

Relying on tar balls on ftp makes us rely on something with rando-failures, and with no infra to figure them out.

Yeah, right, triggering on multiple builds.

Tarballs may go away.

Using tarballs will make doing the same for Firefox on WinCE just harder.

... and on and on.
Comment 6 Mark Finkle (:mfinkle) (use needinfo?) 2009-10-07 07:48:57 PDT
(In reply to comment #5)
> Is there any sense in me commenting? You basically did everything counter the
> spec in comment 0.

It always makes sense to comment on the progress :)

In this case Armen (guided perhaps by my suggestions) started working on the simplest thing that could work. It's certainly not the ideal system, but it's functional.

We need to start focusing on making parts of it better - more inline with what you proposed in comment 0.
Comment 7 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-07 07:50:39 PDT
(In reply to comment #5)
> Is there any sense in me commenting? You basically did everything counter the
> spec in comment 0.
> 
I was trying to understand the structure of what a multi-locale build was more than anything else and this was the faster way to reach it.

I think the multi-locale build that I am producing (even though might look scary or prone to errors) it gets us going temporally.

> Using tarballs will make doing the same for Firefox on WinCE just harder.
> 
> ... and on and on.
I am afraid of seeing us go back to do everything in one slave for Firefox but it might be worth reevaluating the time cost increase for dropping scalability and be able to evaluate in equal conditions.
Comment 8 Axel Hecht [:Pike] 2009-10-08 10:11:29 PDT
The chrome-% targets have landed, so we can now move over to use those.

Details:

- the l10n repos are smaller than fennec tarballs
- unpacking the tarball and extracting the right info won't be really cheaper than packing l10n
- again, at some point this will need to work for wince
- the initial comment explicitly calls out to *not* use all-locales, but maemo-locales. That file is not gonna scale as bad as all-locales, from what we know now.
Comment 9 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-08 11:06:03 PDT
(In reply to comment #8)
> The chrome-% targets have landed, so we can now move over to use those.
> 
> Details:
> 
> - the l10n repos are smaller than fennec tarballs
> - unpacking the tarball and extracting the right info won't be really cheaper
> than packing l10n
> - again, at some point this will need to work for wince
> - the initial comment explicitly calls out to *not* use all-locales, but
> maemo-locales. That file is not gonna scale as bad as all-locales, from what we
> know now.
I will be tackling this either tomorrow or on Tuesday and hope to have more to report at that time.
Good to see those landed.
Comment 10 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-13 14:34:41 PDT
Created attachment 406097 [details]
[log] after applying patch I still can't do "make chrome-%"

I tried to build with the patch just attached to the dependent bug and what happens is:
- if I use --with-l10n-base=.. it builds but when I try to create the chrome for the locale it happens what the log shows
- if I use --with-l10n-base="../l10n-1.9.2" (where the locales are) I get this message:
 configure: error: Invalid value --with-l10n-base, ../l10n-1.9.2 doesn't exist
Comment 11 Axel Hecht [:Pike] 2009-10-13 14:41:50 PDT
Specifying a full path should work at least.

I bet that it's somewhat bound to the multi-project build you're doing. The path you specify needs to be found when running configure, so you're somewhere deep in the build dir when you go to ../l10n-1.9.2. So that path needs to be relative to the obj dir and not the src dir. Not really that well documented, but that's how the code works.
Comment 12 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-14 08:25:44 PDT
That is (In reply to comment #11)
> I bet that it's somewhat bound to the multi-project build you're doing. The
> path you specify needs to be found when running configure, so you're somewhere
> deep in the build dir when you go to ../l10n-1.9.2. So that path needs to be
> relative to the obj dir and not the src dir. Not really that well documented,
> but that's how the code works.
relative to the objdir + mobile project did the work (../../../l10n-1.9.2 since mozilla-1.9.2/objdir/mobile)

I have created a multi-locale the way you mentioned containing fr-FR and es-ES: 
http://people.mozilla.com/~armenzg/fennec/fennec-1.0b5pre.en-US.linux-gnueabi-arm.tar.bz2

I switched to these devices languages/regions and worked smoothly:
- French (France)
- French (Canada)
- Spanish (Spain)
- Spanish (Latin America)
For Canada and Latin America it was able to fall back into the right language. Good job!
Comment 13 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-14 08:39:52 PDT
(In reply to comment #12)
> I have created a multi-locale the way you mentioned containing fr-FR and es-ES: 
Correction: There is not such a thing as fr-FR but fr.
Comment 14 Axel Hecht [:Pike] 2009-10-14 11:33:01 PDT
fwiw, we have 3 latin american spanish variants plus spain, in particular for Spanish we need to know precisely what we're up to.

I'm not really understanding the chrome registry hash and provider and whatnot, I don't know how reproducable the results are if we're not having the exact match for nokia's "latin america" in our offering.
Comment 15 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-14 13:29:24 PDT
Created attachment 406278 [details]
[WIP] this shows how to generate a multi-locale build for Fennec with all the steps

Tomorrow I will work on making it happen within buildbot.

For nightly builds, I have the question if we are going to rename to "multi" instead of "en-US"

I have few questions with regards what we will do for releases.
- Are we going to use for Beta 5 the same release process in which we pull a nightly build, push it to the chinook repo and sign it? If we do the release will be l10n-merged since nightly builds will be l10n-merged.
- I assume that it makes sense to have an l10n-changesets file that will contain the revisions that we are going to use for each locale. This conflicts with previous question since for nightly builds we would be pulling "tip"
- What about tagging? Are we going to pull a specific tag from m192? Are we going to tag mobile-browser?
Comment 16 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-14 13:33:05 PDT
(In reply to comment #8)
> - again, at some point this will need to work for wince
do you refer to Fennec for Wince or Firefox for Wince? are you referring to the WinMo build that is being provided through a cab file and that can as well be used on a WinCe platform?
Comment 17 Axel Hecht [:Pike] 2009-10-14 14:04:15 PDT
Yes, I expect an l10n-changesets file to come into action for fennec, both for the multilocale build and the individual locale deliverables.

You should never pull "tip", of course, but "default".

The script looks good, I'd probably had made it loop over a list right away, but it's giving the right idea.

For windows, that's a regular Firefox, not a Fennec. But the overarching concept of how we build it should be similar.
Comment 18 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-14 14:08:42 PDT
Comment on attachment 406278 [details]
[WIP] this shows how to generate a multi-locale build for Fennec with all the steps

NOTE: This script is WIP and it is just to capture the process for creating a multi-locale. Use at your own discretion since it is not polished
Comment 19 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-14 14:44:47 PDT
(In reply to comment #17)
> Yes, I expect an l10n-changesets file to come into action for fennec, both for
> the multilocale build and the individual locale deliverables.
> 
OK. Things are starting to fall into place (at least in my head :) ).

Do we have a better idea of what deliverable (langpack, tar ball or deb) we are going to give for individual locales?

Can I assume that these individual locale deliverables do not have to happen at the same time as the Beta 5?
Comment 20 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-16 14:02:02 PDT
Created attachment 406774 [details] [diff] [review]
[WIP](patch 1 of 2) add --with-l10n-base to mozconfig and pass multiLocale and L10nRepoPath variables to the Maemo's nightly factory
Comment 21 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-16 14:04:02 PDT
Created attachment 406775 [details] [diff] [review]
[WIP][buildbotcustom](patch 2 of 2) before packaging we iterate through a list of locales to check them out, run compare-locales and add the chrome for the locale
Comment 22 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-16 14:05:25 PDT
Created attachment 406776 [details] [diff] [review]
[WIP][Approach 2] make target that takes from a file like maemo-locales which locales to add to a multi-locale build
Comment 23 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-16 14:24:16 PDT
Comment on attachment 406775 [details] [diff] [review]
[WIP][buildbotcustom](patch 2 of 2) before packaging we iterate through a list of locales to check them out, run compare-locales and add the chrome for the locale

>diff --git a/process/factory.py b/process/factory.py
>--- a/process/factory.py
>+++ b/process/factory.py
...
>         self.addPreCleanSteps()
>         self.addBaseRepoSteps()
>         self.getMozconfig()
>         self.addPreBuildSteps()
>         self.addBuildSteps()
>+        if multiLocale:
>+            self.l10nRepoPath = l10nRepoPath
>+            self.compareLocalesRepo = self.getRepository(compareLocalesRepoPath)
>+            self.compareLocalesTag = compareLocalesTag
>+            self.compareLocalesSetup()
>+            for locale in ['es-ES', 'fr']:
>+                self.checkOutLocale(locale)
>+                self.compareLocales(locale)
>+                self.addChromeLocale(locale)
>+            # TODO: list locales added
>         self.addPackageSteps()
I have added these two locales in this for loop for testing purposes but it shows us that once the master is reconfigured that is the only list of locales that it will process.
In the normal parallelizable way, in which a scheduler is triggered, we can make it process every time it is triggered the all-locales file and therefore always use the latest list of locales.
In this first approach, since, we are working with a factory we are stuck once reconfigured; that is why I am starting to work on my second approach, which is, to call a make target on mobile/locales to take care of adding the latest list of locales to the multi-locale build. See attachment 406776 [details] [diff] [review] to get an idea of what I am talking about.
Comment 24 Axel Hecht [:Pike] 2009-10-16 14:55:00 PDT
Can we move the async getting of the locales list into the scheduler instead and pass it to the factory as property of the build request?
Comment 25 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-21 09:25:26 PDT
Created attachment 407544 [details] [diff] [review]
[WIP] (patch 1 of 2) add --with-l10n-base to mozconfig and use a MultiNightlyL10n scheduler
Comment 26 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-21 09:28:00 PDT
Created attachment 407546 [details] [diff] [review]
 [WIP][buildbotcustom](patch 2 of 2) allow to submit BuildSet with list of locales as a property and use it in MaemoBuildFactory
Comment 27 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-21 09:32:34 PDT
Both patches have run correctly in my local environment. I will test it on staging and polish the TODO lines.

Axel (wrt IRC convo last night), the methods I added to MaemoBuildFactory can be easily be moved up the inheritance hierarchy but that is not my main concern at this point (mainly because I want multi-locale builds out rather than having to test many more scenarios). We can revisit this request later on when we want to port this to other factories/projects/scenarios. Sounds fair?

NOTE for patch 2 of 2: I have just noticed that I have attached a slightly old (few lines edited)
Comment 28 Axel Hecht [:Pike] 2009-10-21 12:20:38 PDT
I haven't found why the all-locales entries in config got removed. I expect those to stay for the single-locale builds and langpacks. Maybe they're obsolete, though, don't know.

PS: I'm wondering how the single-locale builds will do repacks. As the existing logic just strips en-US, and not "all of a multi-locale build" or something.
Comment 29 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-21 12:56:48 PDT
(In reply to comment #28)
> I haven't found why the all-locales entries in config got removed. I expect
> those to stay for the single-locale builds and langpacks. Maybe they're
> obsolete, though, don't know.
>
They are duplicated. If you look closely you can see that they are duplicated the same as all the others I removed:
 MOBILE_BRANCHES['mobile-1.9.2']['allLocalesFile'] = "locales/all-locales"
+MOBILE_BRANCHES['mobile-1.9.2']['maemoLocalesFile'] = "locales/maemo-locales"
...
-MOBILE_BRANCHES['mobile-1.9.2']['allLocalesFile'] = "locales/all-locales"

> 
> PS: I'm wondering how the single-locale builds will do repacks. As the existing
> logic just strips en-US, and not "all of a multi-locale build" or something.

The logic that I am following is:
- do the same we were doing until now (which is):
  * build
  * repackage en-US
  * upload en-US
  * trigger individual locales since the en-US is on FTP
- what we do different after we trigger individual locales:
  * since the multiLocale variables has been and it is true:
    - setup compare-locales
    - iterate over locales list
       1) compare-locales
       2) add chrome
    - repackage multi-locale (TODO change packaging name)
    - upload multi-locale (TODO do not upload xulrunner since was already uploaded)

Sorry, I should have given the explanation with the patches.
Comment 30 Axel Hecht [:Pike] 2009-10-21 13:58:35 PDT
Sounds good. Depending on what we do we probably want to move the en-US package out of the way, though, and use the "fennec" package name for just the multi-locale package.

mfinkle, any preference on what the actual name of the multi-locale package should be? Is that supposed to be branding dependent? Do we need to worry about what happens if someone has the multi-locale build installed and tries to overwrite with single-locale build? Multiple single-locale builds? Kinda reraises the question on whether we want a single repo with all locales, or a repo per locale, plus possibly a shared repo for the cross-locale xulrunner.

I wonder if we need to switch matchOS off for the single locale builds, I'm somewhat tempted to say "yes", as I somewhat expect to have en-US.jar in xulrunner, still.
Comment 31 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-22 18:09:10 PDT
Created attachment 407917 [details] [diff] [review]
 [WIP] (patch 1 of 2) add --with-l10n-base to mozconfig and use a MultiNightlyL10n scheduler

It differs with the previous patch that:
- it modifies base_workdir to be SBOX_HOME/build/%base_builddir% rather than just SBOX_HOME/build. This fixes the issue that under build/ we would check out a directory l10n/ which is used at the same time by different branches :S
- it passes baseBuildDir=pf['base_builddir'] so we can use the change mentioned just above
- it uses a MultiNightlyL10n only for Maemo nightly builders
Comment 32 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-22 18:12:24 PDT
Created attachment 407918 [details] [diff] [review]
[WIP][buildbotcustom](patch 2 of 2) allow to submit BuildSet with list of locales as a property and use it in MaemoBuildFactory

Differences with previous patch:
- it generate the list of locales sorted "locales = locales.keys()"
- it has a proper addMultiLocaleSteps section
- it makes use of self.baseBuildDir so we can do everything under build/baseBuildDir rather than just build/
Comment 33 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-22 18:22:55 PDT
These patches have worked on staging correctly.

Here is the link to the multi-locale:
http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mobile-trunk/fennec-1.0b5pre.en-US.linux-gnueabi-arm.tar.bz2
NOTE: The way the patch works the single-locale build has been overwritten by the multi-locale

Axel, Chris and Aki could you please have a look to these patches during your Friday? I will ask for formal review when I figure the remaining TODOs at the end of this comment but I would like to get your comments already so we save one run of reviews.

I just have to work on uploading the multi-locale package to the right place while we make a final decision on what the official name should be.

TODO:
- make sure that a commit for this to run on staging would not affect production
- figure out the upload scenario
- test that changes on MaemoBuildFactory do not affect the dependent builds of it
Comment 34 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-23 14:29:03 PDT
Created attachment 408112 [details] [diff] [review]
[WIP] (patch 1 of 2) add --with-l10n-base to mozconfig and use a MultiNightlyL10n scheduler
Comment 35 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-23 14:29:33 PDT
Created attachment 408114 [details] [diff] [review]
 [WIP][buildbotcustom](patch 2 of 2) allow to submit BuildSet with list of locales as a property and use it in MaemoBuildFactory
Comment 36 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-23 14:39:33 PDT
These patches are pretty much final. They solve the name collision of the packages problem.

Comments welcome before I submit them for final review.

fennec-1.0b5pre.multi.linux-gnueabi-arm.tar.bz2
xulrunner-1.9.3a1pre.multi.linux-gnueabi-arm.tar.bz2 <- This file is unnecessary

xulrunner_1.9.3a1pre-20091023090934_armel.deb
fennec_1.0b5pre_armel.deb <- This is a deb of the multi-locale not of the single-locale

xulrunner-1.9.3a1pre.en-US.linux-gnueabi-arm.tar.bz2
fennec-1.0b5pre.en-US.linux-gnueabi-arm.tar.bz2


My TODO list has pretty much been reduced to:
- make sure that a commit for this to run on staging would not affect
production
- one last fine tuning of packageGlob
- bake on staging
Comment 37 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-26 09:15:41 PDT
Created attachment 408397 [details] [diff] [review]
(patch 1 of 2) use a MultiNightlyL10n scheduler
Comment 38 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-26 09:16:49 PDT
Created attachment 408398 [details] [diff] [review]
[buildbotcustom](patch 2 of 2) allow to submit BuildSet with list of locales as a property and use it in MaemoBuildFactory
Comment 39 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-26 09:19:40 PDT
These patches have fixed in them few things I saw over the weekend plus dropping the mozconfig changes. The reason to do so is that the mozconfig is shared between the nightly and the dependent builders. What I do instead is to pass L10NBASEDIR to "make -f client.mk" which is equivalent to --with-l10n-base.

Requesting review from coop and aki for the first round of reviews.
Comment 40 Chris Cooper [:coop] 2009-10-27 14:14:33 PDT
Comment on attachment 408397 [details] [diff] [review]
(patch 1 of 2) use a MultiNightlyL10n scheduler

>diff --git a/mozilla2-staging/mobile_config.py b/mozilla2-staging/mobile_config.py
>--- a/mozilla2-staging/mobile_config.py
>+++ b/mozilla2-staging/mobile_config.py
>-MOBILE_BRANCHES['mobile-trunk']['platforms']['linux-gnueabi-arm']['base_workdir'] = '%s/build' % SBOX_HOME
>+MOBILE_BRANCHES['mobile-trunk']['platforms']['linux-gnueabi-arm']['base_workdir'] = '%s/build/maemo-trunk' % SBOX_HOME
> MOBILE_BRANCHES['mobile-trunk']['platforms']['linux-gnueabi-arm']['base_builddir'] = 'maemo-trunk'

Why the need to append 'maemo-trunk' to the base_workdir here? Isn't the intent that base_builddir cover that?

>-MOBILE_BRANCHES['mobile-1.9.2']['platforms']['linux-gnueabi-arm']['base_workdir'] = '%s/build' % SBOX_HOME
>+MOBILE_BRANCHES['mobile-1.9.2']['platforms']['linux-gnueabi-arm']['base_workdir'] = '%s/build/maemo-1.9.2' % SBOX_HOME
> MOBILE_BRANCHES['mobile-1.9.2']['platforms']['linux-gnueabi-arm']['base_builddir'] = 'maemo-1.9.2'

>+# This is repeated and I believe that the next one is the real one; test before removing

What is repeated? I don't understand this comment.

>-MOBILE_BRANCHES['mobile-1.9.2']['allLocalesFile'] = "locales/all-locales"
>-MOBILE_BRANCHES['mobile-1.9.2']['l10nUploadPath'] = \
>-    '/home/ftp/pub/mozilla.org/firefox/nightly/latest-mobile-1.9.2-l10n'
> MOBILE_BRANCHES['mobile-1.9.2']['enUS_binaryURL'] = \
>     MOBILE_BRANCHES['mobile-1.9.2']['main_config']['download_base_url'] + '/nightly/latest-mobile-1.9.2'
>-MOBILE_BRANCHES['mobile-1.9.2']['tinderbox_tree'] = 'MozillaTest'
>-MOBILE_BRANCHES['mobile-1.9.2']['l10n_tinderbox_tree'] = 'MozillaStaging'

Why these deletions do we not care about reporting to tinderbox anymore for these builds?

>-MOBILE_BRANCHES['mobile-tracemonkey']['platforms']['linux-gnueabi-arm']['base_workdir'] = '%s/build' % SBOX_HOME
>+MOBILE_BRANCHES['mobile-tracemonkey']['platforms']['linux-gnueabi-arm']['base_workdir'] = '%s/build/maemo-tracemonkey' % SBOX_HOME
> MOBILE_BRANCHES['mobile-tracemonkey']['platforms']['linux-gnueabi-arm']['base_builddir'] = 'maemo-tracemonkey'

Same as above re: base_builddir.

>-MOBILE_BRANCHES['mobile-electrolysis']['platforms']['linux-gnueabi-arm']['base_workdir'] = '%s/build' % SBOX_HOME
>+MOBILE_BRANCHES['mobile-electrolysis']['platforms']['linux-gnueabi-arm']['base_workdir'] = '%s/build/maemo-electrolysis' % SBOX_HOME
> MOBILE_BRANCHES['mobile-electrolysis']['platforms']['linux-gnueabi-arm']['base_builddir'] = 'maemo-electrolysis'

Same as above re: base_builddir.

>-                baseWorkDir=pf['base_workdir'],
>+                baseWorkDir=pf['base_workdir']+'-dep',
>+                baseBuildDir=pf['base_builddir']+'-dep',

I'm not a huge fan of adding the suffix at the last minute here. Anyway we can push the '-dep' addition back into the config files, or at least before the factory gets created officialy?
Comment 41 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-27 14:29:45 PDT
(In reply to comment #40)
> (From update of attachment 408397 [details] [diff] [review])
> >diff --git a/mozilla2-staging/mobile_config.py b/mozilla2-staging/mobile_config.py
> >--- a/mozilla2-staging/mobile_config.py
> >+++ b/mozilla2-staging/mobile_config.py
> >-MOBILE_BRANCHES['mobile-trunk']['platforms']['linux-gnueabi-arm']['base_workdir'] = '%s/build' % SBOX_HOME
> >+MOBILE_BRANCHES['mobile-trunk']['platforms']['linux-gnueabi-arm']['base_workdir'] = '%s/build/maemo-trunk' % SBOX_HOME
> > MOBILE_BRANCHES['mobile-trunk']['platforms']['linux-gnueabi-arm']['base_builddir'] = 'maemo-trunk'
> 
> Why the need to append 'maemo-trunk' to the base_workdir here? Isn't the intent
> that base_builddir cover that?
> 
The other option is to go around in factory.py where self.baseWorkDir is being used and append there self.baseBuildDir to it.

> >-MOBILE_BRANCHES['mobile-1.9.2']['platforms']['linux-gnueabi-arm']['base_workdir'] = '%s/build' % SBOX_HOME
> >+MOBILE_BRANCHES['mobile-1.9.2']['platforms']['linux-gnueabi-arm']['base_workdir'] = '%s/build/maemo-1.9.2' % SBOX_HOME
> > MOBILE_BRANCHES['mobile-1.9.2']['platforms']['linux-gnueabi-arm']['base_builddir'] = 'maemo-1.9.2'
> 
> >+# This is repeated and I believe that the next one is the real one; test before removing
> 
> What is repeated? I don't understand this comment.
>
The enUS_binaryURL is repeated but removing that line would have required me to retest the Maemo repackages scenario as well (which I can now)
 MOBILE_BRANCHES['mobile-1.9.2']['enUS_binaryURL'] = \
     config.BRANCHES['mozilla-central']['download_base_url'] + '/nightly/latest-mobile-1.9.2'
 ...
 MOBILE_BRANCHES['mobile-1.9.2']['enUS_binaryURL'] = \
     MOBILE_BRANCHES['mobile-1.9.2']['main_config']['download_base_url'] + '/nightly/latest-mobile-1.9.2'

> >-MOBILE_BRANCHES['mobile-1.9.2']['allLocalesFile'] = "locales/all-locales"
> >-MOBILE_BRANCHES['mobile-1.9.2']['l10nUploadPath'] = \
> >-    '/home/ftp/pub/mozilla.org/firefox/nightly/latest-mobile-1.9.2-l10n'
> > MOBILE_BRANCHES['mobile-1.9.2']['enUS_binaryURL'] = \
> >     MOBILE_BRANCHES['mobile-1.9.2']['main_config']['download_base_url'] + '/nightly/latest-mobile-1.9.2'
> >-MOBILE_BRANCHES['mobile-1.9.2']['tinderbox_tree'] = 'MozillaTest'
> >-MOBILE_BRANCHES['mobile-1.9.2']['l10n_tinderbox_tree'] = 'MozillaStaging'
> 
> Why these deletions do we not care about reporting to tinderbox anymore for
> these builds?
> 
If you look closely it is repeated as well
 MOBILE_BRANCHES['mobile-1.9.2']['tinderbox_tree'] = 'MozillaTest'
 MOBILE_BRANCHES['mobile-1.9.2']['l10n_tinderbox_tree'] = 'MozillaStaging'
-MOBILE_BRANCHES['mobile-1.9.2']['allLocalesFile'] = "locales/all-locales"
-MOBILE_BRANCHES['mobile-1.9.2']['l10nUploadPath'] = \
-    '/home/ftp/pub/mozilla.org/firefox/nightly/latest-mobile-1.9.2-l10n'
 MOBILE_BRANCHES['mobile-1.9.2']['enUS_binaryURL'] = \
     MOBILE_BRANCHES['mobile-1.9.2']['main_config']['download_base_url'] + '/nightly/latest-mobile-1.9.2'
-MOBILE_BRANCHES['mobile-1.9.2']['tinderbox_tree'] = 'MozillaTest'
-MOBILE_BRANCHES['mobile-1.9.2']['l10n_tinderbox_tree'] = 'MozillaStaging'

> 
> >-                baseWorkDir=pf['base_workdir'],
> >+                baseWorkDir=pf['base_workdir']+'-dep',
> >+                baseBuildDir=pf['base_builddir']+'-dep',
> 
> I'm not a huge fan of adding the suffix at the last minute here. Anyway we can
> push the '-dep' addition back into the config files, or at least before the
> factory gets created officialy?
Me neither. I can do this inside the factory.
Comment 42 Chris Cooper [:coop] 2009-10-27 14:30:36 PDT
Comment on attachment 408398 [details] [diff] [review]
[buildbotcustom](patch 2 of 2) allow to submit BuildSet with list of locales as a property and use it in MaemoBuildFactory

>+        if self.multiLocale:
>+            self.addStep(ShellCommand,
>+                name='create_dir_l10n',
>+                command=['mkdir', '-p', 'l10n'],
>+                workdir='%s' % self.baseWorkDir,
>+                description=['create', 'l10n', 'dir']
>+            )
>+            self.addBuildSteps(extraEnv="L10NBASEDIR='../../../l10n'")

As we learned in bug 524010, shouldn't we use a more descriptive name for the multi-locale dir than simply 'l10n?' Too easy to overload it and miss something otherwise. Or should this be replaced by base_l10n_workdir?

>+            if self.multiLocale:
>+                self.addStep(ShellCommand,
>+                    name='clobber_l10n_dir',
>+                    command=['rm', '-rf', 'l10n'],
>+                    env=self.env,
>+                    workdir=self.baseWorkDir,
>+                    timeout=10*60
>+                )

Need to change this too if we change the above.
Comment 43 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-27 14:32:11 PDT
Note: I am doing another run on staging without any config patches applied to see if it would break anything in production. I will attach the third patch to address this issue.
Comment 44 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-27 14:33:41 PDT
(In reply to comment #42)
> (From update of attachment 408398 [details] [diff] [review])
> >+        if self.multiLocale:
> >+            self.addStep(ShellCommand,
> >+                name='create_dir_l10n',
> >+                command=['mkdir', '-p', 'l10n'],
> >+                workdir='%s' % self.baseWorkDir,
> >+                description=['create', 'l10n', 'dir']
> >+            )
> >+            self.addBuildSteps(extraEnv="L10NBASEDIR='../../../l10n'")
> 
> As we learned in bug 524010, shouldn't we use a more descriptive name for the
> multi-locale dir than simply 'l10n?' Too easy to overload it and miss something
> otherwise. Or should this be replaced by base_l10n_workdir?
> 
Now that I am not using the hardcoded line on the mozconfig this can be done. I will add this.

> >+            if self.multiLocale:
> >+                self.addStep(ShellCommand,
> >+                    name='clobber_l10n_dir',
> >+                    command=['rm', '-rf', 'l10n'],
> >+                    env=self.env,
> >+                    workdir=self.baseWorkDir,
> >+                    timeout=10*60
> >+                )
> 
> Need to change this too if we change the above.

Will do
Comment 45 Chris Cooper [:coop] 2009-10-27 14:59:19 PDT
(In reply to comment #41)
> The other option is to go around in factory.py where self.baseWorkDir is being
> used and append there self.baseBuildDir to it.

Ugh. Yes, as I discovered in my updates work, making changes like this are often more invasive than I originally thought. Perhaps a follow-up bug to clean this part up then?

> > >-                baseWorkDir=pf['base_workdir'],
> > >+                baseWorkDir=pf['base_workdir']+'-dep',
> > >+                baseBuildDir=pf['base_builddir']+'-dep',
> > 
> > I'm not a huge fan of adding the suffix at the last minute here. Anyway we can
> > push the '-dep' addition back into the config files, or at least before the
> > factory gets created officialy?
> Me neither. I can do this inside the factory.

I was suggesting that you create a variable to store the compound variable *before* you create the factory. Not a huge deal, I just prefer concatenate the var as we instantiate.
Comment 46 Axel Hecht [:Pike] 2009-10-28 05:44:40 PDT
Comment on attachment 408397 [details] [diff] [review]
(patch 1 of 2) use a MultiNightlyL10n scheduler

Two comments on this one:

- the config variable for the multi-locale list shouldn't be called maemoLocalesFile, IMHO

- l10nNightlyBuilders[builder]['platform'] in ('linux-gnueabi-arm'):

('linux-gnueabi-arm') isn't a list, ('l..',) is.

- Should we do multi-locale builds for the desktop linux builds right away, too?
Comment 47 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-28 06:06:10 PDT
(In reply to comment #46)
> - Should we do multi-locale builds for the desktop linux builds right away,
> too?
We totally could but I believe that this is yours and mobile's call. We could do this in the next iteration after these patches go in. I don't want to add more things in the work of this bug.
Comment 48 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-28 07:19:03 PDT
Created attachment 408839 [details] [diff] [review]
(patch 1 of 2) use a MultiNightlyL10n scheduler
Comment 49 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-28 07:20:06 PDT
Created attachment 408840 [details] [diff] [review]
[buildbotcustom](patch 2 of 2) allow to submit BuildSet with list of locales as a property and use it in MaemoBuildFactory
Comment 50 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-28 07:24:06 PDT
Carrying forward Aki's review and addressed Chris' and Axel's comments.

Chris, I am using self.l10nRepoPath as we use in other factories. I am using two variables for the nightly builder to have '-nightly' instead of the dependent ones (the same naming as other Firefox builders)

Axel, I called the variable multiLocalesFile

As Chris mentioned we can follow on modifying self.baseWorkDir and self.baseBuildDir clean up in another bug

All I have left is to add the patch for production (we can't land before I get this out)
Comment 51 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-28 08:36:07 PDT
Comment on attachment 408839 [details] [diff] [review]
(patch 1 of 2) use a MultiNightlyL10n scheduler

I am going to attach this patch with the production one at the same time. We will carry forward Axel's review.
Comment 52 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-28 09:38:53 PDT
Created attachment 408858 [details] [diff] [review]
[log] make chrome-da failed

I normally see that the compare-locale fails but I made the build go orange. I didn't think too much about if the build should go red or orange if the make chrome step fails.

Nevertheless the build went all the way and uploaded the files.
Funny enough I can see "da" locale in the multi locale build.
http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mobile-trunk/fennec-1.0b5pre.multi.linux-gnueabi-arm.tar.bz2

Axel can you please look into this?
Comment 53 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-28 12:37:41 PDT
Created attachment 408899 [details] [diff] [review]
[buildbot-configs] (patch 1 of 2) enable multi-locale builds for Fennec on staging and on production

This has run to completion for both branches on staging-master2.

This is the last patch I am attaching.

We will keep it running overnight to see that everything goes as expected.
Comment 54 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-29 07:04:31 PDT
Everything was green until one of the slaves run out of space and was not able to remove enough space since anything under scratchbox falls in a dead zone of disk space. Taking bug 524820 to be sure we want hit this problem on production.
Comment 55 Chris Cooper [:coop] 2009-10-29 08:46:24 PDT
Comment on attachment 408840 [details] [diff] [review]
[buildbotcustom](patch 2 of 2) allow to submit BuildSet with list of locales as a property and use it in MaemoBuildFactory

http://hg.mozilla.org/build/buildbotcustom/rev/20342354dc87
Comment 56 Chris Cooper [:coop] 2009-10-29 08:47:09 PDT
Comment on attachment 408899 [details] [diff] [review]
[buildbot-configs] (patch 1 of 2) enable multi-locale builds for Fennec on staging and on production

http://hg.mozilla.org/build/buildbot-configs/rev/995bdd87be29
Comment 57 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-29 09:38:25 PDT
Changes has been landed and masters reconfigured.
We should have a multi-locale build for Maemo tomorrow in the morning.

I am going to proceed in a couple of hours (when I see all the current mobile builds) to do a manual clean up of the slaves where they used to build before.

Meanwhile I will keep on working on the purge_builds.py patch.
Comment 58 Axel Hecht [:Pike] 2009-10-29 11:01:38 PDT
This apparently broke the single-locale tarballs, they're trying to download the en-US bz2, but only the -multi- is on ftp. The fennec deb is more or less empty, but that could be all kinds of things right now, given the respin of a respin of a respin.
Comment 59 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-29 11:19:37 PDT
(In reply to comment #58)
> This apparently broke the single-locale tarballs, they're trying to download
> the en-US bz2, but only the -multi- is on ftp. The fennec deb is more or less
> empty, but that could be all kinds of things right now, given the respin of a
> respin of a respin.

This is what it tried to download:
fennec-1.0b5.en-US.linux-gnueabi-arm.tar.bz2
and this is what it is on ftp:
fennec-1.0b5pre.en-US.linux-gnueabi-arm.tar.bz2

http://mxr.mozilla.org/mobile-browser/source/confvars.sh#48

More to investigate.
Comment 60 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-29 11:28:13 PDT
Awesome.

Version bump:
Thu Oct 29 13:41:27 2009 -0400 (at Thu Oct 29 13:41:27 2009 -0400)
http://hg.mozilla.org/mobile-browser/rev/2b0927e6322c

The nightly happened before the version bump. That is why some of the locales worked before 13:41 PDT.


Besides that, what are we going to do for chrome-da? Not to turn the build red?
The returning code is fail but the actual jar gets generated and added to the multi-locale build.
Comment 61 Axel Hecht [:Pike] 2009-10-29 11:33:46 PDT
The fail is right, because the jar is incomplete. I.e., as a deliverable with N locales, that thing didn't build right.

That there's stuff on ftp after that and that there's even danish stuff in there is OK.

If we're really worried about the danish stuff in there, and we might, we could create a step to be run after chrome-ab-CD that calls a clobber target for ab-CD if the previous step failed. That wouldn't change the result of the total build, though, IMHO.
Comment 62 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-29 13:01:56 PDT
Posted on the mailing list:
http://groups.google.com/group/mozilla.dev.platforms.mobile/browse_frm/thread/257ab891c4d4d484#
but might be worth to have it in here as well.

Hello all,
As of today we are going to be generating with the nightly Maemo build the multi-locale version of it.
You will currently see the nightly logs *RED* but if you look closely you will notice that one of the locales has to come up to date before the build can go green. The multi-locale should still be usable; The en-US build is independent from the multi-locale even though it is created in the same build machine.

These are the multi-locale's deb files:
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mobile-1.9.2/fennec_1.0b5pre_armel.deb
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mobile-1.9.2/xulrunner_1.9.2b2pre-20091029095850_armel.deb
These are the tar balls for it:
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mobile-1.9.2/fennec-1.0b5pre.multi.linux-gnueabi-arm.tar.bz2

NOTE: The only filename that contains the word "multi" is Fennec's tar ball. XulRunner tar and deb files are the same for both the single locale and the multi locale.
NOTE2: There is *no* deb file for the single locale nightly build anymore. There will only be the tar ball.

This is the error you will see due to the locale's problem:
RuntimeError: file not found: answers.xml
make[1]: *** [searchplugins] Error 1
make[1]: Leaving directory `/home/cltbld/build/maemo-1.9.2-nightly/mozilla-1.9.2/objdir/mobile/mobile/locales'
make: *** [chrome-da] Error 2

Here is the bug where the multi-locale has been enabled:
https://bugzilla.mozilla.org/show_bug.cgi?id=519684

There is going to be some follow up work and might change things a little.
Comment 63 Aki Sasaki [:aki] 2009-10-29 13:09:17 PDT
Would it be difficult to make the repack a different builder, so the en-US nightly can go green?
Comment 64 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-29 13:33:01 PDT
(In reply to comment #63)
> Would it be difficult to make the repack a different builder, so the en-US
> nightly can go green?
>
That is my plan but still wouldn't it make sense to fix that locale so the multi-locale can contain what is really wanted.

This was not a good decision I took but we can still fix it.
Comment 65 Axel Hecht [:Pike] 2009-10-29 13:36:02 PDT
I landed a bustage fix on 1.9.2, http://hg.mozilla.org/releases/l10n-mozilla-1.9.2/da/rev/69d0ff6ecfa6, for bug 521984.

Due to the nature of l10n on central, I don't intend to land that change on l10n-central, as that's gonna make branching just harder.

Interesting question on whether we should switch the multi-locale build on on central in the first place, now that I come to think of it.
Comment 66 Armen Zambrano [:armenzg] - Engineering productivity 2009-10-29 14:51:30 PDT
(In reply to comment #63)
> Would it be difficult to make the repack a different builder, so the en-US
> nightly can go green?
>
Bug 525327 filed. It can be taken from my hands if needed since I won't be around until Monday.
Comment 67 Chris Cooper [:coop] 2009-10-30 09:18:09 PDT
[12:01pm] mfinkle: during a multi-locale build we are attempting to repackage the deb, but we are using a bad bz2 archive as a source
[12:01pm] mfinkle: http://tinderbox.mozilla.org/showlog.cgi?log=Mobile/1256889604.1256892737.7178.gz&fulltext=1  (orange nightly log)
[12:02pm] mfinkle: search for "fennec_1.0b5_armel.deb"
[12:02pm] mfinkle: you'll see a 556KB file uploaded
[12:02pm] mfinkle: and a 24KB file
[12:02pm] mfinkle: the 24KB deb is empty
[12:03pm] mfinkle: we try to make the deb from a bz2 file that no longer exists
[12:04pm] mfinkle: bunzip2: Can't open input file ../../dist/fennec-1.0b5.en-US.linux-gnueabi-arm.tar.bz2: No such file or directory.
[12:04pm] mfinkle: (that is the second "mobile make deb")
[12:04pm] mfinkle: at that point the bz2 is called ...
[12:04pm] mfinkle: cd ../../dist && tar -c --owner=0 --group=0 --numeric-owner --mode="go-w" -f - fennec | bzip2 -vf > fennec-1.0b5.multi.linux-gnueabi-arm.tar.bz2
[12:04pm] mfinkle: fennec-1.0b5.multi.linux-gnueabi-arm.tar.bz2
[12:06pm] coop: so it needs to be calling the multi deb instead?
[12:09pm] mfinkle: coop: yeah, but the | make deb | code expects a specificaly named b2
[12:10pm] mfinkle: coop: so it tries to use the bz2 that must have been removed earlier
[12:10pm] mfinkle: coop: maybe a simple fix is to cp the multi locale bz2 to the en-US name
[12:11pm] mfinkle: right before the | make deb |
[12:11pm] coop: mfinkle: yeah, let me try that
[12:12pm] mfinkle: coop: just for sanity, we could rm the file after the deb script completes
[12:13pm] coop: mfinkle: right
Comment 68 Chris Cooper [:coop] 2009-10-30 09:19:54 PDT
(In reply to comment #67)
> [12:01pm] mfinkle: during a multi-locale build we are attempting to repackage
> the deb, but we are using a bad bz2 archive as a source

Filed as bug 525470
Comment 69 Armen Zambrano [:armenzg] - Engineering productivity 2009-11-02 08:13:46 PST
The release work is tracked on bug 525257
Comment 70 Axel Hecht [:Pike] 2009-12-18 12:07:16 PST
I think this can be closed now.
Comment 71 Armen Zambrano [:armenzg] - Engineering productivity 2009-12-18 12:39:59 PST
yay!

Note You need to log in before you can comment on or make changes to this bug.