We need tinderbox and nightly build coverage of localized aviary firefoxen (and later aviary thunderbirds). I have written a post-mozilla script that piggybacks on a regular tinderbox build to test all the available localizations, see http://lxr.mozilla.org/mozilla/source/tools/tinderbox/post-mozilla-l10n.pl and my person machine is running this for the moment, see http://tinderbox.mozilla.org/showbuilds.cgi?tree=MozillaTest This is not really tested with win32/mac, but it should work with few modifications. This most likely needs to be integrated with the post-mozilla-rel script, so that we can stage the localized builds after we build/test them. I currently don't have stage access, so I would be coding in the dark trying to merge the two. It would also help to land bug 257177 before/with this change. I think that patch should land as-is, and the "pretty names" should be managed with symlinks, either created by post-mozilla-rel, or by the stage server itself. Bryner said to contact granrose about this... it is a fairly urgent priority for the pending l10n freeze.
Ok, have the new linux build system set up (chase is working on the win32), and have started hacking post-mozilla-rel.pl to include post-mozilla-l10n. Question: how many products are we doing this for? We're going to need at least one more tinderbox page and I'm wondering which setup is best: - one for all localizations of all products (i.e. Mozilla-l10n) - one per product and branch (i.e. Firefox-branch-l10n, Firefox-trunk-l10n, Thunderbird-branch-l10n, etc.). There are 17 locales for firefox branch, so for all three platforms, all locales, we're talking 54 columns for one product (1 full build + 17 localized * 3 platforms). I'm planning on setting up the latter but wanted to give a chance for feedback.
Status: NEW → ASSIGNED
Priority: -- → P2
> Question: how many products are we doing this for? We're going to need at least Firefox and Thunderbird, in the near term. Towards 1.8 xulrunner will want to be in on the game, as might sunbird. > - one per product and branch (i.e. Firefox-branch-l10n, Firefox-trunk-l10n, > Thunderbird-branch-l10n, etc.). This sounds much more manageable to me. > There are 17 locales for firefox branch, so for all three platforms, all Expect at least 10 more by the time we hit 1.0; the ultimate goal is 50+ --BDS
50+ is a lot of machines! Are we round-robin-ing the builds at all? Maybe one build could do Spanish-French-Japanese-German and cycle on that, print the language out for tinderbox so you know what failed when it does. I agree with Benjamin, you want the l10n status to follow the product around, dumping them all into a single l10n page is going to get noisy.
Just one question: Can we expect official localised (german) builds for PR 2/Beta1 on 10/18? I need to know that, 'cause the answer will influence my decision whether I should release PR 1 now, or wait for PR2/Beta1.
hmm. 50+ locales makes for a lot of builds (151 for each product). The tinderbox-page-per-product-branch idea might not scale well. We'll tackle that when it shows up, I guess, but may have to go to a locale-centric scheme where all the products for a specific locale report to a specific page, e.g. Mozilla-l10n-ruRU with firefox branch, firefox trunk, tbird branch, etc. all reporting to that page to keep it manageable.
Locale-centric scheme would be definetely preferable by l10n teams, because it is more convenient for us to check one page whether all our ab-CD builds are fine than searching all tinderbox-per-product-pages.
for the record: need this to pull and package the l10n files mk_add_options MOZ_CO_LOCALES=all and if you're not using the cvs-mirror, you need a line like mk_add_options LOCALES_CVSROOT=:ext:email@example.com:/l10n
Ok, so for the tinderbox pages, it sounds like locale-specific pages is the preferred way, so any product, any platform that does es-AR (for example) will send all tinderbox logs to Mozilla-l10n-es-AR. There will also be a Mozilla-l10n page that the systems producing the l10n builds can send their full build log to so it doesn't take up space on the primary product page. The linux builds completed, and the win32 is underway. I see 17 firefox*langpack.xpi files, but only one set of installers saved that includes the locale and that's the last locale that was run (he-IL). Time to push bits, so more questions: what files are we supposed to be delivering to the ftp server for nightlies or for a final release? Are we shipping a langpack, tarball and blob installer for each locale? What should the files be named? Currently on linux we have firefox.0.10.ab-CD.langpack.xpi (17 of these) firefox-i686-linux-gtk2+xft.tar.gz firefox-i686-linux-gtk2+xft-installer.tar.gz If we're pushing blob installers and tarballs, where does the locale go in the filename? firefox-i686-linux-gtk2+xft-ab-CD.tar.gz? Do we need to push the langpacks from every platform? If so, do we need to incorporate platform information in the langpack filenames? Should we push langpacks/installers when they return busted or testfailed? I'm assuming no. When are we getting more disk space (stage is almost full)? To push a single product, single platform for 17 locales of langpacks, blob installers and tarballs takes about 244MB (17*7.2MB*2 + 1.7MB) for the 2004-10-13-xx-0.9 directory, plus about the same for latest-0.9, so 480MB. Add two more platforms and that's 1.4GB / day. We currently have 1.6GB free on stage, so this is a blocker unless we're only shipping langpacks. Dropping tarballs cuts that in half, but 800MB per day is a big hit. Speaking of which, do we have to worry about the mirror sites and the bandwidth for mirroring this additional data?
The patch to make the filenames sane is bug 257117. It needs the existing staging tinderboxen to update their config files when it lands. We do not need to push langpacks from every platform. I don't know what to do about disk space.
FWIW, I'm still highly uncomfortable with the idea of using separate machines (rather than a step at the end of the existing builds) for the localized builds. It could mean having differences in configuration between the English build and all the other builds, and it doesn't scale well to lots of products and branches.
Summary of Windows l10n errors based on the chroma errors found on http://tinderbox.mozilla.org/showbuilds.cgi?tree=MozillaTest: cs-CZ: COMPLETE_RESET_HOMEPAGE * da-DK: mozilla/toolkit/locales/da-DK/installer/windows/charset.mk missing de-DE: iconv: conversion to CP1252 unsupported * es-AR: mozilla/browser/locales/es-AR/installer/installer.inc missing he-IL: COMPLETE_RESET_HOMEPAGE hu-HU: iconv: conversion to CP1250 unsupported nb-NO: iconv: conversion to CP1252 unsupported pl-PL: mozilla/browser/locales/pl-PL/installer/installer.inc: can't convert pt-BR: COMPLETE_RESET_HOMEPAGE zh-CN: COMPLETE_RESET_HOMEPAGE The starred errors are also affecting karma and bsmedberg's builds. They are specific to the build system as stored and once fixed for one system should be fixed for all systems. The COMPLETE_RESET_HOMEPAGE error looks the same across the four locales that exhibit it. Here's an instance of the error from the cs-CZ locale: /cygdrive/c/builds/tinderbox/firefox-aviarybranch-l10n/WINNT_5.1_Clobber/mozilla/config/preprocessor.pl:/cygdrive/c/builds/tinderbox/firefox-aviarybranch-l10n/WINNT_5.1_Clobber/mozilla/browser/installer/windows/config.it:394: error using substitution: variable 'COMPLETE_RESET_HOMEPAGE' is not defined The iconv unsupported encoding errors suggest that chroma needs a newer version of iconv or one of its supporting libraries. bsmedberg, have you tested the post-mozilla-l10n modifications on Windows? If so did you need to make any special modifications to add support for the CP1250 and CP1252 character sets?
Re: the CP1250 and CP1252 conversion errors, I think the problem is that in some locales WIN_INSTALLER_CHARSET is getting set to "CP1250\n" and "CP1252\n". On the fi-FI locale, CP1252 is used and the iconv command exits succesfully. The log for fi-FI is at http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1097705640.16638.gz&fulltext=1. A failing CP1252 example can be seen in http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1097705640.15274.gz&fulltext=1. The main difference between the successful log file and the failing log file is that there is a newline after CP1252 in the failing log file. Additionally, 'iconv -l' on chroma includes CP1250 and CP1252 in its list of supported character sets.
The following three files have incorrect line endings which are causing the build failures for their localizations: mozilla/toolkit/locales/de-DE/installer/windows/charset.mk mozilla/toolkit/locales/hu-HU/installer/windows/charset.mk mozilla/toolkit/locales/nb-NO/installer/windows/charset.mk An example of a working charset.mk can be found at: mozilla/toolkit/locales/fi-FI/installer/windows/charset.mk Viewing the files in 'vi' or 'vim' will show the improperly places ^Ms at the end of each line. Each of the non-working files mentioned above should be updated to use Unix line endings. I will spin this out into a separate bug for tracking purposes.
checked in a fix to de-DE/installer/windows/charset.mk for a.topal that I had in my tree locally.
(In reply to comment #14) > Viewing the files in 'vi' or 'vim' will show the improperly places ^Ms at the > end of each line. Each of the non-working files mentioned above should be > updated to use Unix line endings. I will spin this out into a separate bug for > tracking purposes. CVS does the conversion for line endings of text files automatically. It is better to say that all text files should have line endings which match to that of the operating system. That is "0D 0A" for Windows (using WinCVS) and "0A" for unix like operating systems (including Cygwin on Windows if Unix like line endings have been chosen during Cygwin install). It is wrong to recommend updating these files to use Unix line endings. By the way, hu-HU have been fixed.
(In reply to comment #16) > > CVS does the conversion for line endings of text files automatically. It is > better to say that all text files should have line endings which match to that > of the operating system. That is "0D 0A" for Windows (using WinCVS) and "0A" for > unix like operating systems (including Cygwin on Windows if Unix like line > endings have been chosen during Cygwin install). It is wrong to recommend > updating these files to use Unix line endings. > > By the way, hu-HU have been fixed. Please update bug 264278 mentioning your fix. Thanks.
http://tinderbox.mozilla.org/showbuilds.cgi?tree=MozillaTest does not include any swedish (sv-SE) build. I'd like to request the addition of such builds. Is there anything I need to do in order to make this possible?
(In reply to comment #18) > http://tinderbox.mozilla.org/showbuilds.cgi?tree=MozillaTest does not include > any swedish (sv-SE) build. I'd like to request the addition of such builds. Is > there anything I need to do in order to make this possible? Checked in fixes to browser|toolkit/locales/all-locales. sv-SE is red, though, http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1097763420.13140.gz Both chroma and bdsmedberg boxes picked it up.
new standardized file names look good. we need to integrate the en-US and l10n installer builds though. Currently we end up with the langpacks and l10n tarballs in mozilla/dist and only the installer builds for en-US and whichever locale finished last. It would be better if the langpacks, tarballs and the xpi directories ended up in mozilla/dist/install so dist doesn't get swamped as the locales increase in number. And we need to preserve all the localized installers so we have something to ship. I'm going to start looking at this, but I'm not familiar with the l10n packaging, so if ben or one of the l10n gurus could help out that would save a lot of time.
speaking of time, RC is going to be next week, so any l10n team that wants their l10n build shipped as part of this process please make sure it's green by 10/19.
I checked zh-CN locale and the same error: variable 'COMPLETE_RESET_HOMEPAGE' is not defined Now,I changed the mozilla/toolkit/locales/zh-CN/installer/windows/charset.mk to unix format and I checked mozilla/browser/locales/zh-CN/installer/installer.inc file,found that the "#define COMPLETE_RESET_HOMEPAGE ..." is not there,So I added it and now it's OK.
Created attachment 162150 [details] [diff] [review] move xpi dirs and langpacks to dist/install here's a simple diff to move the langpacks and xpi-AB-CD dirs from dist to dist/install
Comment on attachment 162150 [details] [diff] [review] move xpi dirs and langpacks to dist/install I'm not really understanding why it's better to have these in dist/install instead of dist, but you need to update @$(RM) -rf $(DIST)/xpi-$* to point to dist/install also. r=me if this is really necessary
> tarballs in mozilla/dist and only the installer builds for en-US and whichever > locale finished last. I think this is because when you make the installer, it wipes the stub/sea directory before starting. After each invocation of the loop you probably need to copy the result binary to the staging dir.
Is there a place we can set these files aside locally until we're ready to copy the whole set over?
I'm almost done my patch to post-mozilla-rel to save the installers and populate the 2004-xx-xx-xx dir correctly. Once that's done, just manually copy that directory to mozilla/.. to save a set.
chase set up all the l10n tinderbox pages, so that's ready. checked in patch to browser/locales/Makefile.in (including updating the rm line, thx ben). checked in patch to post-mozilla-rel incorporating post-mozilla-l10n and gathering all the various files into a single staging directory for pushing. tested it on linux, works great. chase is testing it on win32 and mac. Once it's ready we'll run a cycle and push a set of l10n builds to ftp.mozilla.org for testing.
Jon: I saw compreg.dat file in "2004-10-20-09-0.9-l10n" win32 zip package. If this file is in SeaMonkey dist archive, TalkBack doesn't start after crash. I can't check it on real crash, because our FF l10n build didn't started, so could you check TalkBack start?
(In reply to comment #30) > Jon: I saw compreg.dat file in "2004-10-20-09-0.9-l10n" win32 zip package. If Adam, I just filed this as bug 256492, simple patch is there.
(In reply to comment #31) > Adam, I just filed this as bug 256492, simple patch is there. that looks like the wrong bug#.
I expect most of this is done, but over to Chase just in case.
Assignee: granrosebugs → chase
Status: ASSIGNED → NEW
This is fixed. Resolving.
Status: NEW → RESOLVED
Last Resolved: 12 years ago
Resolution: --- → FIXED
Component: Tinderbox Configuration → Tinderbox
Product: mozilla.org → Webtools
You need to log in before you can comment on or make changes to this bug.