Install/merge localized tinderbox post-mozilla-l10n and post-mozilla-rel for tinderbox and nightly build coverage

RESOLVED FIXED

Status

P2
major
RESOLVED FIXED
15 years ago
5 years ago

People

(Reporter: benjamin, Assigned: chase)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

15 years ago
We need tinderbox and nightly build coverage of localized aviary firefoxen (and
later aviary thunderbirds). I have written a post-mozilla script that piggybacks
on a regular tinderbox build to test all the available localizations, see

http://lxr.mozilla.org/mozilla/source/tools/tinderbox/post-mozilla-l10n.pl

and my person machine is running this for the moment, see
http://tinderbox.mozilla.org/showbuilds.cgi?tree=MozillaTest

This is not really tested with win32/mac, but it should work with few modifications.

This most likely needs to be integrated with the post-mozilla-rel script, so
that we can stage the localized builds after we build/test them. I currently
don't have stage access, so I would be coding in the dark trying to merge the
two. It would also help to land bug 257177 before/with this change. I think that
patch should land as-is, and the "pretty names" should be managed with symlinks,
either created by post-mozilla-rel, or by the stage server itself.

Bryner said to contact granrose about this... it is a fairly urgent priority for
the pending l10n freeze.
(Reporter)

Comment 1

15 years ago
Damn, that should be bug 257117, pardon my dyslexia.
Depends on: 257117
No longer depends on: 257177

Comment 2

15 years ago
Ok, have the new linux build system set up (chase is working on the win32), and
have started hacking post-mozilla-rel.pl to include post-mozilla-l10n.

Question: how many products are we doing this for? We're going to need at least
one more tinderbox page and I'm wondering which setup is best:

- one for all localizations of all products (i.e. Mozilla-l10n)

- one per product and branch (i.e. Firefox-branch-l10n, Firefox-trunk-l10n,
Thunderbird-branch-l10n, etc.).

There are 17 locales for firefox branch, so for all three platforms, all
locales, we're talking 54 columns for one product (1 full build + 17 localized *
3 platforms).  I'm planning on setting up the latter but wanted to give a chance
for feedback.
Status: NEW → ASSIGNED
Priority: -- → P2
(Reporter)

Comment 3

15 years ago
> Question: how many products are we doing this for? We're going to need at least

Firefox and Thunderbird, in the near term. Towards 1.8 xulrunner will want to be
in on the game, as might sunbird.

> - one per product and branch (i.e. Firefox-branch-l10n, Firefox-trunk-l10n,
> Thunderbird-branch-l10n, etc.).

This sounds much more manageable to me.

> There are 17 locales for firefox branch, so for all three platforms, all

Expect at least 10 more by the time we hit 1.0; the ultimate goal is 50+

--BDS

Comment 4

15 years ago
50+ is a lot of machines!  Are we round-robin-ing the builds at all?
Maybe one build could do Spanish-French-Japanese-German and cycle on that, print the language out
for tinderbox so you know what failed when it does.

I agree with Benjamin, you want the l10n status to follow the product around, dumping them
all into a single l10n page is going to get noisy.
Just one question: Can we expect official localised (german) builds for PR
2/Beta1 on 10/18? I need to know that, 'cause the answer will influence my
decision whether I should release PR 1 now, or wait for PR2/Beta1.

Comment 6

15 years ago
hmm.  50+ locales makes for a lot of builds (151 for each product).  The
tinderbox-page-per-product-branch idea might not scale well.  We'll tackle that
when it shows up, I guess, but may have to go to a locale-centric scheme where
all the products for a specific locale report to a specific page, e.g.
Mozilla-l10n-ruRU with firefox branch, firefox trunk, tbird branch, etc. all
reporting to that page to keep it manageable.
Locale-centric scheme would be definetely preferable by l10n teams, because it
is more convenient for us to check one page whether all our ab-CD builds are
fine than searching all tinderbox-per-product-pages.

Comment 8

15 years ago
for the record: need this to pull and package the l10n files

mk_add_options MOZ_CO_LOCALES=all

and if you're not using the cvs-mirror, you need a line like
mk_add_options LOCALES_CVSROOT=:ext:bsmedberg%covad.net@cvs.mozilla.org:/l10n

Comment 9

15 years ago
Ok, so for the tinderbox pages, it sounds like locale-specific pages is the
preferred way, so any product, any platform that does es-AR (for example) will
send all tinderbox logs to Mozilla-l10n-es-AR.  There will also be a
Mozilla-l10n page that the systems producing the l10n builds can send their full
build log to so it doesn't take up space on the primary product page.

The linux builds completed, and the win32 is underway.  I see 17
firefox*langpack.xpi files, but only one set of installers saved that includes
the locale and that's the last locale that was run (he-IL).

Time to push bits, so more questions:

what files are we supposed to be delivering to the ftp server for nightlies or
for a final release?  Are we shipping a langpack, tarball and blob installer for
each locale?

What should the files be named?  Currently on linux we have
firefox.0.10.ab-CD.langpack.xpi  (17 of these)
firefox-i686-linux-gtk2+xft.tar.gz
firefox-i686-linux-gtk2+xft-installer.tar.gz

If we're pushing blob installers and tarballs, where does the locale go in the
filename?  firefox-i686-linux-gtk2+xft-ab-CD.tar.gz?

Do we need to push the langpacks from every platform?  If so, do we need to
incorporate platform information in the langpack filenames?

Should we push langpacks/installers when they return busted or testfailed?  I'm
assuming no.

When are we getting more disk space (stage is almost full)?  To push a single
product, single platform for 17 locales of langpacks, blob installers and
tarballs takes about 244MB (17*7.2MB*2 + 1.7MB) for the 2004-10-13-xx-0.9
directory, plus about the same for latest-0.9, so 480MB.  Add two more platforms
and that's 1.4GB / day.  We currently have 1.6GB free on stage, so this is a
blocker unless we're only shipping langpacks.  Dropping tarballs cuts that in
half, but 800MB per day is a big hit.

Speaking of which, do we have to worry about the mirror sites and the bandwidth
for mirroring this additional data?
(Reporter)

Comment 10

15 years ago
The patch to make the filenames sane is bug 257117. It needs the existing
staging tinderboxen to update their config files when it lands.

We do not need to push langpacks from every platform.

I don't know what to do about disk space.
FWIW, I'm still highly uncomfortable with the idea of using separate machines
(rather than a step at the end of the existing builds) for the localized builds.
 It could mean having differences in configuration between the English build and
all the other builds, and it doesn't scale well to lots of products and branches.
(Assignee)

Comment 12

15 years ago
Summary of Windows l10n errors based on the chroma errors found on
http://tinderbox.mozilla.org/showbuilds.cgi?tree=MozillaTest:

    cs-CZ: COMPLETE_RESET_HOMEPAGE
  * da-DK: mozilla/toolkit/locales/da-DK/installer/windows/charset.mk missing
    de-DE: iconv: conversion to CP1252 unsupported
  * es-AR: mozilla/browser/locales/es-AR/installer/installer.inc missing
    he-IL: COMPLETE_RESET_HOMEPAGE
    hu-HU: iconv: conversion to CP1250 unsupported
    nb-NO: iconv: conversion to CP1252 unsupported
    pl-PL: mozilla/browser/locales/pl-PL/installer/installer.inc: can't convert
    pt-BR: COMPLETE_RESET_HOMEPAGE
    zh-CN: COMPLETE_RESET_HOMEPAGE

The starred errors are also affecting karma and bsmedberg's builds.  They are
specific to the build system as stored and once fixed for one system should be
fixed for all systems.

The COMPLETE_RESET_HOMEPAGE error looks the same across the four locales that
exhibit it.  Here's an instance of the error from the cs-CZ locale:

/cygdrive/c/builds/tinderbox/firefox-aviarybranch-l10n/WINNT_5.1_Clobber/mozilla/config/preprocessor.pl:/cygdrive/c/builds/tinderbox/firefox-aviarybranch-l10n/WINNT_5.1_Clobber/mozilla/browser/installer/windows/config.it:394:
error using substitution: variable 'COMPLETE_RESET_HOMEPAGE' is not defined

The iconv unsupported encoding errors suggest that chroma needs a newer version
of iconv or one of its supporting libraries.  bsmedberg, have you tested the
post-mozilla-l10n modifications on Windows?  If so did you need to make any
special modifications to add support for the CP1250 and CP1252 character sets?
(Assignee)

Comment 13

15 years ago
Re: the CP1250 and CP1252 conversion errors, I think the problem is that in some
locales WIN_INSTALLER_CHARSET is getting set to "CP1250\n" and "CP1252\n".

On the fi-FI locale, CP1252 is used and the iconv command exits succesfully. 
The log for fi-FI is at
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1097705640.16638.gz&fulltext=1.

A failing CP1252 example can be seen in
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1097705640.15274.gz&fulltext=1.

The main difference between the successful log file and the failing log file is
that there is a newline after CP1252 in the failing log file.  Additionally,
'iconv -l' on chroma includes CP1250 and CP1252 in its list of supported
character sets.
(Assignee)

Comment 14

15 years ago
The following three files have incorrect line endings which are causing the
build failures for their localizations:

  mozilla/toolkit/locales/de-DE/installer/windows/charset.mk
  mozilla/toolkit/locales/hu-HU/installer/windows/charset.mk
  mozilla/toolkit/locales/nb-NO/installer/windows/charset.mk

An example of a working charset.mk can be found at:

  mozilla/toolkit/locales/fi-FI/installer/windows/charset.mk

Viewing the files in 'vi' or 'vim' will show the improperly places ^Ms at the
end of each line.  Each of the non-working files mentioned above should be
updated to use Unix line endings.  I will spin this out into a separate bug for
tracking purposes.
(Assignee)

Updated

15 years ago
Depends on: 264278

Comment 15

15 years ago
checked in a fix to de-DE/installer/windows/charset.mk for a.topal that I had
in my tree locally.

Comment 16

15 years ago
(In reply to comment #14)

> Viewing the files in 'vi' or 'vim' will show the improperly places ^Ms at the
> end of each line.  Each of the non-working files mentioned above should be
> updated to use Unix line endings.  I will spin this out into a separate bug for
> tracking purposes.

CVS does the conversion for line endings of text files automatically. It is
better to say that all text files should have line endings which match to that
of the operating system. That is "0D 0A" for Windows (using WinCVS) and "0A" for
unix like operating systems (including Cygwin on Windows if Unix like line
endings have been chosen during Cygwin install). It is wrong to recommend
updating these files to use Unix line endings.

By the way, hu-HU have been fixed.
(Assignee)

Comment 17

15 years ago
(In reply to comment #16)
> 
> CVS does the conversion for line endings of text files automatically. It is
> better to say that all text files should have line endings which match to that
> of the operating system. That is "0D 0A" for Windows (using WinCVS) and "0A" for
> unix like operating systems (including Cygwin on Windows if Unix like line
> endings have been chosen during Cygwin install). It is wrong to recommend
> updating these files to use Unix line endings.
> 
> By the way, hu-HU have been fixed.

Please update bug 264278 mentioning your fix.  Thanks.

Comment 18

15 years ago
http://tinderbox.mozilla.org/showbuilds.cgi?tree=MozillaTest does not include
any swedish (sv-SE) build. I'd like to request the addition of such builds. Is
there anything I need to do in order to make this possible?

Comment 19

15 years ago
(In reply to comment #18)
> http://tinderbox.mozilla.org/showbuilds.cgi?tree=MozillaTest does not include
> any swedish (sv-SE) build. I'd like to request the addition of such builds. Is
> there anything I need to do in order to make this possible?

Checked in fixes to browser|toolkit/locales/all-locales. sv-SE is red, though,
http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1097763420.13140.gz
Both chroma and bdsmedberg boxes picked it up.
(Assignee)

Updated

15 years ago
Depends on: 264431
(Assignee)

Updated

15 years ago
Depends on: 264434
(Assignee)

Updated

15 years ago
Depends on: 264435
(Assignee)

Updated

15 years ago
Depends on: 264437

Comment 20

15 years ago
new standardized file names look good.  we need to integrate the en-US and l10n
installer builds though.  Currently we end up with the langpacks and l10n
tarballs in mozilla/dist and only the installer builds for en-US and whichever
locale finished last.

It would be better if the langpacks, tarballs and the xpi directories ended up
in mozilla/dist/install so dist doesn't get swamped as the locales increase in
number.  And we need to preserve all the localized installers so we have
something to ship.

I'm going to start looking at this, but I'm not familiar with the l10n
packaging, so if ben or one of the l10n gurus could help out that would save a
lot of time.

Comment 21

15 years ago
speaking of time, RC is going to be next week, so any l10n team that wants their
l10n build shipped as part of this process please make sure it's green by 10/19.

Comment 22

15 years ago
I checked zh-CN locale and the same error:
variable 'COMPLETE_RESET_HOMEPAGE' is not defined
Now,I changed the mozilla/toolkit/locales/zh-CN/installer/windows/charset.mk to 
unix format and I checked mozilla/browser/locales/zh-CN/installer/installer.inc 
file,found that the "#define COMPLETE_RESET_HOMEPAGE ..." is not there,So I 
added it and now it's OK.

Comment 23

15 years ago
Created attachment 162150 [details] [diff] [review]
move xpi dirs and langpacks to dist/install

here's a simple diff to move the langpacks and xpi-AB-CD dirs from dist to
dist/install
(Reporter)

Comment 24

15 years ago
Comment on attachment 162150 [details] [diff] [review]
move xpi dirs and langpacks to dist/install

I'm not really understanding why it's better to have these in dist/install
instead of dist, but you need to update @$(RM) -rf $(DIST)/xpi-$* to point to
dist/install also. r=me if this is really necessary
(Reporter)

Comment 25

15 years ago
> tarballs in mozilla/dist and only the installer builds for en-US and whichever
> locale finished last.

I think this is because when you make the installer, it wipes the stub/sea
directory before starting. After each invocation of the loop you probably need
to copy the result binary to the staging dir.
(Assignee)

Comment 26

15 years ago
Is there a place we can set these files aside locally until we're ready to copy
the whole set over?

Comment 27

15 years ago
I'm almost done my patch to post-mozilla-rel to save the installers and populate
the 2004-xx-xx-xx dir correctly.  Once that's done, just manually copy that
directory to mozilla/.. to save a set.

Comment 28

15 years ago
chase set up all the l10n tinderbox pages, so that's ready.

checked in patch to browser/locales/Makefile.in (including updating the rm line,
thx ben).
checked in patch to post-mozilla-rel incorporating post-mozilla-l10n and
gathering all the various files into a single staging directory for pushing.

tested it on linux, works great.  chase is testing it on win32 and mac.  Once
it's ready we'll run a cycle and push a set of l10n builds to ftp.mozilla.org
for testing.

Comment 29

15 years ago
added dependency upon mac talkback not working and win32 not using 7zip.
Depends on: 265325, 265464

Comment 30

15 years ago
Jon: I saw compreg.dat file in "2004-10-20-09-0.9-l10n" win32 zip package. If
this file is in SeaMonkey dist archive, TalkBack doesn't start after crash. I
can't check it on real crash, because our FF l10n build didn't started, so could
you check TalkBack start?
(Reporter)

Comment 31

15 years ago
(In reply to comment #30)
> Jon: I saw compreg.dat file in "2004-10-20-09-0.9-l10n" win32 zip package. If

Adam, I just filed this as bug 256492, simple patch is there.

(In reply to comment #31)
> Adam, I just filed this as bug 256492, simple patch is there.

that looks like the wrong bug#.

Updated

15 years ago
Depends on: 265564

Comment 33

14 years ago
I expect most of this is done, but over to Chase just in case.
Assignee: granrosebugs → chase
Status: ASSIGNED → NEW
(Assignee)

Comment 34

13 years ago
This is fixed.  Resolving.
Status: NEW → RESOLVED
Last Resolved: 13 years ago
Resolution: --- → FIXED

Updated

8 years ago
Component: Tinderbox Configuration → Tinderbox
Product: mozilla.org → Webtools
Product: Webtools → Webtools Graveyard
You need to log in before you can comment on or make changes to this bug.