902876 - [Tracking Bug] Fix SeaMonkey l10n dep nightly builds

Status: RESOLVED FIXED

(SeaMonkey :: Build Config, defect)

Description

[Tony Mechelynck [:tonymec]]

No SeaMonkey langpacks later than 26 July. The most recent trunk builds are as follows:

- 26-Jul: 2.22a1 be, de, es-AR, es-ES, ga, lt, hu, nl, ro, ru, sk, uk, zh-CN, zh-TW
- 09-Jul: 2.22a1 en-GB, fr, ja, ja-JP-mac, pl 
- 2.19a1 cz, nb-NO
- 2.17a1 pt-PT
- 2.14a1 it
- 2.12a1 sv-SE
- 2.6a1 fi
- 2.1b1pre ka

Comment 1

[Frank Wein [:mcsmurf]]

So regarding the build problems on aurora, I saw a build log yesterday about stdint.h missing. Firefox also had the same problem on their beta channel, see Bug 902091. They solved it by using some rather complicated patches IMHO. I think for us the easiest/best solution would be to fix the include/lib/path used during the build process. The problem is that the l10n process does call "make" directly and does not use client.mk to do the various work. But "make" does not take the env vars from this mozconfig file (http://mxr.mozilla.org/mozilla-central/source/build/win32/mozconfig.vs2010?raw=1&ctype=, it's included in the l10n mozconfig) into account. I would just set those path on those machines (via buildbot) during the build process. Firefox can't do that as they still use VS 2008 for the ESR builds. We only use VS 2010 on Windows afaik, so there's no point in keeping the old VS 2008 INCLUDE/LIB/PATH/LIBPATH/WIN32_REDIST_DIR env vars around.

Comment 2

[Tony Mechelynck [:tonymec]]

And I suppose there are similar mozconfigs for linux-i686 and mac. Strange that make does not obey them; normally it takes env vars as make vars, but if defined also in the Makefile without a test that they already exist you need "make -e" to give envvars precedence over makevars.

Comment 3

[Tony Mechelynck [:tonymec]]

July 26 is also the latest build date for linux-i686 _full_ Suite builds; and, for some time prior to that, new langpacks appeared only in https://ftp.mozilla.org/pub/mozilla.org/seamonkey/nightly/latest-comm-central-trunk-l10n/linux-i686/xpi/ but not in ../../win32/xpi/ or ../../mac/xpi/

Comment 4

[Frank Wein [:mcsmurf]]

BTW: Bug 903118 is working on exporting the env vars from .mozconfig files into the build environment. But not sure how and when and on which branches this patch will land.

Comment 5

[Tony Mechelynck [:tonymec]]

Now that Linux builds are being made again, langpacks have also started being produced again (but only from linux-i686 builds, not win32 or mac) for languages be, de, es-AR, es-ES, gl, hu, lt, nl, pl, ro, ru, sk, uk, zh-CN, zh-TW, i.e., the same ones that were being built just before building stopped altogether on Linux (ga in comment #0 is probably a typo).

No full localized builds either except on Linux; on win32 and mac they are being attempted but fail: IIUC the .txt.gz files in http://ftp.mozilla.org/pub/mozilla.org/seamonkey/nightly/latest-comm-central-trunk-l10n/ are the build logs.

IOW non-Linux testers who want a localized build should grab an en-US build for their platform and a linux-i686 langpack for their language (one among the 15 above).

Comment 6

[Ian Neal]

So what is stopping the ones from the 09-Jul-13 from being built now?

Comment 7

[Tony Mechelynck [:tonymec]]

(In reply to Ian Neal from comment #6)
> So what is stopping the ones from the 09-Jul-13 from being built now?

I don't know. I guess the same thing that prevented them being built from 10 to 26 July.

Let's have a look at http://ftp.mozilla.org/pub/mozilla.org/seamonkey/nightly/latest-comm-central-trunk-l10n/comm-central-trunk-linux-l10n-nightly-en-GB-bm01-universal-build714.txt.gz (which SeaMonkey unzips for me, no need to download it first):

Comparing en-GB to en-US
Properties in ./chrome/chatzilla.properties don't match:
  In ../../../extensions/irc/locales/en-US: (add these to your localization)
    cmd.clear-view.key

make[2]: *** [libs] Error 1

Comment 8

[Philip Chee]

There is also:
Bug 901338 - creation of langpack for for locale "de" fails

Comment 9

[Tony Mechelynck [:tonymec]]

(In reply to Philip Chee from comment #8)
> There is also:
> Bug 901338 - creation of langpack for for locale "de" fails

Yes, but that's for SeaMonkey 2.20. German (and 13 other) langpacks are now being built again on trunk.

Comment 10

[Robert Kaiser]

(In reply to Philip Chee from comment #8)
> There is also:
> Bug 901338 - creation of langpack for for locale "de" fails

That's different, it's not about our build systems but about someone trying to create it himself, without our L10n merging that runs on our build systems.

Comment 11

[Frank Wein [:mcsmurf]]

BTW: I looked at some random OS X build log for trunk from today (http://ftp.mozilla.org/pub/mozilla.org/seamonkey/nightly/latest-comm-central-trunk-l10n/comm-central-trunk-macosx64-l10n-nightly-es-ES-bm01-universal-build2078.txt.gz), the error message says:
========= Started configure done failed (results: 2, elapsed: 30 secs) (at 2013-09-16 23:38:56.702796) ==========
[...]
checking Python environment is Mozilla virtualenv... yes
configure: error: Invalid value --with-l10n-base, ../../l10n-central doesn't exist
*** Fix above errors and then restart with               "make -f client.mk build"
make[1]: *** [configure] Error 1

Comment 12

[Frank Wein [:mcsmurf]]

Justin: Am I correct that probably Bug 852553 needs to be ported to our buildbot version and also the mozconfig changes need to be ported (to get nightly l10n builds working again)? Not sure why exactly this needs to be done now, but ewong told me you basically applied the patch from Bug 852553 locally to the buildbot (not committed to mercurial yet) to get beta builds working.

Comment 13

[Edmund Wong (:ewong)]

(In reply to Frank Wein [:mcsmurf] from comment #12)
> Justin: Am I correct that probably Bug 852553 needs to be ported to our
> buildbot version and also the mozconfig changes need to be ported (to get
> nightly l10n builds working again)? Not sure why exactly this needs to be
> done now, but ewong told me you basically applied the patch from Bug 852553
> locally to the buildbot (not committed to mercurial yet) to get beta builds
> working.

The buildbotcustom changes from bug 852553 has been incorporated into
our buildbotcustom code (from bug 1039139) so this part is done.  We're
now waiting for changes from bug 840427 to be reviewed and pushed to
the repo (which then requires a subsequent master reconfig).  But this
is waiting for the resolution of bug 795354.

So what this bug needs now is the suite/config changes, which I'll provide
asap.

Comment 14

[Edmund Wong (:ewong)]

Attachment: config/mozconfig changes patch (v1)

Comment 15

[Edmund Wong (:ewong)]

Attachment: config/mozconfig changes patch (v2)

Comment 16

[Edmund Wong (:ewong)]

Comment on attachment 8458558 [details] [diff] [review]
config/mozconfig changes patch (v1)

Review of attachment 8458558 [details] [diff] [review]:
-----------------------------------------------------------------

::: suite/config/mozconfigs/macosx-universal/l10n-mozconfig
@@ +9,5 @@
>  
>  . $topsrcdir/build/macosx/universal/mozconfig
>  
>  ac_add_options --enable-application=suite
> +ac_add_options --with-l10n-base=../../l10n

Actually, just noticed that this should have been a ../../../l10n,
so I updated the patch.

Comment 17

[Justin Wood (:Callek)]

Comment on attachment 8459081 [details] [diff] [review]
config/mozconfig changes patch (v2)

Review of attachment 8459081 [details] [diff] [review]:
-----------------------------------------------------------------

not looking closely at non-beta atm, but stamp=me and a=me for needed branches

Comment 18

[Edmund Wong (:ewong)]

Pushed to comm-central:
https://hg.mozilla.org/comm-central/rev/bb2653b7803

Comment 19

[Ian Neal]

(In reply to Edmund Wong (:ewong) from comment #16)
> Comment on attachment 8458558 [details] [diff] [review]
> config/mozconfig changes patch (v1)
> 
> Review of attachment 8458558 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: suite/config/mozconfigs/macosx-universal/l10n-mozconfig
> @@ +9,5 @@
> >  
> >  . $topsrcdir/build/macosx/universal/mozconfig
> >  
> >  ac_add_options --enable-application=suite
> > +ac_add_options --with-l10n-base=../../l10n
> 
> Actually, just noticed that this should have been a ../../../l10n,
> so I updated the patch.

Just wondering why this needs to have the extra ../

Comment 20

[Edmund Wong (:ewong)]

(In reply to Ian Neal from comment #19)
> (In reply to Edmund Wong (:ewong) from comment #16)
> > Comment on attachment 8458558 [details] [diff] [review]
> > config/mozconfig changes patch (v1)
> > 
> > Review of attachment 8458558 [details] [diff] [review]:
> > -----------------------------------------------------------------
> > 
> > ::: suite/config/mozconfigs/macosx-universal/l10n-mozconfig
> > @@ +9,5 @@
> > >  
> > >  . $topsrcdir/build/macosx/universal/mozconfig
> > >  
> > >  ac_add_options --enable-application=suite
> > > +ac_add_options --with-l10n-base=../../l10n
> > 
> > Actually, just noticed that this should have been a ../../../l10n,
> > so I updated the patch.
> 
> Just wondering why this needs to have the extra ../

(aside for mail/  having the same thing) I tried it with only
../../ and it failed due to path issues.  I haven't looked into
it too deeply (since I'm currently fixing the buildbotcustom code),
but once I get it done.. I'll figure out why.

Comment 21

[Edmund Wong (:ewong)]

Comment on attachment 8459081 [details] [diff] [review]
config/mozconfig changes patch (v2)

[Approval Request Comment]
Regression caused by (bug #): 
User impact if declined: no langpacks created. 
Testing completed (on m-c, etc.): 
Risk to taking this patch (and alternatives if risky):
String changes made by this patch:

Comment 22

[Edmund Wong (:ewong)]

Comment on attachment 8459081 [details] [diff] [review]
config/mozconfig changes patch (v2)

Got a+ from Callek over IRC:

Pushed to comm-aurora:
https://hg.mozilla.org/releases/comm-aurora/rev/4013101a7d2e

Pushed to comm-beta:
https://hg.mozilla.org/releases/comm-beta/rev/b52ded91ca91

Comment 23

[Ian Neal]

Comment on attachment 8459081 [details] [diff] [review]
config/mozconfig changes patch (v2)

Marking as a= due to comment 22. Should this be marked as fixed now it is pushed?

Comment 24

[Edmund Wong (:ewong)]

(In reply to Ian Neal from comment #23)
> Comment on attachment 8459081 [details] [diff] [review]
> config/mozconfig changes patch (v2)
> 
> Marking as a= due to comment 22. Should this be marked as fixed now it is
> pushed?

No.  OSX langpacks aren't being generated.  I'm not sure why because we
have the 'same' path as mail/; but mail/ doesn't choke whereas we do.
I haven't figured out why it's not liking the ../../../l10n path.

Comment 25

[Edmund Wong (:ewong)]

Attachment: OSX64 fix.

In the previous patch, I set the --with-l10n-basedir=../../../l10n because
mail/ had it.  I'm not sure if they even use that for their l10n langpacks;
haven't had the chance to check out mail/'s infra logs.
This patch is to make it run with ../../l10n.  (I mean, if it works with
linux* and WinNT, who am I to say it won't work with OSX64? ;P )

Comment 26

[Tony Mechelynck [:tonymec]]

As a workaround, langpacks (and full Linux localized builds) for some languages can be downloaded from the subdirectories of http://l10n.mozilla-community.org/~akalla/unofficial/seamonkey/nightly/

For langpacks (but of course not for full builds), you may disregard the platform but not the branch.

These langpacks and builds are "unofficial" because they are built out of Mozilla control, but (Adrian, please correct me if I'm wrong) they are built from official Mozilla sources.

Comment 27

[Adrian Kalla [:adriank]]

(In reply to Tony Mechelynck [:tonymec] from comment #26)
> As a workaround, langpacks (and full Linux localized builds) for some
> languages can be downloaded from the subdirectories of
> http://l10n.mozilla-community.org/~akalla/unofficial/seamonkey/nightly/

This is not really a workaround for the bug - more a way to get language packs without the need to build them itself ;)

> These langpacks and builds are "unofficial" because they are built out of
> Mozilla control, but (Adrian, please correct me if I'm wrong) they are built
> from official Mozilla sources.

Yep - they are always being built from the latest Mozilla sources (see the Python script I do use to create this builds: https://bitbucket.org/adrianer/seamonkey-build/src/default/build-sm.py ).


Nevertheless: why isn't this bug fixed nearly 1,5 years later...? :(

Comment 28

[Tony Mechelynck [:tonymec]]

(In reply to Adrian Kalla [:adriank] from comment #27)
> (In reply to Tony Mechelynck [:tonymec] from comment #26)
> > As a workaround, langpacks (and full Linux localized builds) for some
> > languages can be downloaded from the subdirectories of
> > http://l10n.mozilla-community.org/~akalla/unofficial/seamonkey/nightly/
> 
> This is not really a workaround for the bug - more a way to get language
> packs without the need to build them itself ;)

Depends on what one calls a workaround. It is a workaround for the user who cannot get langpacks from Mozilla. It is not a workaround for the RelEng and l10n teams trying to produce langpacks and offer them at ftp.mozilla.org, and it is certainly not a fix.

Comment 29

[Edmund Wong (:ewong)]

Comment on attachment 8506536 [details] [diff] [review]
OSX64 fix.

Pushed to comm-central: (bustage fix)
https://hg.mozilla.org/comm-central/rev/c31c999efa37

Comment 30

[Edmund Wong (:ewong)]

(In reply to Edmund Wong (:ewong) from comment #29)
> Comment on attachment 8506536 [details] [diff] [review]
> OSX64 fix.
> 
> Pushed to comm-central: (bustage fix)
> https://hg.mozilla.org/comm-central/rev/c31c999efa37

This was backed out as it's actually wrong.  The original value is
correct, i.e. ../../../l10n.

The reason is because the platform OBJDIR is objdir/i386, whereas
for the other platforms, they're just objdir.

Comment 31

[Edmund Wong (:ewong)]

Attachment: osx64 fix (v3)

Comment 32

[Edmund Wong (:ewong)]

Attachment: osx64 fix (v4)

Previous patch removed the mozconfig include.  Instead, we should skip
the build/macosx/universal/mozconfig (which includes the mozconfig.common file)
and use the mozconfig.common file directly.

Comment 33

[Edmund Wong (:ewong)]

Comment on attachment 8574581 [details] [diff] [review]
osx64 fix (v4)

Rationale for this patch:

$topsrcdir/build/macosx/universal/mozconfig defines build projects which,
for macosx64 = i386 and x86_64.  What this does is during the client.mk 
configure process, it creates under the objdir (in osx64 case, it's 
objdir/i386) two subdirectories (i386 and x86_64) as the new objdir.  However, 
all of the osx64 building assumes the objdir = objdir/i386.   So when the build 
process goes into objdir/i386/i386,  the configure process chokes because it's 
not finding the l10n dir (which is normally ../../../l10n, but with the added 
level, it should be ../../../../l10n).

(Another solution was to use --with-l10n-base=../../../../l10n, but that's
clunky.)

However, since build/macosx/universal/mozconfig also calls the mozconfig.common
file from the same directory, we might as well use the mozconfig.common file.

Comment 34

[Edmund Wong (:ewong)]

Comment on attachment 8574581 [details] [diff] [review]
osx64 fix (v4)

Since only our osx64 nightlies are building, and triggering the l10n
changes,  I can only hope to garner some info from the osx64 nightly l10n
repacks.

I'm going to push this for a post-land review and trigger a repack.

Comment 35

[Edmund Wong (:ewong)]

Comment on attachment 8574581 [details] [diff] [review]
osx64 fix (v4)

Pushed to comm-central:
  https://hg.mozilla.org/comm-central/rev/c462becfd612

Comment 36

[Edmund Wong (:ewong)]

(In reply to Edmund Wong (:ewong) from comment #35)
> Comment on attachment 8574581 [details] [diff] [review]
> osx64 fix (v4)
> 
> Pushed to comm-central:
>   https://hg.mozilla.org/comm-central/rev/c462becfd612

This managed to unhork it from the configure but it fails at the repack_installer
step (which is possibly fixed in the backend..)

So this needs to be pushed to comm-aurora.

Comment 37

[Ian Neal]

Comment on attachment 8574581 [details] [diff] [review]
osx64 fix (v4)

[Triage Comment]
a=me for c-a

Comment 39

[Edmund Wong (:ewong)]

https://hg.mozilla.org/SeaMonkey/puppet/rev/70968214f45fae3d5566700c85ff875985ecabbc
Whitelist c-cen-t-lnx-l10n-ntly temporarily to help fix bug 902876

Comment 40

[Edmund Wong (:ewong)]

Attachment: [mozconfigs] remove no_tooltool=1 from Linux* l10n mozconfigs

repacks now need to use the tooltool-enabled mozconfigs.

Comment 41

[Edmund Wong (:ewong)]

https://hg.mozilla.org/comm-central/rev/887ac30fffb3e349d8774d56c5ea9d65697b249a
Bug 902876 - Remove no_tooltool from Linux* l10n mozconfigs. r+a=bustagefix

Comment 42

[Philip Chee]

Do we want to port Bug 1283438? (Set AUTOCLOBBER to empty for l10n)

Comment 43

[Edmund Wong (:ewong)]

(In reply to Philip Chee from comment #42)
> Do we want to port Bug 1283438? (Set AUTOCLOBBER to empty for l10n)

Possibly something we might need.

Comment 44

[Edmund Wong (:ewong)]

Morphing this to a tracking bug.

Comment 45

[Frank-Rainer Grahl (:frg)]

lets call it fixed.