791209 - Nightly l10n repacks failing on Linux(32+64) due to old GCC (vers. 4.1.1) (Firefox & Thunderbird)

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Michael Foerster]

The update to 2012-09-13-03-05-47 happened OK on my Windows box, but failed to even show up on my Linux64 box. Browsing for the update by hand, I came upon the
Build-Log (2012-09-13-03-05-47-mozilla-central-l10n) for both Linux builds with the same error: gcc to old. 

Version shown: (line 885)
"checking compiler version... gcc (GCC) 4.1.1 20070105 (Red Hat 4.1.1-52)"

Aborted check: (line 944)
"checking for gcc pr39608... yes
This compiler would fail to build firefox, plase upgrade."

Log URL:
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2012/09/2012-09-13-03-05-47-mozilla-central-l10n/mozilla-central-linux64-l10n-nightly-de-bm25-build1-build6596.txt.gz

This maybe a fallout of the removal of GCC 4.2 support in Nighly.

I humbly ask for an upgrade of gcc in l10n (linux32+64) build system.

Comment 1

[Michael Foerster]

see also:

Bug 788276 - Properly check for broken (deprecated) compilers. 

in comment #8 by  Rafael Ávila de Espíndola (:espindola)

"add a check for gcc pr39608"

in comment #10 Gregory Szorc [:gps] gave r+ for that.

potential fallout is noted in #11

Comment 2

[Michael Foerster]

Sadly the discussion on bug 788276 does not bear the fruits needed.

Possible solutions:

link /usr/bin/cc and /usr/bin/gcc to the correct gcc version.

no modification of mozconfig, etc needed beyond that.

Comment 3

[Alexander L. Slovesnik]

Perhaps another option would be to whitelist this specific GCC version in check script until GCC on l10n builders will be upgraded.

Comment 4

[Alexander L. Slovesnik]

Any update on this issue? Currently there are no linux32/64 l10n nightly builds at all (there are only Win32/Mac builds in http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central-l10n/) and nobody step up to fix this bug.

Comment 5

[Pascal Chevrel:pascalc (PTO until April 26)]

+1 on getting an update, most of our Firefox localizers are on Linux and haven't been able to check their translation work or QA their builds for a couple of weeks now.

Comment 6

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: patch

NightlyRepackFactory inherits from BaseRepackFactory and BaseRepackFactory has a getMozconfig that copies self.mozconfig if it is defined, so the fix might be as simple as this patch, but I have no idea how to test it.

I did check that configure correctly merges arguments passed on the command line and in the .mozconfig, so while we can/should move things like 'l10n_extra_configure_args to dedicated mozconfig files, we don't need to do it right now.

Comment 7

[Chris AtLee [:catlee]]

Comment on attachment 662606 [details] [diff] [review]
patch

Review of attachment 662606 [details] [diff] [review]:
-----------------------------------------------------------------

yeah, this is the simplest solution...but it needs a lot of testing to make sure it doesn't bust on any branches/platforms.

I'm working on another patch at the moment which uses the in-tree mozconfigs instead of the ones in buildbot-configs. This is more desirable for releng because it means far fewer headaches when code merges between branches, but also requires the same amount of testing.

I hope to have this completed next week.

Comment 8

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Thanks!

Comment 9

[Mark Banner (:standard8)]

This is also affecting the Thunderbird builders, although somehow the Firefox ones seem to be passing atm.

Comment 10

[Chris AtLee [:catlee]]

Attachment: Use in-tree l10n-mozconfig for dep/nightly repacks

This also addresses bug 741756. By moving the 'l10n_revision' into the poller, we ensure that each dep repack gets the proper revision.

The in-tree mozconfigs will need some adjustment as well.

Comment 11

[Chris AtLee [:catlee]]

Attachment: l10n-mozconfig changes

I needed the --disable-elfhack option for linux to work around this error:

/tools/gcc-4.5-0moz3/bin/g++  -Wall -Wpointer-arith -Woverloaded-virtual -Werror=return-type -Wtype-limits -Wempty-body -Wno-ctor-dtor-privacy -Wno-overlength-strings -Wno-invalid-offsetof -Wno-variadic-macros -Wcast-align -Wno-long-long -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -std=gnu++0x -pthread -pipe -fexceptions  -DNDEBUG -DTRIMMED -g -Os -freorder-blocks -finline-limit=50 -fomit-frame-pointer -fPIC -shared -Wl,-z,defs -Wl,--gc-sections -Wl,-h,test-array.so -o test-array.so -lpthread  -Wl,-z,noexecstack   -Wl,-rpath-link,/builds/slave/m-cen-lnx64-l10n-dep/build/mozilla-central/dist/bin -Wl,-rpath-link,/usr/local/lib  test-array.o -nostartfiles
===
=== If you get failures below, please file a bug describing the error
=== and your environment (compiler and linker versions), and use
=== --disable-elf-hack until this is fixed.
===
# Fail if the library doesn't have INIT_ARRAY .dynamic info
readelf -d test-array.so | grep '(INIT_ARRAY)'
 0x0000000000000019 (INIT_ARRAY)         0x207330
/builds/slave/m-cen-lnx64-l10n-dep/build/mozilla-central/build/unix/elfhack/elfhack -b -f test-array.so
/builds/slave/m-cen-lnx64-l10n-dep/build/mozilla-central/build/unix/elfhack/elfhack: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.14' not found (required by /builds/slave/m-cen-lnx64-l10n-dep/build/mozilla-central/build/unix/elfhack/elfhack)
make[4]: *** [test-array.so] Error 1

although perhaps just the --enable-stdcxx-compat would be sufficient to fix that.

The macosx-universal mozconfig changes are required so that MOZ_APP_DISPLAYNAME is set properly. We need to look for FirefoxNightly.app for nightly/dep repacks. Release repacks will look for FirefoxFirefox.app I think...is that right?

Comment 12

[[e-mail] (:bhearsum)]

Comment on attachment 664980 [details] [diff] [review]
l10n-mozconfig changes

Review of attachment 664980 [details] [diff] [review]:
-----------------------------------------------------------------

Modulo the FirefoxFirefox.app weirdness, this looks good to me.

Comment 13

[[e-mail] (:bhearsum)]

(In reply to Chris AtLee [:catlee] from comment #11)
> The macosx-universal mozconfig changes are required so that
> MOZ_APP_DISPLAYNAME is set properly. We need to look for FirefoxNightly.app
> for nightly/dep repacks. Release repacks will look for FirefoxFirefox.app I
> think...is that right?

This doesn't sound right to me.....found in a release repack log:
	  set -ex; rm -rf /builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../dist/unpack.tmp; mkdir -p /builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../dist/unpack.tmp; /builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/build/package/mac_osx/unpack-diskimage /builds/slave/rel-m-rel-osx64-rpk-1/firefox.dmg /tmp/Firefox-unpack /builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../dist/unpack.tmp; rsync -a "/builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../dist/unpack.tmp/Firefox.app" firefox; test -n "/builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../dist/branding/dsstore" && rsync -a "/builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../dist/unpack.tmp/.DS_Store" "/builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../dist/branding/dsstore"; test -n "/builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../dist/branding/background.png" && rsync -a "/builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../dist/unpack.tmp/.background/`basename "/builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../dist/branding/background.png"`" "/builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../dist/branding/background.png"; test -n "/builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../dist/branding/disk.icns" && rsync -a "/builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../dist/unpack.tmp/.VolumeIcon.icns" "/builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../dist/branding/disk.icns"; rm -rf /builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../dist/unpack.tmp; if test -n "" ; then cp /builds/slave/rel-m-rel-osx64-rpk-1/firefox.dmg Firefox.tmp.dmg && hdiutil unflatten Firefox.tmp.dmg && { /Developer/Tools/DeRez -skip plst -skip blkx Firefox.tmp.dmg > "" || { rm -f Firefox.tmp.dmg && false; }; } && rm -f Firefox.tmp.dmg; fi && (cd firefox/Firefox.app/Contents/MacOS && /builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/_virtualenv/bin/python /builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/config/optimizejars.py --deoptimize /builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../jarlog//en-US ./ ./ && /usr/bin/unzip -o omni.ja && rm -f components/binary.manifest && for m in components/*.manifest; do sed -e 's/^#binary-component/binary-component/' $m > tmp.manifest && mv tmp.manifest $m; done)

Comment 14

[Chris AtLee [:catlee]]

The FirefoxFirefox.app weirdness comes from our buildbot-configs where we pass
l10n_extra_configure_args= ['--with-macbundlename-prefix=Firefox']

for mozilla-central and mozilla-aurora.

This shouldn't be a problem for release builds.

Comment 15

[Chris AtLee [:catlee]]

I re-ran the repacks without --disable-elfhack and they work fine. Looks like just the --enable-stdcxx-compat flag is sufficient.

Comment 16

[Chris AtLee [:catlee]]

Attachment: l10n-mozconfig changes

Comment 17

[Chris AtLee [:catlee]]

Comment on attachment 664978 [details] [diff] [review]
Use in-tree l10n-mozconfig for dep/nightly repacks

good enough to ship then?

we need to land the mozconfigs on both m-c and aurora first

Comment 18

[[e-mail] (:bhearsum)]

Comment on attachment 664978 [details] [diff] [review]
Use in-tree l10n-mozconfig for dep/nightly repacks

Yup!

Comment 19

[(not currently active) Ted Mielczarek]

Comment on attachment 665586 [details] [diff] [review]
l10n-mozconfig changes

Review of attachment 665586 [details] [diff] [review]:
-----------------------------------------------------------------

::: browser/config/mozconfigs/macosx-universal/l10n-mozconfig
@@ +3,5 @@
> +ac_add_options --with-l10n-base=../../l10n-central
> +
> +if [ "${MOZ_UPDATE_CHANNEL}" = "release" -o "${MOZ_UPDATE_CHANNEL}" = "beta" ]; then
> +ac_add_options --enable-official-branding
> +fi

Why is this here but not in the linux mozconfigs?

Comment 20

[Chris AtLee [:catlee]]

(In reply to Ted Mielczarek [:ted] from comment #19)
> Comment on attachment 665586 [details] [diff] [review]
> l10n-mozconfig changes
> 
> Review of attachment 665586 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: browser/config/mozconfigs/macosx-universal/l10n-mozconfig
> @@ +3,5 @@
> > +ac_add_options --with-l10n-base=../../l10n-central
> > +
> > +if [ "${MOZ_UPDATE_CHANNEL}" = "release" -o "${MOZ_UPDATE_CHANNEL}" = "beta" ]; then
> > +ac_add_options --enable-official-branding
> > +fi
> 
> Why is this here but not in the linux mozconfigs?

For the OSX repacks on central and aurora we pass in --with-macbundlename-prefix=Firefox. If we don't disable the official branding, we end up looking for FirefoxFirefox.app instead of FirefoxNightly.app.

This simply isn't a problem for linux/win32 repacks. I could add the same code there if you like.

Comment 21

[Chris AtLee [:catlee]]

Attachment: l10n mozconfigs for comm-central

Comment 22

[Mark Banner (:standard8)]

Comment on attachment 667603 [details] [diff] [review]
l10n mozconfigs for comm-central

Review of attachment 667603 [details] [diff] [review]:
-----------------------------------------------------------------

Ok, these are from the nightly mozconfigs, but I think that should be fine, as we are adding the specific l10n options via buildbot, and things like official branding shouldn't really matter as en-US will determine if they are official or not.

We probably need to keep a close eye on this for the first beta release though.

It does make me wonder that if the l10n specific options are provided by buildbot, why do we need l10n-mozconfigs at all, couldn't we just use nightly or release according to the branch we're on? (though this sounds like a follow-up).

Comment 23

[Justin Wood (:Callek)]

Comment on attachment 664978 [details] [diff] [review]
Use in-tree l10n-mozconfig for dep/nightly repacks

catlee, unless I'm staring wrong, this was never actually checked in -- is there a reason it wasn't?

Comment 24

[Chris AtLee [:catlee]]

Comment on attachment 667603 [details] [diff] [review]
l10n mozconfigs for comm-central

Bug caused by (feature/regressing bug #): switching to newer compilers
User impact if declined: no l10n repacks
Testing completed (on m-c, etc.): releng testing in staging
Risk to taking this patch (and alternatives if risky): busted l10n repacks
String or UUID changes made by this patch: none

Comment 25

[Chris AtLee [:catlee]]

Comment on attachment 665586 [details] [diff] [review]
l10n-mozconfig changes

Bug caused by (feature/regressing bug #): switching to newer compilers
User impact if declined: no l10n repacks
Testing completed (on m-c, etc.): releng testing in staging
Risk to taking this patch (and alternatives if risky): busted l10n repacks
String or UUID changes made by this patch: none

Comment 26

[Justin Wood (:Callek)]

Comment on attachment 667603 [details] [diff] [review]
l10n mozconfigs for comm-central

bugzilla-wise I can't approve this, but a=me.

Comment 27

[Mark Banner (:standard8)]

Comment on attachment 667603 [details] [diff] [review]
l10n mozconfigs for comm-central

Please sync the CLIENT_PY_ARGS change with the changes in this file:

http://hg.mozilla.org/comm-central/rev/5a49ea8a7052

r+a=me for landing the updated patch on comm-central and comm-aurora.

This can land with DONTBUILD in both locations.

Comment 28

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 665586 [details] [diff] [review]
l10n-mozconfig changes

stamp.

Comment 29

[Chris AtLee [:catlee]]

Comment on attachment 665586 [details] [diff] [review]
l10n-mozconfig changes

http://hg.mozilla.org/mozilla-central/rev/be64e9425b0b
http://hg.mozilla.org/releases/mozilla-aurora/rev/47bf308b741b

Comment 30

[Chris AtLee [:catlee]]

Comment on attachment 667603 [details] [diff] [review]
l10n mozconfigs for comm-central

http://hg.mozilla.org/comm-central/rev/3e92de76ca7a
http://hg.mozilla.org/releases/comm-aurora/rev/b49b7cd48e6c

Comment 31

[Aki Sasaki (not active)]

This is now live.