Nightly l10n repacks failing on Linux(32+64) due to old GCC (vers. 4.1.1) (Firefox & Thunderbird)

RESOLVED FIXED

Status

Release Engineering
General
P2
major
RESOLVED FIXED
6 years ago
5 years ago

People

(Reporter: Michael Foerster, Assigned: catlee)

Tracking

({regression})

other
All
Linux
regression
Dependency tree / graph

Firefox Tracking Flags

(firefox18 fixed, firefox19 fixed)

Details

Attachments

(4 attachments, 1 obsolete attachment)

(Reporter)

Description

6 years ago
The update to 2012-09-13-03-05-47 happened OK on my Windows box, but failed to even show up on my Linux64 box. Browsing for the update by hand, I came upon the
Build-Log (2012-09-13-03-05-47-mozilla-central-l10n) for both Linux builds with the same error: gcc to old. 

Version shown: (line 885)
"checking compiler version... gcc (GCC) 4.1.1 20070105 (Red Hat 4.1.1-52)"

Aborted check: (line 944)
"checking for gcc pr39608... yes
This compiler would fail to build firefox, plase upgrade."

Log URL:
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2012/09/2012-09-13-03-05-47-mozilla-central-l10n/mozilla-central-linux64-l10n-nightly-de-bm25-build1-build6596.txt.gz

This maybe a fallout of the removal of GCC 4.2 support in Nighly.

I humbly ask for an upgrade of gcc in l10n (linux32+64) build system.
(Reporter)

Updated

6 years ago
Keywords: regression
(Reporter)

Comment 1

6 years ago
see also:

Bug 788276 - Properly check for broken (deprecated) compilers. 

in comment #8 by  Rafael Ávila de Espíndola (:espindola)

"add a check for gcc pr39608"

in comment #10 Gregory Szorc [:gps] gave r+ for that.

potential fallout is noted in #11
See Also: → bug 788276
(Reporter)

Comment 2

6 years ago
Sadly the discussion on bug 788276 does not bear the fruits needed.

Possible solutions:

link /usr/bin/cc and /usr/bin/gcc to the correct gcc version.

no modification of mozconfig, etc needed beyond that.

Updated

6 years ago
Depends on: 788276

Updated

6 years ago
Status: UNCONFIRMED → NEW
Ever confirmed: true
Perhaps another option would be to whitelist this specific GCC version in check script until GCC on l10n builders will be upgraded.
Any update on this issue? Currently there are no linux32/64 l10n nightly builds at all (there are only Win32/Mac builds in http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central-l10n/) and nobody step up to fix this bug.
+1 on getting an update, most of our Firefox localizers are on Linux and haven't been able to check their translation work or QA their builds for a couple of weeks now.
Created attachment 662606 [details] [diff] [review]
patch

NightlyRepackFactory inherits from BaseRepackFactory and BaseRepackFactory has a getMozconfig that copies self.mozconfig if it is defined, so the fix might be as simple as this patch, but I have no idea how to test it.

I did check that configure correctly merges arguments passed on the command line and in the .mozconfig, so while we can/should move things like 'l10n_extra_configure_args to dedicated mozconfig files, we don't need to do it right now.
Attachment #662606 - Flags: review?(catlee)
(Assignee)

Comment 7

6 years ago
Comment on attachment 662606 [details] [diff] [review]
patch

Review of attachment 662606 [details] [diff] [review]:
-----------------------------------------------------------------

yeah, this is the simplest solution...but it needs a lot of testing to make sure it doesn't bust on any branches/platforms.

I'm working on another patch at the moment which uses the in-tree mozconfigs instead of the ones in buildbot-configs. This is more desirable for releng because it means far fewer headaches when code merges between branches, but also requires the same amount of testing.

I hope to have this completed next week.
Attachment #662606 - Flags: review?(catlee)
Thanks!
Assignee: nobody → catlee
This is also affecting the Thunderbird builders, although somehow the Firefox ones seem to be passing atm.
Summary: Nightly l10n repacks failing on Linux(32+64) due to old GCC (vers. 4.1.1) → Nightly l10n repacks failing on Linux(32+64) due to old GCC (vers. 4.1.1) (Firefox & Thunderbird)
(Assignee)

Comment 10

6 years ago
Created attachment 664978 [details] [diff] [review]
Use in-tree l10n-mozconfig for dep/nightly repacks

This also addresses bug 741756. By moving the 'l10n_revision' into the poller, we ensure that each dep repack gets the proper revision.

The in-tree mozconfigs will need some adjustment as well.
Attachment #664978 - Flags: feedback?(bhearsum)
(Assignee)

Comment 11

6 years ago
Created attachment 664980 [details] [diff] [review]
l10n-mozconfig changes

I needed the --disable-elfhack option for linux to work around this error:

/tools/gcc-4.5-0moz3/bin/g++  -Wall -Wpointer-arith -Woverloaded-virtual -Werror=return-type -Wtype-limits -Wempty-body -Wno-ctor-dtor-privacy -Wno-overlength-strings -Wno-invalid-offsetof -Wno-variadic-macros -Wcast-align -Wno-long-long -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -std=gnu++0x -pthread -pipe -fexceptions  -DNDEBUG -DTRIMMED -g -Os -freorder-blocks -finline-limit=50 -fomit-frame-pointer -fPIC -shared -Wl,-z,defs -Wl,--gc-sections -Wl,-h,test-array.so -o test-array.so -lpthread  -Wl,-z,noexecstack   -Wl,-rpath-link,/builds/slave/m-cen-lnx64-l10n-dep/build/mozilla-central/dist/bin -Wl,-rpath-link,/usr/local/lib  test-array.o -nostartfiles
===
=== If you get failures below, please file a bug describing the error
=== and your environment (compiler and linker versions), and use
=== --disable-elf-hack until this is fixed.
===
# Fail if the library doesn't have INIT_ARRAY .dynamic info
readelf -d test-array.so | grep '(INIT_ARRAY)'
 0x0000000000000019 (INIT_ARRAY)         0x207330
/builds/slave/m-cen-lnx64-l10n-dep/build/mozilla-central/build/unix/elfhack/elfhack -b -f test-array.so
/builds/slave/m-cen-lnx64-l10n-dep/build/mozilla-central/build/unix/elfhack/elfhack: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.14' not found (required by /builds/slave/m-cen-lnx64-l10n-dep/build/mozilla-central/build/unix/elfhack/elfhack)
make[4]: *** [test-array.so] Error 1

although perhaps just the --enable-stdcxx-compat would be sufficient to fix that.

The macosx-universal mozconfig changes are required so that MOZ_APP_DISPLAYNAME is set properly. We need to look for FirefoxNightly.app for nightly/dep repacks. Release repacks will look for FirefoxFirefox.app I think...is that right?
Attachment #664980 - Flags: feedback?(ted.mielczarek)
Attachment #664980 - Flags: feedback?(bhearsum)
Attachment #664978 - Flags: feedback?(bhearsum) → feedback+
Comment on attachment 664980 [details] [diff] [review]
l10n-mozconfig changes

Review of attachment 664980 [details] [diff] [review]:
-----------------------------------------------------------------

Modulo the FirefoxFirefox.app weirdness, this looks good to me.
Attachment #664980 - Flags: feedback?(bhearsum) → feedback+
(In reply to Chris AtLee [:catlee] from comment #11)
> The macosx-universal mozconfig changes are required so that
> MOZ_APP_DISPLAYNAME is set properly. We need to look for FirefoxNightly.app
> for nightly/dep repacks. Release repacks will look for FirefoxFirefox.app I
> think...is that right?

This doesn't sound right to me.....found in a release repack log:
	  set -ex; rm -rf /builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../dist/unpack.tmp; mkdir -p /builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../dist/unpack.tmp; /builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/build/package/mac_osx/unpack-diskimage /builds/slave/rel-m-rel-osx64-rpk-1/firefox.dmg /tmp/Firefox-unpack /builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../dist/unpack.tmp; rsync -a "/builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../dist/unpack.tmp/Firefox.app" firefox; test -n "/builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../dist/branding/dsstore" && rsync -a "/builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../dist/unpack.tmp/.DS_Store" "/builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../dist/branding/dsstore"; test -n "/builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../dist/branding/background.png" && rsync -a "/builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../dist/unpack.tmp/.background/`basename "/builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../dist/branding/background.png"`" "/builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../dist/branding/background.png"; test -n "/builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../dist/branding/disk.icns" && rsync -a "/builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../dist/unpack.tmp/.VolumeIcon.icns" "/builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../dist/branding/disk.icns"; rm -rf /builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../dist/unpack.tmp; if test -n "" ; then cp /builds/slave/rel-m-rel-osx64-rpk-1/firefox.dmg Firefox.tmp.dmg && hdiutil unflatten Firefox.tmp.dmg && { /Developer/Tools/DeRez -skip plst -skip blkx Firefox.tmp.dmg > "" || { rm -f Firefox.tmp.dmg && false; }; } && rm -f Firefox.tmp.dmg; fi && (cd firefox/Firefox.app/Contents/MacOS && /builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/_virtualenv/bin/python /builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/config/optimizejars.py --deoptimize /builds/slave/rel-m-rel-osx64-rpk-1/mozilla-release/obj-l10n/browser/locales/../../jarlog//en-US ./ ./ && /usr/bin/unzip -o omni.ja && rm -f components/binary.manifest && for m in components/*.manifest; do sed -e 's/^#binary-component/binary-component/' $m > tmp.manifest && mv tmp.manifest $m; done)
(Assignee)

Comment 14

6 years ago
The FirefoxFirefox.app weirdness comes from our buildbot-configs where we pass
l10n_extra_configure_args= ['--with-macbundlename-prefix=Firefox']

for mozilla-central and mozilla-aurora.

This shouldn't be a problem for release builds.
(Assignee)

Comment 15

6 years ago
I re-ran the repacks without --disable-elfhack and they work fine. Looks like just the --enable-stdcxx-compat flag is sufficient.
(Assignee)

Comment 16

6 years ago
Created attachment 665586 [details] [diff] [review]
l10n-mozconfig changes
Attachment #664980 - Attachment is obsolete: true
Attachment #664980 - Flags: feedback?(ted.mielczarek)
Attachment #665586 - Flags: review?(ted.mielczarek)
(Assignee)

Comment 17

6 years ago
Comment on attachment 664978 [details] [diff] [review]
Use in-tree l10n-mozconfig for dep/nightly repacks

good enough to ship then?

we need to land the mozconfigs on both m-c and aurora first
Attachment #664978 - Flags: review?(bhearsum)
Comment on attachment 664978 [details] [diff] [review]
Use in-tree l10n-mozconfig for dep/nightly repacks

Yup!
Attachment #664978 - Flags: review?(bhearsum) → review+
Comment on attachment 665586 [details] [diff] [review]
l10n-mozconfig changes

Review of attachment 665586 [details] [diff] [review]:
-----------------------------------------------------------------

::: browser/config/mozconfigs/macosx-universal/l10n-mozconfig
@@ +3,5 @@
> +ac_add_options --with-l10n-base=../../l10n-central
> +
> +if [ "${MOZ_UPDATE_CHANNEL}" = "release" -o "${MOZ_UPDATE_CHANNEL}" = "beta" ]; then
> +ac_add_options --enable-official-branding
> +fi

Why is this here but not in the linux mozconfigs?
Attachment #665586 - Flags: review?(ted.mielczarek) → review+
(Assignee)

Comment 20

6 years ago
(In reply to Ted Mielczarek [:ted] from comment #19)
> Comment on attachment 665586 [details] [diff] [review]
> l10n-mozconfig changes
> 
> Review of attachment 665586 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: browser/config/mozconfigs/macosx-universal/l10n-mozconfig
> @@ +3,5 @@
> > +ac_add_options --with-l10n-base=../../l10n-central
> > +
> > +if [ "${MOZ_UPDATE_CHANNEL}" = "release" -o "${MOZ_UPDATE_CHANNEL}" = "beta" ]; then
> > +ac_add_options --enable-official-branding
> > +fi
> 
> Why is this here but not in the linux mozconfigs?

For the OSX repacks on central and aurora we pass in --with-macbundlename-prefix=Firefox. If we don't disable the official branding, we end up looking for FirefoxFirefox.app instead of FirefoxNightly.app.

This simply isn't a problem for linux/win32 repacks. I could add the same code there if you like.
(Assignee)

Comment 21

6 years ago
Created attachment 667603 [details] [diff] [review]
l10n mozconfigs for comm-central
Attachment #667603 - Flags: review?(mbanner)
Comment on attachment 667603 [details] [diff] [review]
l10n mozconfigs for comm-central

Review of attachment 667603 [details] [diff] [review]:
-----------------------------------------------------------------

Ok, these are from the nightly mozconfigs, but I think that should be fine, as we are adding the specific l10n options via buildbot, and things like official branding shouldn't really matter as en-US will determine if they are official or not.

We probably need to keep a close eye on this for the first beta release though.

It does make me wonder that if the l10n specific options are provided by buildbot, why do we need l10n-mozconfigs at all, couldn't we just use nightly or release according to the branch we're on? (though this sounds like a follow-up).
Attachment #667603 - Flags: review?(mbanner) → review+
Comment on attachment 664978 [details] [diff] [review]
Use in-tree l10n-mozconfig for dep/nightly repacks

catlee, unless I'm staring wrong, this was never actually checked in -- is there a reason it wasn't?
(Assignee)

Updated

5 years ago
Priority: -- → P2
(Assignee)

Comment 24

5 years ago
Comment on attachment 667603 [details] [diff] [review]
l10n mozconfigs for comm-central

Bug caused by (feature/regressing bug #): switching to newer compilers
User impact if declined: no l10n repacks
Testing completed (on m-c, etc.): releng testing in staging
Risk to taking this patch (and alternatives if risky): busted l10n repacks
String or UUID changes made by this patch: none
Attachment #667603 - Flags: approval-mozilla-aurora?
(Assignee)

Comment 25

5 years ago
Comment on attachment 665586 [details] [diff] [review]
l10n-mozconfig changes

Bug caused by (feature/regressing bug #): switching to newer compilers
User impact if declined: no l10n repacks
Testing completed (on m-c, etc.): releng testing in staging
Risk to taking this patch (and alternatives if risky): busted l10n repacks
String or UUID changes made by this patch: none
Attachment #665586 - Flags: approval-mozilla-aurora?
Comment on attachment 667603 [details] [diff] [review]
l10n mozconfigs for comm-central

bugzilla-wise I can't approve this, but a=me.
Comment on attachment 667603 [details] [diff] [review]
l10n mozconfigs for comm-central

Please sync the CLIENT_PY_ARGS change with the changes in this file:

http://hg.mozilla.org/comm-central/rev/5a49ea8a7052

r+a=me for landing the updated patch on comm-central and comm-aurora.

This can land with DONTBUILD in both locations.
Attachment #667603 - Flags: approval-mozilla-aurora?
Comment on attachment 665586 [details] [diff] [review]
l10n-mozconfig changes

stamp.
Attachment #665586 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
(Assignee)

Updated

5 years ago
Attachment #664978 - Flags: checked-in+

Comment 31

5 years ago
This is now live.
(Assignee)

Updated

5 years ago
Blocks: 804090
(Assignee)

Updated

5 years ago
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
status-firefox18: --- → fixed
status-firefox19: --- → fixed
Blocks: 741756
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.