Closed Bug 947462 Opened 9 years ago Closed 8 years ago

please do b2g desktop linux builds in the mozilla-centos6-x86_64 chroot not the i686 one

Categories

(Release Engineering :: General, defect)

x86_64
Linux
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: tbsaunde, Assigned: glandium)

References

Details

Attachments

(1 file, 2 obsolete files)

it should probably be enough to copy the list of i686 packages we install for desktop builds into b2g-config.py and change mock_target.
This converts all required-by-build package dependencies to .i686 packages, and adds the list of extra packages needed-because-yum-sucks, picked from the "linux" platform in mozilla/config.py.

This should work but I can't guarantee that 100%. Please run on staging?
Attachment #8513303 - Flags: review?(nthomas)
Assignee: nobody → mh+mozilla
Status: NEW → ASSIGNED
Comment on attachment 8513303 [details] [diff] [review]
Use mozilla-centos6-x86_64 chroots for linux32 b2g builds

This fails mock install with:
 Error: Protected multilib versions: freetype-2.3.11-6.el6_1.8.i686 != freetype-2.4.12-6.el6.1.x86_64

The repos used are
 http://mockbuild-repos.pub.build.mozilla.org/releng/public/CentOS/6/x86_64
 http://mockbuild-repos.pub.build.mozilla.org/mirrors/centos/6/latest/updates/x86_64
 http://mockbuild-repos.pub.build.mozilla.org/mirrors/centos/6/latest/os/x86_64
which have no 2.3.11 patch level 2.9 for x86_64, but we do have a 2.4.12 in the releng repo instead. The firefox desktop builds on x64_64 seem to use 2.4.12 even though they specify 2.3.11 p2.9.

Downgrading the freetype requirement to freetype-2.3.11-6.el6_1.8 gets through mock install, but you also need to add the x64_64 package too. So r- for you to figure out the best solution.

Builds are running (or will do soon) at
http://dev-master1.build.mozilla.org:8710/builders/b2g_mozilla-central_linux32_gecko%20nightly/builds/7
http://dev-master1.build.mozilla.org:8710/builders/b2g_mozilla-central_linux32_gecko-debug%20build/builds/0
http://dev-master1.build.mozilla.org:8710/builders/b2g_mozilla-central_linux32_gecko_localizer%20nightly/builds/1
Attachment #8513303 - Flags: review?(nthomas) → review-
Let's go with 1.8 with x64_64 added.
Attachment #8514112 - Flags: review?(nthomas)
Attachment #8513303 - Attachment is obsolete: true
Comment on attachment 8514112 [details] [diff] [review]
Use mozilla-centos6-x86_64 chroots for linux32 b2g builds

Downloading B2G SDK...
wget -q -c "http://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/2014/08/2014-08-12-04-02-01-mozilla-central/b2g-34.0a1.multi.linux-x86_64.tar.bz2"
tar xjf "b2g-34.0a1.multi.linux-x86_64.tar.bz2" -C "/builds/slave/m-cen-linux32_g-ntly-000000000/build/gaia/b2g_sdk/34.0a1-2014-08-12-04-02-01"
test -f /builds/slave/m-cen-linux32_g-ntly-000000000/build/gaia/b2g_sdk/34.0a1-2014-08-12-04-02-01/b2g/xpcshell
mkdir -p /builds/slave/m-cen-linux32_g-ntly-000000000/build/gaia/build_stage
/builds/slave/m-cen-linux32_g-ntly-000000000/build/gaia/b2g_sdk/34.0a1-2014-08-12-04-02-01/b2g/xpcshell: error while loading shared libraries: libXt.so.6: cannot open shared object file: No such file or directory
make[6]: *** [app] Error 127
Attachment #8514112 - Flags: review?(nthomas) → review-
ah, b2g builds use xpcshell...
I took the mock image 58d76c6acca148a1aedcbec7fc1b8212e12807b4, got the b2g tarball with xpcshell, and installed packages until xpcshell would start... which turned out to be libXt.x86_64 only.
Attachment #8514729 - Flags: review?(nthomas)
Attachment #8514112 - Attachment is obsolete: true
Comment on attachment 8514729 [details] [diff] [review]
Use mozilla-centos6-x86_64 chroots for linux32 b2g builds

Seems fine to me, holding off on review until I've chucked at staging overnight.
Comment on attachment 8514729 [details] [diff] [review]
Use mozilla-centos6-x86_64 chroots for linux32 b2g builds

Green in staging for these builders:
 b2g_mozilla-central_linux32_gecko nightly
 b2g_mozilla-central_linux32_gecko-debug build
 b2g_mozilla-central_linux32_gecko_localizer nightly

Will we need a clobber when this takes gets merged to production ?
Attachment #8514729 - Flags: review?(nthomas) → review+
This shouldn't need a clobber.
From: https://treeherder.mozilla.org/ui/logviewer.html#?job_id=3572615&repo=mozilla-inbound



=======================================================
Fetching...
retry: Calling <function run_with_timeout at 0x7f80dead7578> with args: (['/tools/tooltool.py', '--url', 'http://runtime-binaries.pvt.build.mozilla.org/tooltool', '--overwrite', '-m', 'b2g/config/tooltool-manifests/linux32/releng.manifest', 'fetch', '-c', '/builds/tooltool_cache'], 300, None, None, False, True), kwargs: {}, attempt #1
Executing: ['/tools/tooltool.py', '--url', 'http://runtime-binaries.pvt.build.mozilla.org/tooltool', '--overwrite', '-m', 'b2g/config/tooltool-manifests/linux32/releng.manifest', 'fetch', '-c', '/builds/tooltool_cache']
INFO - rm tree: moztt
INFO - untarring "moztt.tar.bz2"
Bootstraping...
+ rm -rf sccache
+ tar -xjf sccache.tar.bz2
program finished with exit code 0
elapsedTime=3.343323
========= Finished 'sh /builds/slave/m-in-linux32_g-000000000000000/tools/scripts/tooltool/tooltool_wrapper.sh ...' (results: 0, elapsed: 3 secs) (at 2014-11-04 05:08:31.348376) =========
========= Started compile failed (results: 2, elapsed: 4 secs) (at 2014-11-04 05:08:31.348765) =========
mock_mozilla -r mozilla-centos6-x86_64 --cwd /builds/slave/m-in-linux32_g-000000000000000/build --unpriv --shell '/usr/bin/env HG_SHARE_BASE_DIR="/builds/hg-shared" LOCALES_FILE="locales/languages_dev.json" TOOLTOOL_HOME="/builds" SYMBOL_SERVER_HOST="symbolpush.mozilla.org" CCACHE_DIR="/builds/ccache" POST_SYMBOL_UPLOAD_CMD="/usr/local/bin/post-symbol-upload.py" MOZ_AUTOMATION="1" MOZ_OBJDIR="obj-firefox" SYMBOL_SERVER_SSH_KEY="/home/cltbld/.ssh/ffxbld_rsa" LOCALE_BASEDIR="/builds/slave/m-in-linux32_g-000000000000000/build-gaia-l10n" TINDERBOX_OUTPUT="1" CCACHE_COMPRESS="1" TOOLTOOL_CACHE="/builds/tooltool_cache" SYMBOL_SERVER_PATH="/mnt/netapp/breakpad/symbols_ffx/" PATH="/tools/python27-mercurial/bin:/tools/python27/bin:${PATH}:/tools/buildbot/bin" MOZ_CRASHREPORTER_NO_REPORT="1" SYMBOL_SERVER_USER="ffxbld" WGET_OPTS="-q -c" LC_ALL="C" CCACHE_UMASK="002" make -f client.mk build '"'"'MOZ_BUILD_DATE=20141104050144'"'"''
in dir /builds/slave/m-in-linux32_g-000000000000000/build (timeout 10800 secs) (maxTime 19800 secs)
watching logfiles {}
argv: ['mock_mozilla', '-r', 'mozilla-centos6-x86_64', '--cwd', '/builds/slave/m-in-linux32_g-000000000000000/build', '--unpriv', '--shell', u'/usr/bin/env HG_SHARE_BASE_DIR="/builds/hg-shared" LOCALES_FILE="locales/languages_dev.json" TOOLTOOL_HOME="/builds" SYMBOL_SERVER_HOST="symbolpush.mozilla.org" CCACHE_DIR="/builds/ccache" POST_SYMBOL_UPLOAD_CMD="/usr/local/bin/post-symbol-upload.py" MOZ_AUTOMATION="1" MOZ_OBJDIR="obj-firefox" SYMBOL_SERVER_SSH_KEY="/home/cltbld/.ssh/ffxbld_rsa" LOCALE_BASEDIR="/builds/slave/m-in-linux32_g-000000000000000/build-gaia-l10n" TINDERBOX_OUTPUT="1" CCACHE_COMPRESS="1" TOOLTOOL_CACHE="/builds/tooltool_cache" SYMBOL_SERVER_PATH="/mnt/netapp/breakpad/symbols_ffx/" PATH="/tools/python27-mercurial/bin:/tools/python27/bin:${PATH}:/tools/buildbot/bin" MOZ_CRASHREPORTER_NO_REPORT="1" SYMBOL_SERVER_USER="ffxbld" WGET_OPTS="-q -c" LC_ALL="C" CCACHE_UMASK="002" make -f client.mk build \'MOZ_BUILD_DATE=20141104050144\'']
environment:
CCACHE_COMPRESS=1
CCACHE_DIR=/builds/ccache
CCACHE_HASHDIR=
CCACHE_UMASK=002
CVS_RSH=ssh
G_BROKEN_FILENAMES=1
HG_SHARE_BASE_DIR=/builds/hg-shared
HISTCONTROL=ignoredups
HISTSIZE=1000
HOME=/home/cltbld
HOSTNAME=bld-linux64-spot-008.build.releng.use1.mozilla.com
LANG=en_US.UTF-8
LC_ALL=C
LESSOPEN=|/usr/bin/lesspipe.sh %s
LOCALES_FILE=locales/languages_dev.json
LOCALE_BASEDIR=/builds/slave/m-in-linux32_g-000000000000000/build-gaia-l10n
LOGNAME=cltbld
MAIL=/var/spool/mail/cltbld
MOZ_AUTOMATION=1
MOZ_CRASHREPORTER_NO_REPORT=1
MOZ_OBJDIR=obj-firefox
PATH=/tools/python27-mercurial/bin:/tools/python27/bin:/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin:/tools/buildbot/bin
POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py
PWD=/builds/slave/m-in-linux32_g-000000000000000/build
SHELL=/bin/bash
SHLVL=1
SYMBOL_SERVER_HOST=symbolpush.mozilla.org
SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_ffx/
SYMBOL_SERVER_SSH_KEY=/home/cltbld/.ssh/ffxbld_rsa
SYMBOL_SERVER_USER=ffxbld
TERM=linux
TINDERBOX_OUTPUT=1
TMOUT=86400
TOOLTOOL_CACHE=/builds/tooltool_cache
TOOLTOOL_HOME=/builds
USER=cltbld
WGET_OPTS=-q -c
_=/tools/buildbot/bin/python
using PTY: False
INFO: mock_mozilla.py version 1.0.3 starting...
State Changed: init plugins
INFO: selinux disabled
State Changed: start
State Changed: lock buildroot
State Changed: shell
( echo 'export MOZ_AUTOMATION_BUILD_SYMBOLS=1'; echo 'export MOZ_AUTOMATION_L10N_CHECK=1'; echo 'export MOZ_AUTOMATION_PACKAGE=1'; echo 'export MOZ_AUTOMATION_PACKAGE_TESTS=1'; echo 'export MOZ_AUTOMATION_INSTALLER=0'; echo 'export MOZ_AUTOMATION_UPDATE_PACKAGING=0'; echo 'export MOZ_AUTOMATION_UPLOAD=1'; echo 'export MOZ_AUTOMATION_UPLOAD_SYMBOLS=0'; echo 'export MOZ_AUTOMATION_BUILD_SYMBOLS=1'; echo 'export MOZ_AUTOMATION_L10N_CHECK=1'; echo 'export MOZ_AUTOMATION_PACKAGE=1'; echo 'export MOZ_AUTOMATION_PACKAGE_TESTS=1'; echo 'export MOZ_AUTOMATION_INSTALLER=0'; echo 'export MOZ_AUTOMATION_UPDATE_PACKAGING=0'; echo 'export MOZ_AUTOMATION_UPLOAD=1'; echo 'export MOZ_AUTOMATION_UPLOAD_SYMBOLS=0'; echo 'export SCCACHE_BUCKET=mozilla-releng-s3-cache-us-east-1-prod'; echo 'export SCCACHE_NAMESERVER=169.254.169.253'; echo 'export UPLOAD_EXTRA_FILES+=sccache.log.gz'; ) > /builds/slave/m-in-linux32_g-000000000000000/build/obj-firefox/.mozconfig.mk
make -f /builds/slave/m-in-linux32_g-000000000000000/build/client.mk realbuild CREATE_MOZCONFIG_JSON=
Adding client.mk options from /builds/slave/m-in-linux32_g-000000000000000/build/.mozconfig:
AUTOCLOBBER=1
export MOZ_AUTOMATION_BUILD_SYMBOLS=1
export MOZ_AUTOMATION_L10N_CHECK=1
export MOZ_AUTOMATION_PACKAGE=1
export MOZ_AUTOMATION_PACKAGE_TESTS=1
export MOZ_AUTOMATION_INSTALLER=0
export MOZ_AUTOMATION_UPDATE_PACKAGING=0
export MOZ_AUTOMATION_UPLOAD=1
export MOZ_AUTOMATION_UPLOAD_SYMBOLS=0
AUTOCLOBBER=1
export MOZ_AUTOMATION_BUILD_SYMBOLS=1
export MOZ_AUTOMATION_L10N_CHECK=1
export MOZ_AUTOMATION_PACKAGE=1
export MOZ_AUTOMATION_PACKAGE_TESTS=1
export MOZ_AUTOMATION_INSTALLER=0
export MOZ_AUTOMATION_UPDATE_PACKAGING=0
export MOZ_AUTOMATION_UPLOAD=1
export MOZ_AUTOMATION_UPLOAD_SYMBOLS=0
CONFIG_GUESS=i686-pc-linux
export SCCACHE_BUCKET=mozilla-releng-s3-cache-us-east-1-prod
export SCCACHE_NAMESERVER=169.254.169.253
MOZ_PREFLIGHT_ALL+=build/sccache.mk
MOZ_POSTFLIGHT_ALL+=build/sccache.mk
export UPLOAD_EXTRA_FILES+=sccache.log.gz
MOZ_OBJDIR=/builds/slave/m-in-linux32_g-000000000000000/build/obj-firefox
OBJDIR=/builds/slave/m-in-linux32_g-000000000000000/build/obj-firefox
FOUND_MOZCONFIG=/builds/slave/m-in-linux32_g-000000000000000/build/.mozconfig
make[1]: Entering directory `/builds/slave/m-in-linux32_g-000000000000000/build'
set -e; \
for mkfile in build/sccache.mk; do \
make -f /builds/slave/m-in-linux32_g-000000000000000/build/$mkfile preflight_all TOPSRCDIR=/builds/slave/m-in-linux32_g-000000000000000/build OBJDIR=/builds/slave/m-in-linux32_g-000000000000000/build/obj-firefox MOZ_OBJDIR=/builds/slave/m-in-linux32_g-000000000000000/build/obj-firefox; \
done
make[2]: Entering directory `/builds/slave/m-in-linux32_g-000000000000000/build'
# Terminate any sccache server that might still be around
python2.7 /builds/slave/m-in-linux32_g-000000000000000/build/sccache/sccache.py > /dev/null 2>&1
make[2]: Leaving directory `/builds/slave/m-in-linux32_g-000000000000000/build'
Generating /builds/slave/m-in-linux32_g-000000000000000/build/configure using autoconf
cd /builds/slave/m-in-linux32_g-000000000000000/build; /usr/bin/autoconf-2.13
Generating /builds/slave/m-in-linux32_g-000000000000000/build/js/src/configure using autoconf
cd /builds/slave/m-in-linux32_g-000000000000000/build/js/src; /usr/bin/autoconf-2.13
make[2]: Entering directory `/builds/slave/m-in-linux32_g-000000000000000/build'
cp /builds/slave/m-in-linux32_g-000000000000000/build/.mozconfig /builds/slave/m-in-linux32_g-000000000000000/build/obj-firefox/.mozconfig
cd /builds/slave/m-in-linux32_g-000000000000000/build/obj-firefox
/builds/slave/m-in-linux32_g-000000000000000/build/configure
Adding configure options from /builds/slave/m-in-linux32_g-000000000000000/build/.mozconfig
--enable-crashreporter
--enable-release
--enable-crashreporter
--enable-release
--enable-elf-hack
--enable-stdcxx-compat
--target=i686-pc-linux
--host=i686-pc-linux
--x-libraries=/usr/lib
--enable-update-channel=
--enable-update-packaging
--enable-signmar
--with-compiler-wrapper=python2.7 /builds/slave/m-in-linux32_g-000000000000000/build/sccache/sccache.py
--enable-application=b2g
--disable-elf-hack
loading cache ./config.cache
checking host system type... i686-pc-linux-gnu
checking target system type... i686-pc-linux-gnu
checking build system type... i686-pc-linux-gnu
checking for gawk... (cached) gawk
checking for python2.7... (cached) /tools/python27/bin/python2.7
Creating Python environment
checking Python environment is Mozilla virtualenv... /builds/slave/m-in-linux32_g-000000000000000/build/obj-firefox/_virtualenv/bin/python: error while loading shared libraries: libpython2.7.so.1.0: wrong ELF class: ELFCLASS64
configure: error: Python environment does not appear to be sane.
------ config.log ------
This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake.
configure:1185: checking host system type
configure:1206: checking target system type
configure:1224: checking build system type
configure:1300: checking for gawk
configure:1385: checking for python2.7
configure:1495: checking Python environment is Mozilla virtualenv
configure: error: Python environment does not appear to be sane.
*** Fix above errors and then restart with\
"make -f client.mk build"
make[2]: *** [configure] Error 1
make[2]: Leaving directory `/builds/slave/m-in-linux32_g-000000000000000/build'
make[1]: *** [/builds/slave/m-in-linux32_g-000000000000000/build/obj-firefox/config.status] Error 2
make[1]: Leaving directory `/builds/slave/m-in-linux32_g-000000000000000/build'
make: *** [build] Error 2
State Changed: unlock buildroot
program finished with exit code 2
elapsedTime=4.701965
========= Finished compile failed (results: 2, elapsed: 4 secs) (at 2014-11-04 05:08:36.072368) =========
========= Skipped (results: not started, elapsed: not started) =========
This error appears to be about a python 32 bit interpreter trying to use a python 64 bit library - I think this is likely to be fall out from this bug, so will roll back, and reconfig.

In addition, this bug got picked up in a reconfig, but bugzilla API failed to update this bug to say so - have raised bug 1093600 about this.
also that this hit in https://treeherder.mozilla.org/ui/#/jobs?repo=mozilla-inbound&revision=1dd40ce6fc5a only the B2g Linux Builds opt and debug , Builds like Linux opt are ok
Reconfig in progress
Attachment #8514729 - Flags: checked-in-
pmoore was wondering if this just needed a clobber instead. That looks likely given that it's the objdir/_virtualenv/python that's failing, since it copies the system python into the objdir when setting up the virtualenv. I doubt it knows to copy the new binary when the system python changes.
(In reply to Michael Shal [:mshal] from comment #16)
> pmoore was wondering if this just needed a clobber instead. That looks
> likely given that it's the objdir/_virtualenv/python that's failing, since
> it copies the system python into the objdir when setting up the virtualenv.
> I doubt it knows to copy the new binary when the system python changes.

In fairness, I was just stealing nthomas's idea in comment 8 =)
So to summarize - I backed this out, reconfig'd, and then Tomcat clobbered, and all is ok again.

The fact Tomcat needed to clobber after the backout also suggests that a clobber was probably necessary. Therefore, it might be worth landing this change again, and then coordinating a clobber with a reconfig, to see if that fixes everything.
Indeed, this looks very much like it needs a clobber of the 32-bits b2g builders. Relanded.

https://hg.mozilla.org/build/buildbot-configs/rev/d029e3d04215
Attachment #8514729 - Flags: checked-in- → checked-in+
Reconfiging now... We will clobber after reconfig completes.
Tomcat has clobbered
Blocks: 1075183
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Blocks: 1119997
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.