816793 - switch Linux B2G Desktop builds to CentOS 6 build environment

Status: RESOLVED WORKSFORME

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: use CentOS 6 for B2G desktop builds

We should be building linux desktop builds to the same environment that the Firefox nightly builds use.

I've tested that this environment is able to build desktop b2g, and Myk and I tested that the resulting builds fix the underlying issue that is being experienced without regressing what currently works.  I've tested this patch with ./setup-masters -t -R build and it passes.

I'm not sure who should review this change, but it's relatively important that it get in soon, since it's causing a lot of trouble for r2d2b2g users.

Comment 1

[Myk Melez [:myk] [@mykmelez]]

John: who's the best reviewer for this change?

Comment 2

[Myk Melez [:myk] [@mykmelez]]

Erm, in case it was confusing, as there are two Johns on this bug, the question in comment 1 was directed to joduinn!

Comment 3

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 686872 [details] [diff] [review]
use CentOS 6 for B2G desktop builds

Callek suggested on IRC that you might be an appropriate reviewer.  If that's not the case, or if there is a more appropriate reviewer, please feel free to reassign.

Comment 4

[Rail Aliiev [:rail]]

Comment on attachment 686872 [details] [diff] [review]
use CentOS 6 for B2G desktop builds

Review of attachment 686872 [details] [diff] [review]:
-----------------------------------------------------------------

Please s/mercurial-python27-mercurial/mozilla-python27-mercurial/g before you land this change.

Comment 5

[John Ford [:jhford] CET/CEST Berlin Time]

http://hg.mozilla.org/build/buildbot-configs/rev/1e2bf1b3746b

Comment 6

[[e-mail] (:bhearsum)]

in production now.

Comment 7

[[e-mail] (:bhearsum)]

Comment on attachment 686872 [details] [diff] [review]
use CentOS 6 for B2G desktop builds

Apologies, but I had to back this out because it broke the gaia multilocale stuff that I just landed. Specifically, 'hg' can't be found when trying to clone:
10:32:46     INFO - #####
10:32:46     INFO - ##### Running checkout-gaia-l10n step.
10:32:46     INFO - #####
10:32:46     INFO - mkdir: /builds/slave/m-cen-linux32-gecko-ntly/build-gaia-l10n
10:32:46     INFO - Changing directory to /builds/slave/m-cen-linux32-gecko-ntly/build-gaia-l10n.
10:32:46     INFO - Setting /builds/slave/m-cen-linux32-gecko-ntly/build-gaia-l10n/fr to https://hg.mozilla.org/gaia-l10n/fr using shared directory /builds/hg-shared.
10:32:46     INFO - Checking if share extension works.
10:32:46     INFO - Getting output from command: ['hg', '--config', 'ui.merge=internal:merge', 'help', 'share']
10:32:46     INFO - Copy/paste: hg --config ui.merge=internal:merge help share
Traceback (most recent call last):
  File "mozharness/scripts/clone_gaia_locales.py", line 94, in <module>
    myScript.run()
  File "/builds/slave/m-cen-linux32-gecko-ntly/mozharness/mozharness/base/script.py", line 726, in run
    self._possibly_run_method(method_name, error_if_missing=True)
  File "/builds/slave/m-cen-linux32-gecko-ntly/mozharness/mozharness/base/script.py", line 683, in _possibly_run_method
    return getattr(self, method_name)()
  File "mozharness/scripts/clone_gaia_locales.py", line 88, in checkout_gaia_l10n
    self.pull_gaia_locale_source(l10n_config, parse_config_file(languages_file).keys(), l10n_base_dir)
  File "/builds/slave/m-cen-linux32-gecko-ntly/mozharness/mozharness/mozilla/l10n/locales.py", line 199, in pull_gaia_locale_source
    self.vcs_checkout_repos(repo_list=repos, parent_dir=base_dir)
  File "/builds/slave/m-cen-linux32-gecko-ntly/mozharness/mozharness/base/vcs/vcsbase.py", line 95, in vcs_checkout_repos
    revision_list.append(self.vcs_checkout(**kwargs))
  File "/builds/slave/m-cen-linux32-gecko-ntly/mozharness/mozharness/base/vcs/vcsbase.py", line 66, in vcs_checkout
    got_revision = vcs_obj.ensure_repo_and_revision()
  File "/builds/slave/m-cen-linux32-gecko-ntly/mozharness/mozharness/base/vcs/mercurial.py", line 440, in ensure_repo_and_revision
    if share_base and not self.query_can_share():
  File "/builds/slave/m-cen-linux32-gecko-ntly/mozharness/mozharness/base/vcs/mercurial.py", line 304, in query_can_share
    throw_exception=True)
  File "/builds/slave/m-cen-linux32-gecko-ntly/mozharness/mozharness/base/script.py", line 582, in get_output_from_command
    cwd=cwd, stderr=tmp_stderr, env=env)
  File "/usr/lib/python2.6/subprocess.py", line 639, in __init__
    errread, errwrite)
  File "/usr/lib/python2.6/subprocess.py", line 1228, in _execute_child
    raise child_exception

Comment 8

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: add hg location to path

This patch adds the HG custom location to the path so that HG is found.  I expanded the changes to cover the localization builders, but please ensure that this is valid, not sure what's going on there.

Comment 9

[[e-mail] (:bhearsum)]

Comment on attachment 688388 [details] [diff] [review]
add hg location to path

Didn't do the trick, unfortunately. Still getting the same error.

Comment 10

[John Ford [:jhford] CET/CEST Berlin Time]

The issue here is that the mozharness script needs to pass through the /tools/python27-mercurial/bin directory (or link to hg script) to the mozharness vcs routines.

Comment 11

[John Ford [:jhford] CET/CEST Berlin Time]

We think that bug 812836 has something related to a fix here.

Comment 12

[John Ford [:jhford] CET/CEST Berlin Time]

This seems to be an issue somewhere between buildbot and mozharness or mozharness and calling hg.  I don't really have an environment to test this.

Unassigning myself for someone with a testing environment to take a look into this.

As far as the product/build concerns go, a mock environment produced with the packages specified in my patch result in a valid and working build that the r2d2b2g team has tested and validated.

Comment 13

[Myk Melez [:myk] [@mykmelez]]

joduinn: who has a suitable testing environment and can look at this issue to figure out why the existing patches are regressing other builds?

Comment 14

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

myk: Usual process would be for jhford to file bug asking for access to loaner machine, and from bug traffic so far, he seems to have most context here, so seems best person to continue on. I'd be happy to file the bug if its helps.

If this is something you want RelEng to now take on, we can look at this after we get ahead of our blocking-basecamp+ bugs. I dont have context on urgency, so if this should be blocking-basecamp, please nom.

Comment 15

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to John O'Duinn [:joduinn] from comment #14)
> myk: Usual process would be for jhford to file bug asking for access to
> loaner machine, and from bug traffic so far, he seems to have most context
> here, so seems best person to continue on. I'd be happy to file the bug if
> its helps.

All of the context that I have for this bug has been expressed in the patch that I've attached, see comment 12.  This issue is in the releng automation, which would require a full master and slave to be set up to properly test this.

> If this is something you want RelEng to now take on, we can look at this
> after we get ahead of our blocking-basecamp+ bugs. I dont have context on
> urgency, so if this should be blocking-basecamp, please nom.

Comment 16

[Myk Melez [:myk] [@mykmelez]]

This is a development blocker, as it means the Linux builds don't work at all for many Linux users.  I'm not sure if it should block Basecamp, but it does hurt our efforts to get developers to build apps for Basecamp devices, especially down in Brazil, where Linux is popular.  Another Linux development blocker, bug 789399, was marked blocking-basecamp+.  Nominating this one.

Comment 17

[Andrew Overholt [:overholt]]

(In reply to Myk Melez [:myk] [@mykmelez] from comment #16)
> This is a development blocker, as it means the Linux builds don't work at
> all for many Linux users.

What makes this the case?  Is it the versions of the libraries against which we're linking?

We discussed this during triage and aren't quite convinced it's worth holding the release for.

Comment 18

[Myk Melez [:myk] [@mykmelez]]

(In reply to Andrew Overholt [:overholt] from comment #17)
> (In reply to Myk Melez [:myk] [@mykmelez] from comment #16)
> > This is a development blocker, as it means the Linux builds don't work at
> > all for many Linux users.
> 
> What makes this the case?  Is it the versions of the libraries against which
> we're linking?

Yes, the Linux builds are currently built on Fedora 16 with glibc 2.14, which means they only work on installations with glibc 2.14 or newer.  That excludes a number of Linux distributions, including relatively new ones like Ubuntu 11.10 (glibc 2.13) as well as older supported ones like Ubuntu 10.04 LTS.

See https://github.com/mozilla/r2d2b2g/issues/105 for the range of Linux users reporting problems starting B2G Desktop via the Simulator (although some of the users commenting on the issue may be experiencing other problems).

Firefox nightlies are built on CentOS 16 with glibc 2.12, which covers a broader range of distributions, and this bug is about building B2G Desktop on the same build systems, so B2G Desktop nightlies work everywhere that Firefox nightlies do.

Comment 19

[Andrew Overholt [:overholt]]

(In reply to Myk Melez [:myk] [@mykmelez] from comment #18)
> (In reply to Andrew Overholt [:overholt] from comment #17)
> > (In reply to Myk Melez [:myk] [@mykmelez] from comment #16)
> > > This is a development blocker, as it means the Linux builds don't work at
> > > all for many Linux users.
> > 
> > What makes this the case?  Is it the versions of the libraries against which
> > we're linking?
> 
> Yes, the Linux builds are currently built on Fedora 16 with glibc 2.14,
> [...]
> Firefox nightlies are built on CentOS 16 with glibc 2.12

Thanks for the clarification.  We will discuss this again in tomorrow's triage since Fabrice said he wanted to take a look at the B2G desktop issues today.

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

s/CentOS 16/CentOS 6/ in comment 18 and 19.

Comment 21

[Andrew Overholt [:overholt]]

This isn't something for which we can hold the B2G v1 release.  We discussed it during triage and are all in favour of supporting as many (older) distros as possible, so let's do this.  Will something need to be uplifted or is it a server/tbpl-side change?

Comment 22

[Chris AtLee [:catlee]]

Attachment: use centos6 for b2g desktop/gecko builds

This WFM in staging. Requires a clobber so that configure picks up the new python.

Comment 23

[Rail Aliiev [:rail]]

Comment on attachment 749879 [details] [diff] [review]
use centos6 for b2g desktop/gecko builds

Woot! Die F16 repos, die!

Comment 24

[Mike Hommey [:glandium]]

Considering we're switching linux 32-bits builds to 64-bits environment, you might as well do that at the same time.

Comment 25

[Chris AtLee [:catlee]]

good point, I'll make that happen

Comment 26

[Chris AtLee [:catlee]]

Attachment: use centos6 for b2g desktop/gecko builds

leaving 32-bit builds in a 32-bit env for now. there were some errors when switching them to the 64-bit env that I don't have time to track down.

Comment 27

[Armen [:armenzg]]

Live in production.

Comment 28

[Chris AtLee [:catlee]]

Desktop builds are now being done in CentOS 6 environments. I'm going look into doing the 32-bit builds in a 64-bit environment.

Comment 29

[Chris AtLee [:catlee]]

I took a stab at this this morning, and got as far as building gaia. gaia fails because it downloads the 64-bit xulrunner and then tries to run it, but we don't have the 64-bit libs installed in the chroot.

Any preferences on if we should download the 32-bit xulrunner instead, or install the 64-bit libs required by the 64-bit xulrunner?

Comment 30

[Chris AtLee [:catlee]]

Attachment: include linux32 mozconfig for 32-bit b2g gecko builds

I think this is safe to land now. When we switch the builds over to 64-bit, the logic in build/unix/mozconfig.linux32 will kick in and add the 32-bit specific compiler flags.

Comment 31

[Chris AtLee [:catlee]]

Attachment: switch 32-bit linux b2g gecko builds to use 64-bit mock env.

the WGET_OPTS makes wget be quiet when fetching the xulrunner SDK

Comment 32

[Justin Wood (:Callek)]

Comment on attachment 766068 [details] [diff] [review]
switch 32-bit linux b2g gecko builds to use 64-bit mock env.

pep8 enforcer here:

/tmp/tmp.0qwQLUb3nA/mozilla/b2g_config.py:231:26: E502 the backslash is redundant between brackets
        'mock_packages': \
                         ^
/tmp/tmp.0qwQLUb3nA/mozilla/b2g_config.py:232:21: E126 continuation line over-indented for hanging indent
                    ['autoconf213', 'python', 'zip', 'mozilla-python27-mercurial', 'git', 'ccache',
                    ^
/tmp/tmp.0qwQLUb3nA/mozilla/b2g_config.py:233:21: E128 continuation line under-indented for visual indent
                    'glibc-static.i686', 'libstdc++-static.i686', 'perl-Test-Simple', 'perl-Config-General',
                    ^
/tmp/tmp.0qwQLUb3nA/mozilla/b2g_config.py:268:21: E124 closing bracket does not match visual indentation
                    ],
                    ^
/tmp/tmp.0qwQLUb3nA/mozilla/b2g_config.py:515:26: E502 the backslash is redundant between brackets
        'mock_packages': \
                         ^
/tmp/tmp.0qwQLUb3nA/mozilla/b2g_config.py:516:21: E126 continuation line over-indented for hanging indent
                    ['autoconf213', 'python', 'zip', 'mozilla-python27-mercurial', 'git', 'ccache',
                    ^
/tmp/tmp.0qwQLUb3nA/mozilla/b2g_config.py:517:21: E128 continuation line under-indented for visual indent
                    'glibc-static.i686', 'libstdc++-static.i686', 'perl-Test-Simple', 'perl-Config-General',
                    ^
/tmp/tmp.0qwQLUb3nA/mozilla/b2g_config.py:552:21: E124 closing bracket does not match visual indentation
                    ],
                    ^

Comment 33

[Justin Wood (:Callek)]

Comment on attachment 766068 [details] [diff] [review] [diff] [review]
switch 32-bit linux b2g gecko builds to use 64-bit mock env.

---------AUTOMATIC COMMENT---------
--  filter pep8-callek-june-2013 --
---------AUTOMATIC COMMENT---------

$pep8 <dir> --diff --max-line-length=159 --show-source < attachment.diff
/tmp/tmp.dz8xrIE2tB/mozilla/b2g_config.py:231:26: E502 the backslash is redundant between brackets
        'mock_packages': \
                         ^
/tmp/tmp.dz8xrIE2tB/mozilla/b2g_config.py:232:21: E126 continuation line over-indented for hanging indent
                    ['autoconf213', 'python', 'zip', 'mozilla-python27-mercurial', 'git', 'ccache',
                    ^
/tmp/tmp.dz8xrIE2tB/mozilla/b2g_config.py:233:21: E128 continuation line under-indented for visual indent
                    'glibc-static.i686', 'libstdc++-static.i686', 'perl-Test-Simple', 'perl-Config-General',
                    ^
/tmp/tmp.dz8xrIE2tB/mozilla/b2g_config.py:268:21: E124 closing bracket does not match visual indentation
                    ],
                    ^
/tmp/tmp.dz8xrIE2tB/mozilla/b2g_config.py:515:26: E502 the backslash is redundant between brackets
        'mock_packages': \
                         ^
/tmp/tmp.dz8xrIE2tB/mozilla/b2g_config.py:516:21: E126 continuation line over-indented for hanging indent
                    ['autoconf213', 'python', 'zip', 'mozilla-python27-mercurial', 'git', 'ccache',
                    ^
/tmp/tmp.dz8xrIE2tB/mozilla/b2g_config.py:517:21: E128 continuation line under-indented for visual indent
                    'glibc-static.i686', 'libstdc++-static.i686', 'perl-Test-Simple', 'perl-Config-General',
                    ^
/tmp/tmp.dz8xrIE2tB/mozilla/b2g_config.py:552:21: E124 closing bracket does not match visual indentation
                    ],
                    ^

Comment 34

[Chris AtLee [:catlee]]

Comment on attachment 766064 [details] [diff] [review]
include linux32 mozconfig for 32-bit b2g gecko builds

it hurtses us!

Comment 35

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Found in triage, moving to correct component. 

(In reply to Chris AtLee [:catlee] from comment #29)
> I took a stab at this this morning, and got as far as building gaia. gaia
> fails because it downloads the 64-bit xulrunner and then tries to run it,
> but we don't have the 64-bit libs installed in the chroot.
> 
> Any preferences on if we should download the 32-bit xulrunner instead, or
> install the 64-bit libs required by the 64-bit xulrunner?

jhford: do you have a preference?

Comment 36

[John Ford [:jhford] CET/CEST Berlin Time]

If you guys are already using USE_LOCAL_XULRUNNER_SDK, I don't see any problem with using a 32-bit xulrunner.  If you aren't, it's probably easier to just install the 64bit runtime deps, most should be there as you're already in a 64-bit environment.

The build system is supposed to do the same thing on all platforms, so you should be fine with either xulrunner package.  I have no preference which option is used.

Comment 37

[John Ford [:jhford] CET/CEST Berlin Time]

If you aren't using USE_LOCAL_XULLRUNNER_SDK, we could also do something like:

diff --git a/Makefile b/Makefile
index 6974098..3d5f48b 100644
--- a/Makefile
+++ b/Makefile
@@ -145,11 +145,15 @@ SHELL := /bin/bash

 # what OS are we on?
 SYS=$(shell uname -s)
+ifeq ($(FORCE_ARCH),)
 ARCH?=$(shell uname -m)
 MSYS_FIX=
 ifeq (${SYS}/${ARCH},Darwin/i386)
 ARCH=x86_64
 endif
+else
+ARCH:=$(FORCE_ARCH)
+endif # FORCE_ARCH
 SEP=/
 SEP_FOR_SED=/
 ifneq (,$(findstring MINGW32_,$(SYS)))

in Gaia and something in Gecko to set FORCE_ARCH correctly