857697 - Switch 32-bits linux builds to x86-64 mock environments

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Trevor Saunders (:tbsaunde)]

we'd like to test cross compiling x86 linux builds on x86_64 to do that we need the follow packages installed on the x86_64 build slaves.

     * glibc-devel.i686
     * cairo-devel.i686
     * fontconfig-devel.i686
     * gtk2-devel.i686
     * dbus-glib-devel.i686
     * glib2-devel.i686
     * gdk-pixbuf2-devel.i686
     * pango-devel.i686
     * pixman-devel.i686
     * freetype-devel.i686
     * libpng-devel.i686
     * libXrender-devel.i686
     * libX11-devel.i686
     * libxcb-devel.i686
     * libXau-devel.i686
     * atk-devel.i686
     * libnotify-devel.i686
     * dbus-devel.i686
     * libcurl-devel.i686
     * libXt-devel.i686
     * libXext-devel.i686
     * libstdc++-devel.i686
     * zlib-devel.i686

Comment 1

[Armen [:armenzg]]

Would you mind borrowing one of our build machines and confirming that those packages would allow us to do cross-compilation?

Comment 2

[Trevor Saunders (:tbsaunde)]

(In reply to Armen Zambrano G. [:armenzg] from comment #1)
> Would you mind borrowing one of our build machines and confirming that those
> packages would allow us to do cross-compilation?

sure,  fwiw I did check on a fresh centos6 chroot so I really expect it will work.

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

found in triage.

Comment 4

[Justin Wood (:Callek)]

I got nudged to put-in-bug my privately stated concern with this ask.

"I'm mostly concerned about possibly affecting our official linux64 builds by adding the packages, but I can't say for sure one way or the other" I'll leave it to you guys to either dismiss disprove or otherwise that though.

Comment 5

[Trevor Saunders (:tbsaunde)]

ok, So I finished up testing stuff here, it turns out we need two more packages than I mentioned in comment 0 we also need alsa-lib-devel.i686 and libgcc-devel.i686

Comment 6

[Trevor Saunders (:tbsaunde)]

> "I'm mostly concerned about possibly affecting our official linux64 builds
> by adding the packages, but I can't say for sure one way or the other" I'll
> leave it to you guys to either dismiss disprove or otherwise that though.

Its certainly possible, in theory any change we make could do that, but this might be a little more likely.  However I'd think the chances are fairly remote and we should be able to check to be sure nothing actually broke.  I checked that after installing the i686 packages I could still do a build using the linux64 nightly mozconfig with gcc-4.7, but I didn't run any tests on that build.

Comment 7

[Mike Hommey [:glandium]]

Except if some .x86-64 packages are installed, .i686 packages have no reason to alter the environment when used for .x86-64 type of stuff, which is what they are used for. Even more so for -devel.i686 packages.

Comment 8

[Armen [:armenzg]]

Out of ignorance, why do we want to do cross compilation rather than staying with the current mock setup we have? (IIUC we use mock to be able to compile the 32-bit builds ona 64-bit machine - that's as far as I know and I'm not sure)

In other words, what is the ultimate goal?

For testing purposes, we could put the machine on staging and make it trigger jobs on a staging master with a 32-bit and 64-bit testers could test the builds.

Comment 9

[Mike Hommey [:glandium]]

32-bits builds OOM with PGO with gcc 4.7. Which means going forward, we need the extra memory for the toolchain.

Comment 10

[Trevor Saunders (:tbsaunde)]

(In reply to Armen Zambrano G. [:armenzg] from comment #8)
> Out of ignorance, why do we want to do cross compilation rather than staying
> with the current mock setup we have? (IIUC we use mock to be able to compile
> the 32-bit builds ona 64-bit machine - that's as far as I know and I'm not
> sure)
> 
> In other words, what is the ultimate goal?
> 
> For testing purposes, we could put the machine on staging and make it
> trigger jobs on a staging master with a 32-bit and 64-bit testers could test
> the builds.

we could certainly test more, but me and glandium decided we both think the risk is too low to spend the effort, we should just get those packages shipped on the slaves so we can move forward with changing compiler to stop the pain of gcc 4.5 crashing in PGO builds.

Comment 11

[Mike Hommey [:glandium]]

Attachment: Install necessary i686 -devel packages on linux64 build slaves (try only)

Comment 12

[Justin Wood (:Callek)]

Comment on attachment 736490 [details] [diff] [review]
Install necessary i686 -devel packages on linux64 build slaves (try only)

Review of attachment 736490 [details] [diff] [review]:
-----------------------------------------------------------------

the syntax looks ok, will need the main list changed and setup for MERGE DAY if/when we verify that the packages added either don't change any deps or get a confirm they are ok-changes to take for trunk builds. But in the short-term this is good to make it useable for try build tests.

I really don't expect any issues, though I did not actually review the list of packages you are grabbing, so consider that part of this patch rs+

Comment 13

[Mike Hommey [:glandium]]

https://hg.mozilla.org/build/buildbot-configs/rev/92e344aaa828

Keeping open for the actual non-try deployment.

Comment 14

[Mike Hommey [:glandium]]

Fixup
https://hg.mozilla.org/build/buildbot-configs/rev/0ef29e288b6f

Comment 15

[Aki Sasaki (not active)]

This is in production.

Comment 16

[Mike Hommey [:glandium]]

Attachment: Install libgcc.i686 instead of libgcc-devel.i686

Comment 17

[Mike Hommey [:glandium]]

(In reply to Mike Hommey [:glandium] from comment #16)
> Created attachment 737094 [details] [diff] [review]
> Install libgcc.i686 instead of libgcc-devel.i686

It looks like libgcc-devel.i686 doesn't exist. At least, it's not installed, and I think it's not necessary anyway. But libgcc.i686 is.

Comment 18

[Mike Hommey [:glandium]]

Attachment: Use x86_64 build environment for linux and linux-debug builds on the "date" branch, and install the necessary i686 -devel packages

This probably requires adjustments to mozilla/project_branches.py, but since the ones i did in bug 860246 don't seem to have entirely worked as expected, I'll leave these changes to someone who knows better.

Comment 19

[Mike Hommey [:glandium]]

Attachment: Use x86_64 build environment for linux and linux-debug builds on the "date" branch, and install the necessary i686 -devel packages

Refreshed against latest update to bug 860246

Comment 20

[Justin Wood (:Callek)]

Comment on attachment 738436 [details] [diff] [review]
Use x86_64 build environment for linux and linux-debug builds on the "date" branch, and install the necessary i686 -devel packages

Review of attachment 738436 [details] [diff] [review]:
-----------------------------------------------------------------

r- because as written would break configs (needs the line continuation) but other than that looks good

::: mozilla/config.py
@@ +1738,5 @@
>          ]
>  
> +for platform in ['linux', 'linux-debug']:
> +    BRANCHES['date']['platforms'][platform]['mock_target'] = 'mozilla-centos6-x86_64'
> +    BRANCHES['date']['platforms'][platform]['mock_packages'] = 

this needs a line continuation

@@ +1749,5 @@
> +        'atk-devel.i686', 'libnotify-devel.i686', 'dbus-devel.i686',
> +        'libcurl-devel.i686', 'libXt-devel.i686', 'libXext-devel.i686',
> +        'libstdc++-devel.i686', 'zlib-devel.i686', 'alsa-lib-devel.i686',
> +        'libgcc.i686',
> +        ]

this whole block violates PEP8, please make it conform.

Comment 21

[Justin Wood (:Callek)]

Comment on attachment 737094 [details] [diff] [review]
Install libgcc.i686 instead of libgcc-devel.i686

Review of attachment 737094 [details] [diff] [review]:
-----------------------------------------------------------------

simple, might as well just roll this into the fixed patch though, since this depends on other patch in this bug

Comment 22

[Mike Hommey [:glandium]]

Attachment: Use x86_64 build environment for linux and linux-debug builds on the "date" branch, and install the necessary i686 -devel packages

Comment 23

[Justin Wood (:Callek)]

Comment on attachment 738699 [details] [diff] [review]
Use x86_64 build environment for linux and linux-debug builds on the "date" branch, and install the necessary i686 -devel packages

Review of attachment 738699 [details] [diff] [review]:
-----------------------------------------------------------------

Actually, sorry for churn, but this fails test masters, because there is no linux/linux-debug on date yet.

So we are trying to copy from (and assign to) a platform that is missing right now.

Comment 24

[Mike Hommey [:glandium]]

Attachment: Use x86_64 build environment for linux and linux-debug builds on the "date" branch, and install the necessary i686 -devel packages

This depends on https://bugzilla.mozilla.org/attachment.cgi?id=738901

Comment 25

[Justin Wood (:Callek)]

Comment on attachment 739044 [details] [diff] [review]
Use x86_64 build environment for linux and linux-debug builds on the "date" branch, and install the necessary i686 -devel packages

Review of attachment 739044 [details] [diff] [review]:
-----------------------------------------------------------------

r+ as long as it lands after https://bug860246.bugzilla.mozilla.org/attachment.cgi?id=738901

Comment 26

[Chris Cooper [:coop] (he/him)]

Comment on attachment 739044 [details] [diff] [review]
Use x86_64 build environment for linux and linux-debug builds on the "date" branch, and install the necessary i686 -devel packages

This is in production now.

Comment 27

[Mike Hommey [:glandium]]

Attachment: Add more i686 packages on the x86-64 linux32 environment on the "date" branch

GConf is definitely needed, mesa, i'd say it's safer to have the i686 package, and gstreamer will be useful for the future (was added in bug 855492).

Comment 28

[Mike Hommey [:glandium]]

Attachment: Use a complete list of packages for the x86-64 linux32 environment on the "date" branch

Since the end goal is to switch the builders, it is actually better to start from the list we have in the i386 mock environments and switch the devel packages to i686 than to add packages to the existing list, which was fine for try (except it blew things up).

Comment 29

[Mike Hommey [:glandium]]

Comment on attachment 739815 [details] [diff] [review]
Use a complete list of packages for the x86-64 linux32 environment on the "date" branch

http://hg.mozilla.org/build/buildbot-configs/rev/6df75987d4ce

Comment 30

[Aki Sasaki (not active)]

This is in production.

Comment 31

[Mike Hommey [:glandium]]

Attachment: Add even more i686 packages to the x86-64 linux32 environment on the "date" branch

Yum is being annoying (see comments in the patch). Yum has an option to make things less annoying, but it can't be used for all the packages in the list, and mock --install doesn't support it anyways. (fwiw: --exclude=\*.x86_64), so it would also require modifying tinderbox scripts. In the end, it's just simpler to list all -devel packages.

Comment 32

[Justin Wood (:Callek)]

Comment on attachment 740753 [details] [diff] [review]
Add even more i686 packages to the x86-64 linux32 environment on the "date" branch

Review of attachment 740753 [details] [diff] [review]:
-----------------------------------------------------------------

with the caveat that I didn't look at these packages, and that I hope/suspect there is a better way/list we can use when we are ready to move this to m-c. The change is ok from a "for date only" releng perspective though!

Comment 33

[Mike Hommey [:glandium]]

(In reply to Justin Wood (:Callek) from comment #32)
> with the caveat that I didn't look at these packages, and that I
> hope/suspect there is a better way/list we can use when we are ready to move
> this to m-c.

AFAIK, short of modifying the tinderbox build scripts, there's no better way.

Comment 34

[Mike Hommey [:glandium]]

Comment on attachment 740753 [details] [diff] [review]
Add even more i686 packages to the x86-64 linux32 environment on the "date" branch

http://hg.mozilla.org/build/buildbot-configs/rev/b343fcc209b8

Comment 35

[Justin Wood (:Callek)]

in production as of 5p PT yesterday.

Comment 36

[Mike Hommey [:glandium]]

bug 864262 landed on m-i. The date branch is looking good with both gcc 4.7 and gcc 4.5, so it's time to think about how to land this on the main development branches. We can easily switch m-i and m-c (once current m-i is merged). What do we do with project branches? What about aurora/beta/release? Do we want to ride the trains or push it to some or all of them? (thus requiring a backport of bug 864262)

Comment 37

[Ryan VanderMeulen [:RyanVM]]

Not sure who would actually make the final decision here, but my opinion FWIW: change over all m-c/c-c based branches along with aurora and leave the rest to ride the trains.

Comment 38

[Mike Hommey [:glandium]]

For posterity:
<RyanVM> Try will get a bit complicated, though
<RyanVM> well, I guess not
<RyanVM> because if they push an older branch, it'll use that mozconfig, right?
<RyanVM> yay for being in-tree
<glandium> RyanVM: good point
<glandium> RyanVM: bug 864262 could be landed on all branches, independently of whether we want to switch on them
<glandium> it's a no-op when builds don't happen on 64-bits systems
<RyanVM> I would think it could land NPOTB
<glandium> that wouldn't solve the problem for people pushing old trees
<RyanVM> wouldn't that be controlled by the gcc version being pointed to in the in-tree mozconfig?
<glandium> RyanVM: that's orthogonal to the gcc version change
<RyanVM> what do you mean? if they push an old tree, wouldn't they just get the older version?
<RyanVM> assuming that it's still available for use
<glandium> RyanVM: it's a change of build environment, not of compiler
<RyanVM> oh, duh
<glandium> and that requires a specific mozconfig 
<RyanVM> hmm, that makes life more difficult
<glandium> yeah
<RyanVM> b2g definitely pushes b2g18 to try
<glandium> RyanVM: but they probably don't care about linux builds
<RyanVM> i wouldn't be too sure about that

Comment 39

[Trevor Saunders (:tbsaunde)]

Attachment: use the x86_64 mock env to do linux builds on all mozilla-central based trees

I believe this has all checked out on date so I think we're ready to actually make this change.

sorry about the ugly python.  This way of doing things doesn't really thrill me, but changing all the defaults and munging the other way for release branches sounds more risky and painful and once its everywhere we'll need to clean up anyway.

Comment 40

[Justin Wood (:Callek)]

Comment on attachment 743890 [details] [diff] [review]
use the x86_64 mock env to do linux builds on all mozilla-central based trees

When making this 'real' I personally want to make this part of the defaults, and trim down for branches it shouldn't rather than the other way...

we have ways to ensure stuff hasn't changed. And I'd also think running this through newsgroups or product/relman is worth it, since this has the try-will-be-painful story for other trees (e.g. b2g18)

Comment 41

[Mike Hommey [:glandium]]

(In reply to Justin Wood (:Callek) from comment #40)
> Comment on attachment 743890 [details] [diff] [review]
> use the x86_64 mock env to do linux builds on all mozilla-central based trees
> 
> When making this 'real' I personally want to make this part of the defaults,
> and trim down for branches it shouldn't rather than the other way...
> 
> we have ways to ensure stuff hasn't changed.

... which is why I wanted someone from releng to write the patch...

> And I'd also think running this
> through newsgroups or product/relman is worth it, since this has the
> try-will-be-painful story for other trees (e.g. b2g18)

Except -release, all main branches have received the patch from bug 864262. Project branches just need to merge a recent m-c if that's not already done.

Comment 42

[Trevor Saunders (:tbsaunde)]

(In reply to Mike Hommey [:glandium] from comment #41)
> (In reply to Justin Wood (:Callek) from comment #40)
> > Comment on attachment 743890 [details] [diff] [review]
> > use the x86_64 mock env to do linux builds on all mozilla-central based trees
> > 
> > When making this 'real' I personally want to make this part of the defaults,
> > and trim down for branches it shouldn't rather than the other way...
> > 
> > we have ways to ensure stuff hasn't changed.
> 
> ... which is why I wanted someone from releng to write the patch...

yeah, I certainly didn't want to spend time writing python, but I also wanted this out of my list of things to think about...  anyway I should have been able to realize I could check nothing had changed just by diffing output.

Comment 43

[Trevor Saunders (:tbsaunde)]

Attachment: do linux builds in a linux64 chroo accept on release branches

this patch makes me happier.  I also checked that no changes were made to the config for date so all other trees should have the same config as it, and the only change in aurora / beta was the order of valgrind vs pulseaudio-libs-devel.

Comment 44

[Trevor Saunders (:tbsaunde)]

Comment on attachment 744289 [details] [diff] [review]
do linux builds in a linux64 chroo accept on release branches

I think callek said he was looking at this today, but fixing r? anyway :)

Comment 45

[Justin Wood (:Callek)]

Comment on attachment 744289 [details] [diff] [review]
do linux builds in a linux64 chroo accept on release branches

Review of attachment 744289 [details] [diff] [review]:
-----------------------------------------------------------------

Just to call out surprises I saw in terms of builders that were changed by this:

release-mozilla-beta-fennec_source SingleSourceFactory
-- this is fine, its just the valgrind ordering, and does make sense, still surprising

release-mozilla-beta-firefox_source SingleSourceFactory
-- Same

This is just a consideration that should be made when this hits beta, both jobs probably fine to run as linux64 instead of linux fwiw, (and better that way since we'll have less stuff to install)

====

We should keep an eye on increased disk usage req's with this. And adjust accordingly as needed. Also please coordinate with buildduty when landing to be sure we can deploy soonish after this lands to make any emergency backout easy.

I'm not a fan of the way we specify all the .i686 deps here, but since it works on your project branch, and I don't have a better solution lets go with this

Comment 46

[Mike Hommey [:glandium]]

https://hg.mozilla.org/build/buildbot-configs/rev/219655584ea7

Comment 47

[Hal Wine [:hwine] use NI!]

in production

Comment 48

[Jordan Lund (:jlund)]

Attachment: 857697_remove_mock_linux_environment_gecko18-140326.patch

So IIUC, the only branches that will hit gecko 24 are mozilla-b2g18 and mozilla-b2g18_v1_1_0_hd. 

Both of those branches have set platforms and locked_out additional ones. The platforms they specify do not include 'linux' and 'linux-debug'. As far as I can tell, this block is not doing anything anymore.

Comment 49

[Justin Wood (:Callek)]

Comment on attachment 8397278 [details] [diff] [review]
857697_remove_mock_linux_environment_gecko18-140326.patch

Review of attachment 8397278 [details] [diff] [review]:
-----------------------------------------------------------------

can you do a dump master and compare builders before and after for this? I'm cautious here that we don't accidentally regress an old branch we still have builders for

Comment 50

[Jordan Lund (:jlund)]

Comment on attachment 8397278 [details] [diff] [review]
857697_remove_mock_linux_environment_gecko18-140326.patch

pushed to default: https://hg.mozilla.org/build/buildbot-configs/rev/df255296d2d1

dump master yielded a 0 diff

Comment 51

[John Hopkins (:jhopkins)]

in production.