Switch 32-bits linux builds to x86-64 mock environments

RESOLVED FIXED

Status

task
RESOLVED FIXED
6 years ago
a year ago

People

(Reporter: tbsaunde, Assigned: glandium)

Tracking

Details

(Whiteboard: [leave open])

Attachments

(5 attachments, 7 obsolete attachments)

2.30 KB, patch
Callek
: review+
Details | Diff | Splinter Review
2.86 KB, patch
catlee
: review+
glandium
: checked-in+
Details | Diff | Splinter Review
2.45 KB, patch
Callek
: review+
glandium
: checked-in+
Details | Diff | Splinter Review
11.94 KB, patch
Callek
: review+
Details | Diff | Splinter Review
2.25 KB, patch
Callek
: review+
jlund
: checked-in+
Details | Diff | Splinter Review
(Reporter)

Description

6 years ago
we'd like to test cross compiling x86 linux builds on x86_64 to do that we need the follow packages installed on the x86_64 build slaves.

     * glibc-devel.i686
     * cairo-devel.i686
     * fontconfig-devel.i686
     * gtk2-devel.i686
     * dbus-glib-devel.i686
     * glib2-devel.i686
     * gdk-pixbuf2-devel.i686
     * pango-devel.i686
     * pixman-devel.i686
     * freetype-devel.i686
     * libpng-devel.i686
     * libXrender-devel.i686
     * libX11-devel.i686
     * libxcb-devel.i686
     * libXau-devel.i686
     * atk-devel.i686
     * libnotify-devel.i686
     * dbus-devel.i686
     * libcurl-devel.i686
     * libXt-devel.i686
     * libXext-devel.i686
     * libstdc++-devel.i686
     * zlib-devel.i686

Comment 1

6 years ago
Would you mind borrowing one of our build machines and confirming that those packages would allow us to do cross-compilation?
(Reporter)

Comment 2

6 years ago
(In reply to Armen Zambrano G. [:armenzg] from comment #1)
> Would you mind borrowing one of our build machines and confirming that those
> packages would allow us to do cross-compilation?

sure,  fwiw I did check on a fresh centos6 chroot so I really expect it will work.
found in triage.
Component: Release Engineering → Release Engineering: Platform Support
QA Contact: coop
I got nudged to put-in-bug my privately stated concern with this ask.

"I'm mostly concerned about possibly affecting our official linux64 builds by adding the packages, but I can't say for sure one way or the other" I'll leave it to you guys to either dismiss disprove or otherwise that though.
(Reporter)

Comment 5

6 years ago
ok, So I finished up testing stuff here, it turns out we need two more packages than I mentioned in comment 0 we also need alsa-lib-devel.i686 and libgcc-devel.i686
(Reporter)

Comment 6

6 years ago
> "I'm mostly concerned about possibly affecting our official linux64 builds
> by adding the packages, but I can't say for sure one way or the other" I'll
> leave it to you guys to either dismiss disprove or otherwise that though.

Its certainly possible, in theory any change we make could do that, but this might be a little more likely.  However I'd think the chances are fairly remote and we should be able to check to be sure nothing actually broke.  I checked that after installing the i686 packages I could still do a build using the linux64 nightly mozconfig with gcc-4.7, but I didn't run any tests on that build.
(Assignee)

Comment 7

6 years ago
Except if some .x86-64 packages are installed, .i686 packages have no reason to alter the environment when used for .x86-64 type of stuff, which is what they are used for. Even more so for -devel.i686 packages.

Comment 8

6 years ago
Out of ignorance, why do we want to do cross compilation rather than staying with the current mock setup we have? (IIUC we use mock to be able to compile the 32-bit builds ona 64-bit machine - that's as far as I know and I'm not sure)

In other words, what is the ultimate goal?

For testing purposes, we could put the machine on staging and make it trigger jobs on a staging master with a 32-bit and 64-bit testers could test the builds.
(Assignee)

Comment 9

6 years ago
32-bits builds OOM with PGO with gcc 4.7. Which means going forward, we need the extra memory for the toolchain.
(Assignee)

Updated

6 years ago
Blocks: gcc-4.7
(Reporter)

Comment 10

6 years ago
(In reply to Armen Zambrano G. [:armenzg] from comment #8)
> Out of ignorance, why do we want to do cross compilation rather than staying
> with the current mock setup we have? (IIUC we use mock to be able to compile
> the 32-bit builds ona 64-bit machine - that's as far as I know and I'm not
> sure)
> 
> In other words, what is the ultimate goal?
> 
> For testing purposes, we could put the machine on staging and make it
> trigger jobs on a staging master with a 32-bit and 64-bit testers could test
> the builds.

we could certainly test more, but me and glandium decided we both think the risk is too low to spend the effort, we should just get those packages shipped on the slaves so we can move forward with changing compiler to stop the pain of gcc 4.5 crashing in PGO builds.
(Assignee)

Updated

6 years ago
Assignee: nobody → mh+mozilla
Comment on attachment 736490 [details] [diff] [review]
Install necessary i686 -devel packages on linux64 build slaves (try only)

Review of attachment 736490 [details] [diff] [review]:
-----------------------------------------------------------------

the syntax looks ok, will need the main list changed and setup for MERGE DAY if/when we verify that the packages added either don't change any deps or get a confirm they are ok-changes to take for trunk builds. But in the short-term this is good to make it useable for try build tests.

I really don't expect any issues, though I did not actually review the list of packages you are grabbing, so consider that part of this patch rs+
Attachment #736490 - Flags: review?(bugspam.Callek) → review+
(Assignee)

Comment 13

6 years ago
https://hg.mozilla.org/build/buildbot-configs/rev/92e344aaa828

Keeping open for the actual non-try deployment.
(Assignee)

Updated

6 years ago
Whiteboard: [leave open]
(Reporter)

Updated

6 years ago
Blocks: 860246
(Assignee)

Updated

6 years ago
No longer blocks: 860246
This is in production.
(Assignee)

Comment 16

6 years ago
Attachment #737094 - Flags: review?(bugspam.Callek)
(Assignee)

Comment 17

6 years ago
(In reply to Mike Hommey [:glandium] from comment #16)
> Created attachment 737094 [details] [diff] [review]
> Install libgcc.i686 instead of libgcc-devel.i686

It looks like libgcc-devel.i686 doesn't exist. At least, it's not installed, and I think it's not necessary anyway. But libgcc.i686 is.
(Assignee)

Comment 18

6 years ago
This probably requires adjustments to mozilla/project_branches.py, but since the ones i did in bug 860246 don't seem to have entirely worked as expected, I'll leave these changes to someone who knows better.
Attachment #737582 - Flags: review?(bugspam.Callek)
(Assignee)

Updated

6 years ago
Attachment #736490 - Attachment is obsolete: true
(Assignee)

Updated

6 years ago
Attachment #737582 - Attachment is obsolete: true
Attachment #737582 - Flags: review?(bugspam.Callek)
Comment on attachment 738436 [details] [diff] [review]
Use x86_64 build environment for linux and linux-debug builds on the "date" branch, and install the necessary i686 -devel packages

Review of attachment 738436 [details] [diff] [review]:
-----------------------------------------------------------------

r- because as written would break configs (needs the line continuation) but other than that looks good

::: mozilla/config.py
@@ +1738,5 @@
>          ]
>  
> +for platform in ['linux', 'linux-debug']:
> +    BRANCHES['date']['platforms'][platform]['mock_target'] = 'mozilla-centos6-x86_64'
> +    BRANCHES['date']['platforms'][platform]['mock_packages'] = 

this needs a line continuation

@@ +1749,5 @@
> +        'atk-devel.i686', 'libnotify-devel.i686', 'dbus-devel.i686',
> +        'libcurl-devel.i686', 'libXt-devel.i686', 'libXext-devel.i686',
> +        'libstdc++-devel.i686', 'zlib-devel.i686', 'alsa-lib-devel.i686',
> +        'libgcc.i686',
> +        ]

this whole block violates PEP8, please make it conform.
Attachment #738436 - Flags: review?(bugspam.Callek) → review-
Comment on attachment 737094 [details] [diff] [review]
Install libgcc.i686 instead of libgcc-devel.i686

Review of attachment 737094 [details] [diff] [review]:
-----------------------------------------------------------------

simple, might as well just roll this into the fixed patch though, since this depends on other patch in this bug
Attachment #737094 - Flags: review?(bugspam.Callek) → review+
(Assignee)

Updated

6 years ago
Attachment #737094 - Attachment is obsolete: true
(Assignee)

Updated

6 years ago
Attachment #738436 - Attachment is obsolete: true
Attachment #738699 - Flags: review?(bugspam.Callek) → review+
Comment on attachment 738699 [details] [diff] [review]
Use x86_64 build environment for linux and linux-debug builds on the "date" branch, and install the necessary i686 -devel packages

Review of attachment 738699 [details] [diff] [review]:
-----------------------------------------------------------------

Actually, sorry for churn, but this fails test masters, because there is no linux/linux-debug on date yet.

So we are trying to copy from (and assign to) a platform that is missing right now.
Attachment #738699 - Flags: review+ → review-
(Assignee)

Updated

6 years ago
Attachment #738699 - Attachment is obsolete: true
Comment on attachment 739044 [details] [diff] [review]
Use x86_64 build environment for linux and linux-debug builds on the "date" branch, and install the necessary i686 -devel packages

Review of attachment 739044 [details] [diff] [review]:
-----------------------------------------------------------------

r+ as long as it lands after https://bug860246.bugzilla.mozilla.org/attachment.cgi?id=738901
Attachment #739044 - Flags: review?(bugspam.Callek) → review+
Comment on attachment 739044 [details] [diff] [review]
Use x86_64 build environment for linux and linux-debug builds on the "date" branch, and install the necessary i686 -devel packages

This is in production now.
Attachment #739044 - Flags: checked-in+
(Assignee)

Comment 27

6 years ago
GConf is definitely needed, mesa, i'd say it's safer to have the i686 package, and gstreamer will be useful for the future (was added in bug 855492).
Attachment #739774 - Flags: review?(catlee)
(Assignee)

Comment 28

6 years ago
Since the end goal is to switch the builders, it is actually better to start from the list we have in the i386 mock environments and switch the devel packages to i686 than to add packages to the existing list, which was fine for try (except it blew things up).
Attachment #739815 - Flags: review?(catlee)
(Assignee)

Updated

6 years ago
Attachment #739774 - Attachment is obsolete: true
Attachment #739774 - Flags: review?(catlee)
(Assignee)

Updated

6 years ago
Summary: please install some i686 packages on all centos 6 x86_64 linux build slaves → Switch 32-bits linux builds to x86-64 mock environments
(Assignee)

Updated

6 years ago
Depends on: 864262
Attachment #739815 - Flags: review?(catlee) → review+
(Assignee)

Comment 29

6 years ago
Comment on attachment 739815 [details] [diff] [review]
Use a complete list of packages for the x86-64 linux32 environment on the "date" branch

http://hg.mozilla.org/build/buildbot-configs/rev/6df75987d4ce
Attachment #739815 - Flags: checked-in+
This is in production.
(Assignee)

Comment 31

6 years ago
Yum is being annoying (see comments in the patch). Yum has an option to make things less annoying, but it can't be used for all the packages in the list, and mock --install doesn't support it anyways. (fwiw: --exclude=\*.x86_64), so it would also require modifying tinderbox scripts. In the end, it's just simpler to list all -devel packages.
Attachment #740753 - Flags: review?(catlee)
Comment on attachment 740753 [details] [diff] [review]
Add even more i686 packages to the x86-64 linux32 environment on the "date" branch

Review of attachment 740753 [details] [diff] [review]:
-----------------------------------------------------------------

with the caveat that I didn't look at these packages, and that I hope/suspect there is a better way/list we can use when we are ready to move this to m-c. The change is ok from a "for date only" releng perspective though!
Attachment #740753 - Flags: review?(catlee) → review+
(Assignee)

Comment 33

6 years ago
(In reply to Justin Wood (:Callek) from comment #32)
> with the caveat that I didn't look at these packages, and that I
> hope/suspect there is a better way/list we can use when we are ready to move
> this to m-c.

AFAIK, short of modifying the tinderbox build scripts, there's no better way.
(Assignee)

Comment 34

6 years ago
Comment on attachment 740753 [details] [diff] [review]
Add even more i686 packages to the x86-64 linux32 environment on the "date" branch

http://hg.mozilla.org/build/buildbot-configs/rev/b343fcc209b8
Attachment #740753 - Flags: checked-in+
in production as of 5p PT yesterday.
(Assignee)

Comment 36

6 years ago
bug 864262 landed on m-i. The date branch is looking good with both gcc 4.7 and gcc 4.5, so it's time to think about how to land this on the main development branches. We can easily switch m-i and m-c (once current m-i is merged). What do we do with project branches? What about aurora/beta/release? Do we want to ride the trains or push it to some or all of them? (thus requiring a backport of bug 864262)
Not sure who would actually make the final decision here, but my opinion FWIW: change over all m-c/c-c based branches along with aurora and leave the rest to ride the trains.
(Assignee)

Comment 38

6 years ago
For posterity:
<RyanVM> Try will get a bit complicated, though
<RyanVM> well, I guess not
<RyanVM> because if they push an older branch, it'll use that mozconfig, right?
<RyanVM> yay for being in-tree
<glandium> RyanVM: good point
<glandium> RyanVM: bug 864262 could be landed on all branches, independently of whether we want to switch on them
<glandium> it's a no-op when builds don't happen on 64-bits systems
<RyanVM> I would think it could land NPOTB
<glandium> that wouldn't solve the problem for people pushing old trees
<RyanVM> wouldn't that be controlled by the gcc version being pointed to in the in-tree mozconfig?
<glandium> RyanVM: that's orthogonal to the gcc version change
<RyanVM> what do you mean? if they push an old tree, wouldn't they just get the older version?
<RyanVM> assuming that it's still available for use
<glandium> RyanVM: it's a change of build environment, not of compiler
<RyanVM> oh, duh
<glandium> and that requires a specific mozconfig 
<RyanVM> hmm, that makes life more difficult
<glandium> yeah
<RyanVM> b2g definitely pushes b2g18 to try
<glandium> RyanVM: but they probably don't care about linux builds
<RyanVM> i wouldn't be too sure about that
(Reporter)

Comment 39

6 years ago
I believe this has all checked out on date so I think we're ready to actually make this change.

sorry about the ugly python.  This way of doing things doesn't really thrill me, but changing all the defaults and munging the other way for release branches sounds more risky and painful and once its everywhere we'll need to clean up anyway.
Attachment #743890 - Flags: review?
Comment on attachment 743890 [details] [diff] [review]
use the x86_64 mock env to do linux builds on all mozilla-central based trees

When making this 'real' I personally want to make this part of the defaults, and trim down for branches it shouldn't rather than the other way...

we have ways to ensure stuff hasn't changed. And I'd also think running this through newsgroups or product/relman is worth it, since this has the try-will-be-painful story for other trees (e.g. b2g18)
Attachment #743890 - Flags: feedback-
(Assignee)

Comment 41

6 years ago
(In reply to Justin Wood (:Callek) from comment #40)
> Comment on attachment 743890 [details] [diff] [review]
> use the x86_64 mock env to do linux builds on all mozilla-central based trees
> 
> When making this 'real' I personally want to make this part of the defaults,
> and trim down for branches it shouldn't rather than the other way...
> 
> we have ways to ensure stuff hasn't changed.

... which is why I wanted someone from releng to write the patch...

> And I'd also think running this
> through newsgroups or product/relman is worth it, since this has the
> try-will-be-painful story for other trees (e.g. b2g18)

Except -release, all main branches have received the patch from bug 864262. Project branches just need to merge a recent m-c if that's not already done.
(Reporter)

Comment 42

6 years ago
(In reply to Mike Hommey [:glandium] from comment #41)
> (In reply to Justin Wood (:Callek) from comment #40)
> > Comment on attachment 743890 [details] [diff] [review]
> > use the x86_64 mock env to do linux builds on all mozilla-central based trees
> > 
> > When making this 'real' I personally want to make this part of the defaults,
> > and trim down for branches it shouldn't rather than the other way...
> > 
> > we have ways to ensure stuff hasn't changed.
> 
> ... which is why I wanted someone from releng to write the patch...

yeah, I certainly didn't want to spend time writing python, but I also wanted this out of my list of things to think about...  anyway I should have been able to realize I could check nothing had changed just by diffing output.
(Reporter)

Comment 43

6 years ago
this patch makes me happier.  I also checked that no changes were made to the config for date so all other trees should have the same config as it, and the only change in aurora / beta was the order of valgrind vs pulseaudio-libs-devel.
Attachment #743890 - Attachment is obsolete: true
Attachment #743890 - Flags: review?
Attachment #744289 - Flags: review?
(Reporter)

Comment 44

6 years ago
Comment on attachment 744289 [details] [diff] [review]
do linux builds in a linux64 chroo accept on release branches

I think callek said he was looking at this today, but fixing r? anyway :)
Attachment #744289 - Flags: review? → review?(bugspam.Callek)
Comment on attachment 744289 [details] [diff] [review]
do linux builds in a linux64 chroo accept on release branches

Review of attachment 744289 [details] [diff] [review]:
-----------------------------------------------------------------

Just to call out surprises I saw in terms of builders that were changed by this:

release-mozilla-beta-fennec_source SingleSourceFactory
-- this is fine, its just the valgrind ordering, and does make sense, still surprising

release-mozilla-beta-firefox_source SingleSourceFactory
-- Same

This is just a consideration that should be made when this hits beta, both jobs probably fine to run as linux64 instead of linux fwiw, (and better that way since we'll have less stuff to install)

====

We should keep an eye on increased disk usage req's with this. And adjust accordingly as needed. Also please coordinate with buildduty when landing to be sure we can deploy soonish after this lands to make any emergency backout easy.

I'm not a fan of the way we specify all the .i686 deps here, but since it works on your project branch, and I don't have a better solution lets go with this
Attachment #744289 - Flags: review?(bugspam.Callek) → review+
in production
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Blocks: 873989
Product: mozilla.org → Release Engineering
So IIUC, the only branches that will hit gecko 24 are mozilla-b2g18 and mozilla-b2g18_v1_1_0_hd. 

Both of those branches have set platforms and locked_out additional ones. The platforms they specify do not include 'linux' and 'linux-debug'. As far as I can tell, this block is not doing anything anymore.
Attachment #8397278 - Flags: review?(bugspam.Callek)
Comment on attachment 8397278 [details] [diff] [review]
857697_remove_mock_linux_environment_gecko18-140326.patch

Review of attachment 8397278 [details] [diff] [review]:
-----------------------------------------------------------------

can you do a dump master and compare builders before and after for this? I'm cautious here that we don't accidentally regress an old branch we still have builders for
Attachment #8397278 - Flags: review?(bugspam.Callek) → review+
Comment on attachment 8397278 [details] [diff] [review]
857697_remove_mock_linux_environment_gecko18-140326.patch

pushed to default: https://hg.mozilla.org/build/buildbot-configs/rev/df255296d2d1

dump master yielded a 0 diff
Attachment #8397278 - Flags: checked-in+
in production.
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.