772563 - Determine binary compatibility issues of doing linux desktop builds on centos6

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Comment 1

[Chris AtLee [:catlee]]

from our mozilla-centos6-x86_64 environment:

libstdc++-4.4.6-3.el6
glibc-2.12-1.47.el6_2.5
gtk2-2.18.9-6.el6.centos

Comment 2

[Benjamin Smedberg]

What are the equivalent versions on centOS 5?

Comment 3

[Chris AtLee [:catlee]]

From our 32-bit centos5 machine:
libstdc++-4.1.1-52.el5
glibc-2.5-12
gtk2-2.10.4-16.el5

From our 64-bit centos5 machine:
libstdc++-4.1.1-52.el5
glibc-2.5-12
gtk2-2.10.4-16.el5

Comment 4

[Mike Hommey [:glandium]]

I don't expect problems with glibc. libstdc++ is dealt with stdc++compat.cpp. That leaves GTK. For reference, there is a list of version found in some common distros in bug 666735. With our requirement for FT 2.3, we are already excluding RHEL5 and SLES 10. When building against gtk 2.18, I'm not sure what compatibility requirement we'd end up with.

Comment 5

[Mike Hommey [:glandium]]

Chris, could you do test builds in the centos6 environment and put them up somewhere so that we can check the symbols it requires?

Comment 6

[John Hopkins (:jhopkins)]

Mike, here is a test build I've created from mozilla-inbound revision d00ae2e9b92a:

 http://people.mozilla.org/~jhopkins/bug772563/

Comment 7

[Mike Hommey [:glandium]]

(In reply to John Hopkins (:jhopkins) from comment #6)
> Mike, here is a test build I've created from mozilla-inbound revision
> d00ae2e9b92a:
> 
>  http://people.mozilla.org/~jhopkins/bug772563/

This build depends on gtk/gdk 2.24. Did you really use the centos6-x86_64 environment, or are the versions listed in comment 1 wrong?

Comment 8

[Mike Hommey [:glandium]]

(In reply to Mike Hommey [:glandium] from comment #7)
> This build depends on gtk/gdk 2.24. Did you really use the centos6-x86_64
> environment, or are the versions listed in comment 1 wrong?

Oh, my bad, it's glib 2.24, not gtk 2.24. gtk/gdk dependency is on version 2.18, but with a very small effort, we can actually drop that to 2.10. It should also be easily possible to drop the glib dependency requirement to 2.16, 2.14 or 2.12, depending how hard we want to be backwards compatible.

Comment 9

[Chris AtLee [:catlee]]

Does that require changes to the build environment, or are you referring to code changes?

Comment 10

[Mike Hommey [:glandium]]

(In reply to Chris AtLee [:catlee] from comment #9)
> Does that require changes to the build environment, or are you referring to
> code changes?

I'm referring to code changes. But before doing any code change, we need to decide what level of compatibility we want to provide.

Comment 11

[Chris AtLee [:catlee]]

We'd like to make this change for Firefox 17 if possible. How do we make a decision here?

Comment 12

[Mike Hommey [:glandium]]

I'll write a compatibility matrix, and from there, the product team will be able to choose.

Comment 13

[Mike Hommey [:glandium]]

This the data I gathered for current distros, and some older ones that should be compatible with the builds John did, so that the info can be reused later, when we upgrade our build environment again:
               glib    gtk    
fedora 16      2.30    2.24
fedora 17      2.32    2.24
rhel 6         2.22    2.18
opensuse 11.4  2.28    2.22
opensuse 11.3  2.24    2.20
opensuse 11.2  2.22    2.18
sles 11        2.18    2.14
sles 11sp1     2.22    2.18
sles 11sp2     2.22    2.18
opensuse 12.1  2.30    2.24
opensuse 12.2  2.32    2.24
debian 6       2.24    2.20
debian 7       2.32    2.24

The builds John did require glib 2.24 and gtk 2.18. That is compatible with most of the above except:
rhel 6         2.22    2.18
sles 11        2.18    2.14
sles 11sp1     2.22    2.18
sles 11sp2     2.22    2.18

A two-liner should get us compatible with rhel 6, sles 11sp1 and 11sp2. It would be a little more work for sles 11 compatibility, but it's probably not worth it. I think we can leave the product team alone, considering the amount of work involved.

Comment 14

[Mike Hommey [:glandium]]

Attachment: test patch

John, or someone else, can you create a new build from the centos6 build environment, with this patch applied? I'd like to double check that the resulting binaries have the right dependencies.

Comment 15

[Karl Tomlinson (:karlt)]

Even CentOS 6.3 looks like it still has glib2-2.22.5-7.el6.src.rpm
http://vault.centos.org/6.3/os/Source/SPackages/

Also note that SLES 11-SP2 may have glibc 2.11.3
http://distrowatch.com/table.php?distribution=sle
http://susestudio.com/a/Y7LZsm/petechadwicks-sles-11-sp2-jeos/packages?image_type=ec2&version=0.0.3
or glibc 2.11.1
http://www.suse.com/releasenotes/x86_64/SUSE-SLES/11-SP2/#id1168561

Comment 16

[Karl Tomlinson (:karlt)]

(In reply to Karl Tomlinson (:karlt) from comment #15)
> Even CentOS 6.3 looks like it still has glib2-2.22.5-7.el6.src.rpm
> http://vault.centos.org/6.3/os/Source/SPackages/

Oh, its the gmalloc-overflow.patch that adds g_malloc_n etc.

Comment 17

[Karl Tomlinson (:karlt)]

Given that looks like a security patch, SLES may have those symbols in their packages, but I can't find the packages to check.

Comment 18

[Mike Hommey [:glandium]]

(In reply to Karl Tomlinson (:karlt) from comment #17)
> Given that looks like a security patch, SLES may have those symbols in their
> packages, but I can't find the packages to check.

Given that this only has a benefit if you recompile all things depending on glib, might not be so true. The glib package i could find for opensuse 11.2 doesn't have it.

Comment 19

[Mike Hommey [:glandium]]

(In reply to Karl Tomlinson (:karlt) from comment #15)
> Also note that SLES 11-SP2 may have glibc 2.11.3
> http://distrowatch.com/table.php?distribution=sle
> http://susestudio.com/a/Y7LZsm/petechadwicks-sles-11-sp2-jeos/
> packages?image_type=ec2&version=0.0.3
> or glibc 2.11.1
> http://www.suse.com/releasenotes/x86_64/SUSE-SLES/11-SP2/#id1168561

glibc is not a concern. The build John did requires 2.7.

Comment 20

[Mike Hommey [:glandium]]

(In reply to Mike Hommey [:glandium] from comment #18)
> The glib package i could find for opensuse 11.2 doesn't have it.

And I could verify that SLES 11 sp1 or sp2 don't have it either.

Comment 21

[scientes]

Centos 6 has libnotify1, while many distros are currently shipping libnotify4, which causes my Bug 750141 (broken scheme:// and download/mime-type stuff when missing libnotify(1)).

Perhaps we could ship the libnotify1 (or 4) .so file with "make package"?

Comment 22

[Mike Hommey [:glandium]]

(In reply to scientes from comment #21)
> Centos 6 has libnotify1, while many distros are currently shipping
> libnotify4, which causes my Bug 750141 (broken scheme:// and
> download/mime-type stuff when missing libnotify(1)).
> 
> Perhaps we could ship the libnotify1 (or 4) .so file with "make package"?

That's fixed by bug 783765

Comment 23

[Rail Aliiev [:rail]]

(In reply to Mike Hommey [:glandium] from comment #14)
> Created attachment 646166 [details] [diff] [review]
> test patch
> 
> John, or someone else, can you create a new build from the centos6 build
> environment, with this patch applied? I'd like to double check that the
> resulting binaries have the right dependencies.

Sorry for the delay...

I created linux/linux64 builds based on a fresh m-c clone + your patch using mock based centos6 slaves:

http://people.mozilla.org/~raliiev/centos6/

Comment 24

[Mike Hommey [:glandium]]

(In reply to Rail Aliiev [:rail] (off Sep 8-16) from comment #23)
> Sorry for the delay...
> 
> I created linux/linux64 builds based on a fresh m-c clone + your patch using
> mock based centos6 slaves:
> 
> http://people.mozilla.org/~raliiev/centos6/

These builds look good. I'll come up with a real patch to replace this test patch. Would you prefer this work to go in a separate bug?

Comment 25

[Nick Thomas [:nthomas] (UTC+12)]

To follow up on the IRC discussion today, here's some metrics data:

AUS pings on 2012-09-18, divide by ~2.2 to get blocklist ADU because AUS ping interval is more than once per day

   GTK vers   PING
    2.10       113
    2.12        60
    2.14        24
    2.16        51
    2.18       260
    2.20       985
    2.22       167
    2.24      4489

    total     6149

So about 8% of our users have GTK 2.18 and lower, which we'd use as a proxy for requiring glib 2.24.

Comment 26

[Nick Thomas [:nthomas] (UTC+12)]

Forgot to say that's the nightly update channel, a version like 18.0, and includes both 32 and 64bit linux.

Comment 27

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [cvs] Block GTK 2.11 through to 2.18 prior to lowering requirements

This would let us move to mock slaves prior to landing a patch from glandium that lowers the requirements again, by blocking updates for the users still on older libraries.

Comment 28

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 662728 [details] [diff] [review]
[cvs] Block GTK 2.11 through to 2.18 prior to lowering requirements

$ cvs ci -m "Bug 772563, block older GTK for switch to mock-based linux build machines, r=catlee"
Checking in config-dist.php;
/cvsroot/mozilla/webtools/aus/xml/inc/config-dist.php,v  <--  config-dist.php
new revision: 1.225; previous revision: 1.224
done

$ cvs tag AUS2_PRODUCTION config-dist.php 
W config-dist.php : AUS2_PRODUCTION already exists on version 1.224 : NOT MOVING tag to version 1.225

$ cvs tag -F AUS2_PRODUCTION config-dist.php 
T config-dist.php

Comment 29

[Mike Hommey [:glandium]]

The relevant information is in comment 13, and the fix for builds after the switch to mock landed in bug 793634. I don't think there's anything left to do?

Comment 30

[Karl Tomlinson (:karlt)]

We can now provide updates for systems with GTK+ 2.18.

Comment 31

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [cvs] final patch, allow gtk 2.18 again

Just turn updates back on for people with gtk 2.18/glib 2.22 after the patch in bug 793634 has a chance to get into nightlies.

Comment 32

[John Hopkins (:jhopkins)]

We will need a similar AUS config patch for Thunderbird

Comment 33

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 664657 [details] [diff] [review]
[cvs] final patch, allow gtk 2.18 again

new revision: 1.226; previous revision: 1.225

Comment 34

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [cvs] thunderbird too

Copy over from thunderbird and update bug reference.

Comment 35

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [cvs] handle merge to aurora

Switch to version 17.0a2 for using mock on aurora, and bring over the thunderbird change from the last patch.

Comment 36

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 666735 [details] [diff] [review]
[cvs] handle merge to aurora

new revision: 1.227; previous revision: 1.226

Also moved AUS2_PRODUCTION from rev 1.225 to 1.227.

Comment 37

[Mike Hommey [:glandium]]

I think this is good now.