773769 - upgrade centos puppetagain systems to centos-6.5

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

We're coming up on 6 months from the first snapshot (early March).

bug 773379 mirrored CentOS 6.3, so the long part's done.

Once other dust has settled, we'll need to change repos on a few hosts to use the date instead of "latest", then see what breaks.

We'll need to decide if we want to run 'yum upgrade', or just let the 'ensure => latest' lines take care of pulling in updated versions of packages we care about, and leave everything else as-is.  If we do the latter, we should also re-image a few hosts to make sure that the upgraded versions work, too.

I'm getting this filed so it's on the radar.  This isn't a high priority now, but IMHO we should plan to do it in early Q4.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

This was re-mirrored on 2012-09-06.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

We had intended to do this twice a year.  It's now been a year.

http://wiki.centos.org/Manuals/ReleaseNotes/CentOS6.4

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Huh, I have no idea how I got to this bug this morning and accidentally resolved it :(

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Almost two years, and three versions..

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

I mirrored 6.5, and did some checks with various toplevel classes using relabs.  Here are the upgrades just from running puppet against the new repos, using relabs.  The toplevel classes are listed in parentheses at the end.

> OpenIPMI: current_value 2.0.16-12.el6, should be 0:2.0.16-14.el6 (toplevel::base)
> bind-utils: current_value 9.7.3-8.P3.el6_2.2, should be 32:9.8.2-0.17.rc1.el6_4.6 (toplevel::server::foopy)
> createrepo: current_value 0.9.8-4.el6, should be 0:0.9.9-18.el6 (toplevel::server::puppetmaster)
> cronie: current_value 1.4.4-7.el6, should be 0:1.4.4-12.el6 (toplevel::server)
> gcc: current_value 4.4.6-3.el6, should be 0:4.4.7-4.el6 (toplevel::server::signing)
> gnupg2: current_value 2.0.14-4.el6, should be 0:2.0.14-6.el6_4 (toplevel::server::signing)
> gtk2: current_value 2.18.9-6.el6.centos, should be 0:2.20.1-4.el6 (toplevel::server::foopy)
> httpd: current_value 2.2.15-15.el6.centos.1, should be 0:2.2.15-29.el6.centos (toplevel::server::mozpool) (toplevel::server::puppetmaster)
> ipmitool: current_value 1.8.11-12.el6_2.1, should be 0:1.8.11-16.el6 (toplevel::base)
> libXext: current_value 1.1-3.el6, should be 0:1.3.1-2.el6 (toplevel::server::foopy)
> libXt: current_value 1.0.7-1.el6, should be 0:1.1.3-1.el6 (toplevel::server::foopy)
> libevent-devel: current_value 1.4.13-1.el6, should be 0:1.4.13-4.el6 (toplevel::server::signing) (toplevel::server::slaveapi)
> libevent: current_value 1.4.13-1.el6, should be 0:1.4.13-4.el6 (toplevel::server::signing) (toplevel::server::slaveapi)
> libstdc++.i686: current_value 4.4.6-3.el6, should be 0:4.4.7-4.el6 (toplevel::server::foopy)
> logrotate: current_value 3.7.8-12.el6_0.1, should be 0:3.7.8-17.el6 (gaia_bumper) (toplevel::server::mozpool)
> mailx: current_value 12.4-6.el6, should be 0:12.4-7.el6 (toplevel::server)
> make: current_value 3.81-19.el6, should be 1:3.81-20.el6 (toplevel::server::signing)
> mod_ssl: current_value 2.2.15-15.el6.centos.1, should be 1:2.2.15-29.el6.centos (toplevel::server::puppetmaster)
> mysql: current_value 5.1.61-1.el6_2.1, should be 0:5.1.71-1.el6 (toplevel::server::mozpool)
> net-snmp-utils: current_value 5.5-37.el6_2.1, should be 1:5.5-49.el6 (toplevel::server::foopy) (toplevel::server::slaveapi)
> net-snmp: current_value 5.5-37.el6_2.1, should be 1:5.5-49.el6 (toplevel::server::slaveapi)
> nss-tools: current_value 3.13.1-7.el6_2, should be 0:3.15.1-15.el6 (toplevel::server::signing)
> ntp: current_value 4.2.4p8-2.el6.centos, should be 0:4.2.6p5-1.el6.centos (toplevel::base)
> openssl: current_value 1.0.0-20.el6_2.2, should be 0:1.0.1e-15.el6 (toplevel::server::signing)
> pango: current_value 1.28.1-3.el6_0.5.1.centos, should be 0:1.28.1-7.el6_3 (toplevel::server::foopy)
> rsync: current_value 3.0.6-5.el6_0.1, should be 0:3.0.6-9.el6_4.1 (toplevel::server::foopy) (toplevel::server::puppetmaster)
> rsyslog-mysql: current_value 4.6.2-12.el6, should be 0:5.8.10-8.el6 (toplevel::server::mozpool)
> rsyslog: current_value 4.6.2-12.el6, should be 0:5.8.10-8.el6 (toplevel::server::mozpool)
> strace: current_value 4.5.19-1.11.el6_2.1, should be 0:4.5.19-1.17.el6 (toplevel::server)
> sudo: current_value 1.7.4p5-7.el6, should be 0:1.8.6p3-12.el6 (toplevel::base)
> telnet: current_value 0.17-47.el6, should be 1:0.17-47.el6_3.1 (toplevel::server::foopy)
> tzdata: current_value 2011n-2.el6, should be 0:2013g-1.el6 (toplevel::base)
> vim-minimal: current_value 7.2.411-1.6.el6, should be 2:7.2.411-1.8.el6 (toplevel::base)
> wget: current_value 1.12-1.4.el6, should be 0:1.12-1.8.el6 (slave::build)
> xinetd: current_value 2.3.14-33.el6, should be 2:2.3.14-39.el6_4 (toplevel::server::puppetmaster) (toplevel::server::mozpool)
> xorg-x11-server-Xvfb: current_value 1.10.4-6.el6_2.1.0.1.centos, should be 0:1.13.0-23.el6.centos (slave::build::mock)
> zlib.i686: current_value 1.2.3-27.el6, should be 0:1.2.3-29.el6 (toplevel::server::foopy)

The gcc upgrade is just for signing, and only used for building Python virtualenvs.  It's not the compiler used to do builds -- those occur in the mock environment.

gnupg only changes in a release version, so I don't expect incompatibilities for signing.  Similar for most of these, in fact.

Does anyone else see anything to worry about here?  I'll have more updates for 'yum update' (and equivalently, for new installs)

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Curiously, even on the existing repos, 'yum update' gives some output.  This is because we don't run 'yum update' in the kickstart script -- so we're building only from the 'os' repo.  We can fix that during this upgrade.  Here's those updates (so to be clear, this is still CentOS-6.2):

Installing:
 kernel                     2.6.32-220.7.1.el6
Updating:
 at                         3.1.10-43.el6_2.1
 dhclient                   12:4.1.1-25.P1.el6_2.1
 dhcp-common                12:4.1.1-25.P1.el6_2.1
 dracut                     004-256.el6_2.1
 dracut-kernel              004-256.el6_2.1
 facter                     1:1.7.1-1.el6
 freetype                   2.4.12-6.el6.1
 gmp                        4.3.1-7.el6_2.2
 hiera                      1.2.1-1.el6
 initscripts                9.03.27-1.el6.centos.1
 iptables                   1.4.7-5.1.el6_2
 iptables-ipv6              1.4.7-5.1.el6_2
 jasper-libs                1.900.1-15.el6_1.1
 kernel-firmware            2.6.32-220.7.1.el6
 krb5-libs                  1.9-22.el6_2.1
 libpng                     2:1.2.46-2.el6_2
 libxml2                    2.7.6-4.el6_2.4
 libxml2-python             2.7.6-4.el6_2.4
 module-init-tools          3.9-19.el6_2
 nspr                       4.9.2-1.el6
 openssh                    5.3p1-70.el6_2.2
 openssh-clients            5.3p1-70.el6_2.2
 openssh-server             5.3p1-70.el6_2.2
 passwd                     0.77-4.el6_2.2
 policycoreutils            2.0.83-19.21.el6_2
 puppetlabs-release         6-7
 rng-tools                  2-13.el6_2
 selinux-policy             3.7.19-126.el6_2.10
 selinux-policy-targeted    3.7.19-126.el6_2.10

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

And finally, here's the result of running 'yum update' against the CentOS-6.5 repository.

> abrt                            2.0.4-14.el6.centos       2.0.8-21.el6.centos
> abrt-addon-ccpp                 2.0.4-14.el6.centos       2.0.8-21.el6.centos
> abrt-addon-kerneloops           2.0.4-14.el6.centos       2.0.8-21.el6.centos
> abrt-addon-python               2.0.4-14.el6.centos       2.0.8-21.el6.centos
> abrt-cli                        2.0.4-14.el6.centos       2.0.8-21.el6.centos
> abrt-libs                       2.0.4-14.el6.centos       2.0.8-21.el6.centos
> abrt-tui                        none                      2.0.8-21.el6.centos
> alsa-utils                      1.0.21-3.el6              1.0.22-5.el6
> apr                             1.3.9-3.el6_1.2           1.3.9-5.el6_2
> at                              3.1.10-43.el6             3.1.10-43.el6_2.1
> atk                             1.28.0-2.el6              1.30.0-1.el6
> audit                           2.1.3-3.el6               2.2-2.el6
> audit-libs                      2.1.3-3.el6               2.2-2.el6
> augeas-libs                     0.9.0-4.el6               1.0.0-5.el6
> authconfig                      6.1.12-5.el6              6.1.12-13.el6
> avahi-libs                      0.6.25-11.el6             0.6.25-12.el6
> bash                            4.1.2-8.el6.centos        4.1.2-15.el6_4
> bfa-firmware                    3.0.0.0-1.el6             3.2.21.1-2.el6
> bind-libs                       9.7.3-8.P3.el6_2.2        32:9.8.2-0.17.rc1.el6_4.6
> bind-utils                      9.7.3-8.P3.el6_2.2        32:9.8.2-0.17.rc1.el6_4.6
> binutils                        2.20.51.0.2-5.28.el6      2.20.51.0.2-5.36.el6
> biosdevname                     0.3.11-1.el6              0.5.0-2.el6
> bridge-utils                    1.2-9.el6                 1.2-10.el6
> btparser                        0.13-1.el6                0.17-2.el6
> busybox                         1.15.1-11.el6             1:1.15.1-20.el6
> ca-certificates                 2010.63-3.el6_1.5         2013.1.94-65.0.el6
> centos-release                  6-2.el6.centos.7          6-5.el6.centos.11.2
> chkconfig                       1.3.47-1.el6              1.3.49.3-2.el6_4.1
> coreutils                       8.4-16.el6                8.4-31.el6
> coreutils-libs                  8.4-16.el6                8.4-31.el6
> cpio                            2.10-9.el6                2.10-11.el6_3
> cpp                             4.4.6-3.el6               4.4.7-4.el6
> cpuspeed                        1.5-15.el6                1:1.5-20.el6_4
> createrepo                      0.9.8-4.el6               0.9.9-18.el6
> cryptsetup-luks                 1.2.0-6.el6               1.2.0-7.el6
> cryptsetup-luks-libs            1.2.0-6.el6               1.2.0-7.el6
> cups-libs                       1.4.2-44.el6              1:1.4.2-50.el6_4.5
> curl                            7.19.7-26.el6_1.2         7.19.7-37.el6_4
> cyrus-sasl                      2.1.23-13.el6             2.1.23-13.el6_3.1
> cyrus-sasl-lib                  2.1.23-13.el6             2.1.23-13.el6_3.1
> cyrus-sasl-plain                2.1.23-13.el6             2.1.23-13.el6_3.1
> dash                            0.5.5.1-3.1.el6           0.5.5.1-4.el6
> db4                             4.7.25-16.el6             4.7.25-18.el6_4
> db4-utils                       4.7.25-16.el6             4.7.25-18.el6_4
> dbus                            1.2.24-5.el6_1            1:1.2.24-7.el6_3
> dbus-glib                       0.86-5.el6                0.86-6.el6
> dbus-libs                       1.2.24-5.el6_1            1:1.2.24-7.el6_3
> device-mapper                   1.02.66-6.el6             1.02.79-8.el6
> device-mapper-event             libs-1.02.66-6.el6        1.02.79-8.el6
> device-mapper-event-libs        1.02.66-6.el6             1.02.79-8.el6
> device-mapper-libs              1.02.66-6.el6             1.02.79-8.el6
> device-mapper-persistent-data   none                      0.2.8-2.el6
> dhclient                        4.1.1-25.P1.el6           12:4.1.1-38.P1.el6.centos
> dhcp-common                     4.1.1-25.P1.el6           12:4.1.1-38.P1.el6.centos
> dmraid                          1.0.0.rc16-10.el6         1.0.0.rc16-11.el6
> dmraid-events                   1.0.0.rc16-10.el6         1.0.0.rc16-11.el6
> dracut                          004-256.el6               004-336.el6_5.2
> dracut-kernel                   004-256.el6               004-336.el6_5.2
> e2fsprogs                       1.41.12-11.el6            1.41.12-18.el6
> e2fsprogs-libs                  1.41.12-11.el6            1.41.12-18.el6
> efibootmgr                      0.5.4-9.el6               0.5.4-11.el6
> ethtool                         2.6.33-0.3.el6            2:3.5-1.el6
> expat                           2.0.1-9.1.el6             2.0.1-11.el6_2
> facter                          1.7.0-1.el6               1:1.7.1-1.el6
> file                            5.04-11.el6               5.04-15.el6
> file-libs                       5.04-11.el6               5.04-15.el6
> fprintd                         0.1-19.git04fd09cfa.el6   0.1-21.git04fd09cfa.el6
> fprintd-pam                     0.1-19.git04fd09cfa.el6   0.1-21.git04fd09cfa.el6
> freetype                        2.3.11-6.el6_1.7          2.4.12-6.el6.1
> gawk                            3.1.7-6.el6               3.1.7-10.el6
> gcc                             4.4.6-3.el6               4.4.7-4.el6
> gdb                             7.2-50.el6                7.2-60.el6_4.1
> glib2                           2.22.5-6.el6              2.26.1-3.el6
> glib2-devel                     2.22.5-6.el6              2.26.1-3.el6
> glibc                           2.12-1.47.el6_2.5.i686    2.12-1.132.el6
> glibc-common                    2.12-1.47.el6_2.5         2.12-1.132.el6
> glibc-devel                     2.12-1.47.el6_2.5         2.12-1.132.el6
> glibc-headers                   2.12-1.47.el6_2.5         2.12-1.132.el6
> gmp                             4.3.1-7.el6               4.3.1-7.el6_2.2
> gnupg2                          2.0.14-4.el6              2.0.14-6.el6_4
> gnutls                          2.8.5-4.el6               2.8.5-10.el6_4.2
> grep                            2.6.3-2.el6               2.6.3-4.el6
> grub                            0.97-75.el6               1:0.97-83.el6
> grubby                          7.0.15-2.el6              7.0.15-5.el6
> gtk2                            2.18.9-6.el6.centos       2.20.1-4.el6
> gzip                            1.3.12-18.el6             1.3.12-19.el6_4
> hdparm                          9.16-3.4.el6              9.43-4.el6
> hiera                           1.2.0-1.el6               1.2.1-1.el6
> httpd                           2.2.15-15.el6.centos.1    2.2.15-29.el6.centos
> httpd-tools                     2.2.15-15.el6.centos.1    2.2.15-29.el6.centos
> hwdata                          0.233-7.6.el6             0.233-9.1.el6
> initscripts                     9.03.27-1.el6.centos      9.03.40-2.el6.centos
> iproute                         2.6.32-17.el6             2.6.32-31.el6
> iptables                        1.4.7-4.el6               1.4.7-11.el6
> iptables-ipv6                   1.4.7-4.el6               1.4.7-11.el6
> iputils                         20071127-16.el6           20071127-17.el6_4.2
> irqbalance                      0.55-29.el6               2:1.0.4-6.el6
> iw                              0.9.17-4.el6              3.10-1.1.el6
> jasper-libs                     1.900.1-15.el6            1.900.1-15.el6_1.1
> java-1.6.0-openjdk              1.6.0.0-1.43.1.10.6.el6_2 1:1.6.0.0-1.66.1.13.0.el6
> java-1.6.0-openjdk-devel        1.6.0.0-1.43.1.10.6.el6_2 1:1.6.0.0-1.66.1.13.0.el6
> kernel                          2.6.32-220.el6            2.6.32-431.el6
> kernel-firmware                 2.6.32-220.el6            2.6.32-431.el6
> kernel-headers                  2.6.32-220.7.1.el6        2.6.32-431.el6
> kexec-tools                     2.0.0-209.el6             2.0.0-273.el6
> keyutils-libs                   1.4-3.el6                 1.4-4.el6
> kpartx                          0.4.9-46.el6              0.4.9-72.el6
> krb5-libs                       1.9-22.el6                1.10.3-10.el6_4.6
> libSM                           1.1.0-7.1.el6             1.2.1-2.el6
> libX11                          1.3-2.el6                 1.5.0-4.el6
> libX11-common                   1.3-2.el6                 1.5.0-4.el6
> libXau                          1.0.5-1.el6               1.0.6-4.el6
> libXcomposite                   0.4.1-2.el6               0.4.3-4.el6
> libXcursor                      1.1.10-2.el6              1.1.13-6.20130524git8f677eaea.el6
> libXdamage                      1.1.2-1.el6               1.1.3-4.el6
> libXdmcp                        1.0.3-1.el6               1.1.1-3.el6
> libXext                         1.1-3.el6                 1.3.1-2.el6
> libXfixes                       4.0.4-1.el6               5.0-3.el6
> libXfont                        1.4.1-2.el6_1             1.4.5-2.el6
> libXft                          2.1.13-4.1.el6            2.3.1-2.el6
> libXi                           1.3-3.el6                 1.6.1-3.el6
> libXinerama                     1.1-1.el6                 1.1.2-2.el6
> libXmu                          1.0.5-1.el6               1.1.1-2.el6
> libXrandr                       1.3.0-4.el6               1.4.0-1.el6
> libXrender                      0.9.5-1.el6               0.9.7-2.el6
> libXt                           1.0.7-1.el6               1.1.3-1.el6
> libXtst                         1.0.99.2-3.el6            1.2.1-2.el6
> libblkid                        2.17.2-12.4.el6           2.17.2-12.14.el6
> libcom_err                      1.41.12-11.el6            1.41.12-18.el6
> libcurl                         7.19.7-26.el6_1.2         7.19.7-37.el6_4
> libdrm                          2.4.25-2.el6              2.4.45-2.el6
> libevent                        1.4.13-1.el6              1.4.13-4.el6
> libevent-devel                  1.4.13-1.el6              1.4.13-4.el6
> libevent-doc                    none                      1.4.13-4.el6
> libevent-headers                none                      1.4.13-4.el6
> libgcc                          4.4.6-3.el6.i686          4.4.7-4.el6
> libgcrypt                       1.4.5-9.el6               1.4.5-11.el6_4
> libgomp                         4.4.6-3.el6               4.4.7-4.el6
> libjpeg-turbo                   none                      1.2.1-1.el6
> libnl                           1.1-14.el6                1.1.4-2.el6
> libpcap                         1.0.0-6.20091201git117cb5.el6 14:1.4.0-1.20130826git2dbcaa1.el6
> libpciaccess                    none                      0.13.1-2.el6
> libpng                          1.2.46-1.el6_1            2:1.2.49-1.el6_2
> libproxy                        0.3.0-2.el6               0.3.0-4.el6_3
> libproxy-bin                    0.3.0-2.el6               0.3.0-4.el6_3
> libproxy-python                 0.3.0-2.el6               0.3.0-4.el6_3
> libreport                       2.0.5-20.el6              2.0.9-19.el6.centos
> libreport-cli                   2.0.5-20.el6              2.0.9-19.el6.centos
> libreport-compat                none                      2.0.9-19.el6.centos
> libreport-plugin-kerneloops     2.0.5-20.el6              2.0.9-19.el6.centos
> libreport-plugin-logger         2.0.5-20.el6              2.0.9-19.el6.centos
> libreport-plugin-mailx          2.0.5-20.el6              2.0.9-19.el6.centos
> libreport-plugin-reportuploader 2.0.5-20.el6              2.0.9-19.el6.centos
> libreport-plugin-rhtsupport     2.0.5-20.el6              2.0.9-19.el6.centos
> libreport-python                2.0.5-20.el6              2.0.9-19.el6.centos
> libselinux                      2.0.94-5.3.el6            2.0.94-5.3.el6_4.1
> libselinux-ruby                 2.0.94-5.3.el6            2.0.94-5.3.el6_4.1
> libselinux-utils                2.0.94-5.3.el6            2.0.94-5.3.el6_4.1
> libsemanage                     2.0.43-4.1.el6            2.0.43-4.2.el6
> libss                           1.41.12-11.el6            1.41.12-18.el6
> libssh2                         1.2.2-7.el6_1.1           1.4.2-1.el6
> libstdc++                       4.4.6-3.el6.i686          4.4.7-4.el6
> libtalloc                       2.0.1-1.1.el6             2.0.7-2.el6
> libtar                          1.2.11-16.el6             1.2.11-17.el6_4.1
> libtasn1                        2.3-3.el6                 2.3-3.el6_2.1
> libtdb                          1.2.1-3.el6               1.2.10-1.el6
> libtevent                       none                      0.9.18-3.el6
> libtiff                         3.9.4-1.el6_0.3           3.9.4-9.el6_3
> libtirpc                        0.2.1-5.el6               0.2.1-6.el6_4
> libudev                         147-2.40.el6              147-2.51.el6
> libusb1                         1.0.3-1.el6               1.0.9-0.6.rc1.el6
> libuser                         0.56.13-4.el6_0.1         0.56.13-5.el6
> libuuid                         2.17.2-12.4.el6           2.17.2-12.14.el6
> libxcb                          1.5-1.el6                 1.8.1-1.el6
> libxml2                         2.7.6-4.el6               2.7.6-14.el6
> libxml2-python                  2.7.6-4.el6               2.7.6-14.el6
> lm_sensors                      3.1.1-10.el6              3.1.1-17.el6
> lm_sensors-libs                 3.1.1-10.el6              3.1.1-17.el6
> logrotate                       3.7.8-12.el6_0.1          3.7.8-17.el6
> lsof                            4.82-2.el6                4.82-4.el6
> lvm2                            2.02.87-6.el6             2.02.100-8.el6
> lvm2-libs                       2.02.87-6.el6             2.02.100-8.el6
> lzo                             none                      2.03-3.1.el6
> make                            3.81-19.el6               1:3.81-20.el6
> man                             1.6f-29.el6               1.6f-32.el6
> man-pages                       3.22-17.el6               3.22-20.el6
> man-pages-overrides             6.2.3-2.el6               6.5.2-1.el6
> mdadm                           3.2.2-9.el6               3.2.6-7.el6
> microcode_ctl                   1.17-9.el6                1:1.17-17.el6
> mlocate                         0.22.2-3.el6              0.22.2-4.el6
> mod_ssl                         2.2.15-15.el6.centos.1    1:2.2.15-29.el6.centos
> module-init-tools               3.9-17.el6                3.9-21.el6_4
> mysql                           5.1.61-1.el6_2.1          5.1.71-1.el6
> mysql-libs                      5.1.61-1.el6_2.1          5.1.71-1.el6
> neon                            0.29.3-1.2.el6            0.29.3-3.el6_4
> net-snmp                        5.5-37.el6_2.1            1:5.5-49.el6
> net-snmp-libs                   5.5-37.el6_2.1            1:5.5-49.el6
> net-snmp-utils                  5.5-37.el6_2.1            1:5.5-49.el6
> net-tools                       1.60-109.el6              1.60-110.el6_2
> nspr                            4.8.9-3.el6_2             4.10.0-1.el6
> nss                             3.13.1-7.el6_2            3.15.1-15.el6
> nss-softokn                     3.12.9-11.el6             3.14.3-9.el6
> nss-softokn-freebl              3.12.9-11.el6.i686        3.14.3-9.el6
> nss-sysinit                     3.13.1-7.el6_2            3.15.1-15.el6
> nss-tools                       3.13.1-7.el6_2            3.15.1-15.el6
> nss-util                        3.13.1-3.el6_2            3.15.1-3.el6
> ntsysv                          1.3.47-1.el6              1.3.49.3-2.el6_4.1
> numactl                         none                      2.0.7-8.el6
> openldap                        2.4.23-20.el6             2.4.23-32.el6_4.1
> openssh                         5.3p1-70.el6              5.3p1-94.el6
> openssh-clients                 5.3p1-70.el6              5.3p1-94.el6
> openssh-server                  5.3p1-70.el6              5.3p1-94.el6
> p11-kit                         none                      0.18.5-2.el6
> p11-kit-trust                   none                      0.18.5-2.el6
> pam                             1.1.1-10.el6              1.1.1-17.el6
> pango                           1.28.1-3.el6_0.5.1.centos 1.28.1-7.el6_3
> parted                          2.1-17.el6                2.1-21.el6
> passwd                          0.77-4.el6                0.77-4.el6_2.2
> pciutils                        3.1.4-11.el6              3.1.10-2.el6
> pciutils-libs                   3.1.4-11.el6              3.1.10-2.el6
> pcre                            7.8-3.1.el6               7.8-6.el6
> perl                            URI-1.40-2.el6            4:5.10.1-136.el6
> perl-Compress-Raw-Zlib          2.023-119.el6_1.1         1:2.021-136.el6
> perl-Compress-Zlib              2.020-119.el6_1.1         2.021-136.el6
> perl-Digest-SHA                 5.47-119.el6_1.1          1:5.47-136.el6
> perl-IO-Compress-Base           2.020-119.el6_1.1         2.021-136.el6
> perl-IO-Compress-Zlib           2.020-119.el6_1.1         2.021-136.el6
> perl-Module-Pluggable           3.90-119.el6_1.1          1:3.90-136.el6
> perl-Pod-Escapes                1.04-119.el6_1.1          1:1.04-136.el6
> perl-Pod-Simple                 3.13-119.el6_1.1          1:3.13-136.el6
> perl-libs                       5.10.1-119.el6_1.1        4:5.10.1-136.el6
> perl-version                    0.77-119.el6_1.1          3:0.77-136.el6
> pixman                          0.18.4-1.el6_0.1          0.26.2-5.el6_4
> plymouth                        0.8.3-24.el6.centos       0.8.3-27.el6.centos
> plymouth-core-libs              0.8.3-24.el6.centos       0.8.3-27.el6.centos
> plymouth-scripts                0.8.3-24.el6.centos       0.8.3-27.el6.centos
> pm-utils                        1.2.5-9.el6               1.2.5-10.el6
> policycoreutils                 2.0.83-19.18.el6          2.0.83-19.39.el6
> polkit                          0.96-2.el6_0.1            0.96-5.el6_4
> postgresql-libs                 8.4.9-1.el6_1.1           8.4.18-1.el6_4
> prelink                         0.4.6-3.el6               0.4.6-3.1.el6_4
> procps                          3.2.8-21.el6              3.2.8-25.el6
> psacct                          6.3.2-63.el6              6.3.2-63.el6_3.3
> pulseaudio-libs                 0.9.21-13.el6             0.9.21-14.el6_3
> puppetlabs-release              6-6                       6-7
> python                          2.6.6-29.el6              2.6.6-51.el6
> python-ethtool                  0.6-1.el6                 0.6-5.el6
> python-libs                     2.6.6-29.el6              2.6.6-51.el6
> python-urlgrabber               3.9.1-8.el6               3.9.1-9.el6
> ql2400-firmware                 5.06.02-1.el6             7.00.01-1.el6
> ql2500-firmware                 5.06.02-1.el6             7.00.01-1.el6
> quota                           3.17-16.el6               1:3.17-20.el6
> readahead                       1.5.6-1.el6               1:1.5.6-2.el6
> readline                        6.0-3.el6                 6.0-4.el6
> redhat-logos                    60.0.14-10.el6.centos     60.0.14-12.el6.centos
> rng-tools                       2-10.el6                  2-13.el6_2
> rpcbind                         0.2.0-8.el6               0.2.0-11.el6
> rpm                             4.8.0-19.el6              4.8.0-37.el6
> rpm-build                       4.8.0-19.el6              4.8.0-37.el6
> rpm-libs                        4.8.0-19.el6              4.8.0-37.el6
> rpm-python                      4.8.0-19.el6              4.8.0-37.el6
> rpmdevtools                     7.5-1.el6                 7.5-2.el6
> rsync                           3.0.6-5.el6_0.1           3.0.6-9.el6_4.1
> rsyslog                         4.6.2-12.el6              5.8.10-8.el6
> rsyslog-mysql                   4.6.2-12.el6              5.8.10-8.el6
> ruby                            rgen-0.6.2-1.el6          1.8.7.352-13.el6
> ruby-irb                        1.8.7.352-10.el6_4        1.8.7.352-13.el6
> ruby-libs                       1.8.7.352-10.el6_4        1.8.7.352-13.el6
> ruby-rdoc                       1.8.7.352-10.el6_4        1.8.7.352-13.el6
> rubygems                        1.3.7-1.el6               1.3.7-5.el6
> samba-client                    3.5.10-114.el6            3.6.9-164.el6
> samba-common                    3.5.10-114.el6            3.6.9-164.el6
> samba-winbind                   clients-3.5.10-114.el6    3.6.9-164.el6
> samba-winbind-clients           3.5.10-114.el6            3.6.9-164.el6
> sed                             4.2.1-7.el6               4.2.1-10.el6
> selinux-policy                  3.7.19-126.el6            3.7.19-231.el6
> selinux-policy-targeted         3.7.19-126.el6            3.7.19-231.el6
> setup                           2.8.14-13.el6             2.8.14-20.el6_4.1
> setuptool                       1.19.9-3.el6              1.19.9-4.el6
> shared-mime-info                none                      0.70-4.el6
> smartmontools                   5.39.1-5.el6              1:5.43-1.el6
> snappy                          none                      1.1.0-1.el6
> sos                             2.2-17.el6                2.2-47.el6.centos
> subversion                      1.6.11-2.el6_1.4          1.6.11-9.el6_4
> subversion-perl                 1.6.11-2.el6_1.4          1.6.11-9.el6_4
> sysstat                         9.0.4-18.el6              9.0.4-22.el6
> systemtap-runtime               1.6-4.el6                 2.3-3.el6
> sysvinit-tools                  2.87-4.dsf.el6            2.87-5.dsf.el6
> tar                             1.23-3.el6                2:1.23-11.el6
> tcpdump                         4.0.0-3.20090921gitdf3cb4.1.el6 14:4.0.0-3.20090921gitdf3cb4.2.el6
> tcsh                            6.17-14.el6               6.17-24.el6
> telnet                          0.17-47.el6               1:0.17-47.el6_3.1
> tzdata-java                     2011n-2.el6               2013g-1.el6
> udev                            147-2.40.el6              147-2.51.el6
> upstart                         0.6.5-10.el6              0.6.5-12.el6_4.1
> util-linux-ng                   2.17.2-12.4.el6           2.17.2-12.14.el6
> vim-common                      7.2.411-1.6.el6           2:7.2.411-1.8.el6
> vim-enhanced                    7.2.411-1.6.el6           2:7.2.411-1.8.el6
> virt-what                       1.11-1.1.el6              1.11-1.2.el6
> wget                            1.12-1.4.el6              1.12-1.8.el6
> xinetd                          2.3.14-33.el6             2:2.3.14-39.el6_4
> xkeyboard-config                2.3-1.el6                 2.6-6.el6
> xmlrpc-c                        1.16.24-1200.1840.el6_1.4 1.16.24-1210.1840.el6
> xmlrpc-c-client                 1.16.24-1200.1840.el6_1.4 1.16.24-1210.1840.el6
> xorg-x11-drv-ati-firmware       6.14.2-7.el6              7.1.0-3.el6
> xorg-x11-server-Xvfb            1.10.4-6.el6_2.1.0.1.centos 1.13.0-23.el6.centos
> xorg-x11-server-common          1.10.4-6.el6_2.1.0.1.centos 1.13.0-23.el6.centos
> xorg-x11-xkb-utils              7.4-6.el6                 7.7-4.el6
> yum                             utils-1.1.30-10.el6       3.2.29-40.el6.centos
> yum-plugin-fastestmirror        1.1.30-10.el6             1.1.30-14.el6
> yum-plugin-security             1.1.30-10.el6             1.1.30-14.el6
> yum-utils                       1.1.30-10.el6             1.1.30-14.el6
> zlib                            1.2.3-27.el6.i686         1.2.3-29.el6

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

The last bit is that there are mock_mozilla configs pointing to these repositories, and we will want to control change there very closely.

I think that this is actually very easy: set up *new* mock configs named mozilla-centos65-*.cfg, pointing to baseurl=http://repos/repos/yum/mirrors/centos/6/2013-12-03/os/x86_64, and reconfigure the existing mozilla-centos6-*.cfg to point to baseurl=http://repos/repos/yum/mirrors/centos/6/2012-03-07/os/x86_64 (replacing the 'latest' symlink with an explicit target).  Then set mozilla-central to build against the new configs, and let that change ride the trains.  Once the trains are done (counting ESR) we can delete the old CentOS-6.2 repositories.

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

OK, so that's a bunch of data to chew on!

I sent a separate email to get more eyes on this, and John replied asking about plans.  There aren't any yet - that's the next step - but we need to make some shortly.

There are two actions we have to work with: change repositories with no 'yum update', having the effects in comment 5; and reimages or 'yum update' runs, either of which would have the effects in comment 7.

Aside from the mock environments, the affected machines are the servers and linux builders.  Riding trains is not an option for servers (since they're not branch-specific) and would involve a lot of work and overhead for builders.

CentOS (well, RedHat) is very conservative in the upgrades made to packages, as the version numbers above show, so I don't expect any big changes.

I think that with enough concentration and some targeted testing, we can convince ourselves that these changes are benign, and just land the result someday.  We can always "pin" package versions in puppet where that is deemed necessary.

Have I missed anything?  Other inputs?

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

Well, that went over like a lead balloon.

OpenStack is probably going to force this, as it requires at least CentOS-6.4.

An alternative approach here is to treat CentOS-6.5 as another OS, just like we did with Ubuntu.  That will require some conditionals here and there in the puppet code, but fewer than for Ubuntu.  And it means that we can upgrade either by reinstalling or by manual 'yum upgrade' (the latter being much easier for puppet masters, for example).

I'll start hacking on that in relabs.

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

Repo setup:

lrwxrwxrwx 1 root       root         13 Mar 12 14:19 6.2 -> 6/2012-03-07/

CENTOS_FULL=6.5
rsync -v -n -aP    --filter='-r centos-version.txt'   --exclude isos --exclude drpms --exclude centosplus --exclude xen4 --exclude fasttrack --exclude contrib --exclude cr    --delete --delete-excluded    rsync://linux.mirrors.es.net/centos/$CENTOS_FULL/ /data/repos/yum/mirrors/centos/$CENTOS_FULL/

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug773769.patch

With the repos in place, this configures things to use the appropriate CentOS repos for the running version.

Note that this doesn't update EPEL at all.

Comment 13

[Justin Wood (:Callek)]

Without reading full bug context atm, I'd personally like a method where we can do essentially `yum upgrade` + keep the ensure/enforced versions the same (incase there are newer versions of those packages).... and especially make sure that newly puppetized things with said ensures are fine.

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

I'd like to focus on upgrade by re-imaging -- it's much more time-definite, and prevents any surprises on re-image.  That said, I'm going to experiment with a yum-upgrade process since it will be way easier for puppetmasters.

Comment 15

[Dustin J. Mitchell [:dustin] (he/him)]

Running a 'yum upgrade' followed by a 'puppet agent --test' works for a toplevel::base system, at least.  However, it leaves /etc/issue alone, since it has been modified, which is not what we want (the existing /etc/issue still says CentOS-6.2).  The full list of .rpmnew files is

warning: /etc/issue created as /etc/issue.rpmnew
warning: /etc/group created as /etc/group.rpmnew
warning: /etc/gshadow created as /etc/gshadow.rpmnew
warning: /etc/shadow created as /etc/shadow.rpmnew
warning: /etc/ntp.conf created as /etc/ntp.conf.rpmnew
warning: /etc/ssh/sshd_config created as /etc/ssh/sshd_config.rpmnew

These could probably be merged via some kind of script that wraps 'yum upgrade' to merge issue and the authentication files.  ntp.conf and sshd_config are fully puppet-managed, so it's just as well that rpm didn't overwrite those.

In general, though, I don't think that we should rely on this as a method to upgrade a system, as there are probably other gotchas than these six files.  We've designed a system that makes reimages easy, where a freshly-reimaged system is a known state, and where services are resilient to loss of a single host.  Now, we should take advantage of that design to do upgrades the easy, reliable way, rather than a hacky, fragile way.

Comment 16

[Justin Wood (:Callek)]

(In reply to Dustin J. Mitchell [:dustin] from comment #15)
> In general, though, I don't think that we should rely on this as a method to
> upgrade a system, as there are probably other gotchas than these six files. 
> We've designed a system that makes reimages easy, where a freshly-reimaged
> system is a known state, and where services are resilient to loss of a
> single host.  Now, we should take advantage of that design to do upgrades
> the easy, reliable way, rather than a hacky, fragile way.

Grrr I hate when you convince me with logic. ;-) 

... that said, I'd think it worthwhile to at least peek at what the differences are for those .rpmnew files, so we can take any appropriate updates.

Comment 17

[Dustin J. Mitchell [:dustin] (he/him)]

I had a look, and I don't see anything substantial.  In fact, the group, gshadow, and shadow .rpmnew files are all gone, which I assume means they were automatically merged.  I don't see anything to worry about in the others.

Comment 18

[Justin Wood (:Callek)]

Comment on attachment 8390641 [details] [diff] [review]
bug773769.patch

Review of attachment 8390641 [details] [diff] [review]:
-----------------------------------------------------------------

Ok, This is good (especially for making sure we don't break reimages to deploy this)

That said, I'd love a method to assert a valid facter::operatingsystemrelease for specific node types. In order to prevent deploying an old OS-Release on new reimages once we migrate a class of slaves/machines to the new OS type.

(e.g. by something like an include asserts::centos-latest//asserts::latest etc.)

Comment 19

[Dustin J. Mitchell [:dustin] (he/him)]

That's a good idea.  There's nothing preventing us from trying to bring up an Ubuntu buildmaster right now.  It might even work :)

Comment 20

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8390641 [details] [diff] [review]
bug773769.patch

and merged to production