1060516 - [tracker] Deprecate EPEL

Status: RESOLVED WONTFIX

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

We currently have a 2.5-year-old mirror of EPEL.  Among other things, it does not contain a copy of git-annex, which I'd like for bug 1050915.

We don't use EPEL for mock builds, and anyway we'll soon be free of maintaining those repos.

I'll do a quick survey of what we're installing from EPEL, but I bet it's not anything especially critical.  If not, I think we should re-mirror the latest version.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Here's the analysis for pretty much every centos host type:

> PyYAML : epel                                                 puppetmaster
> dpkg : epel                                                   puppetmaster
> dpkg-devel : epel                                             puppetmaster
> fping : epel                     builder foopy master mozpool puppetmaster signing slaveapi
> ganglia : epel                   builder foopy master mozpool              signing slaveapi
> hddtemp : epel                   builder foopy master mozpool              signing slaveapi
> libconfuse : epel                builder foopy master mozpool              signing slaveapi
> libesmtp : epel                  builder foopy master mozpool              signing slaveapi
> libev : epel                                                  puppetmaster
> libgdiplus : epel                                                          signing
> liboping : epel                  builder foopy master mozpool              signing slaveapi
> libyaml : epel                                                puppetmaster
> mono-core : epel                                                           signing
> mono-data : epel                                                           signing
> mono-data-sqlite : epel                                                    signing
> mono-devel : epel                                                          signing
> mono-extras : epel                                                         signing
> mono-web : epel                                                            signing
> mono-winforms : epel                                                       signing
> nagios-common : epel             builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins : epel            builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-all : epel        builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-breeze : epel     builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-by_ssh : epel     builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-cluster : epel    builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-dhcp : epel       builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-dig : epel        builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-disk : epel       builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-disk_smb : epel   builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-dns : epel        builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-dummy : epel      builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-file_age : epel   builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-flexlm : epel     builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-fping : epel      builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-game : epel       builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-hpjd : epel       builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-http : epel       builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-icmp : epel       builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-ide_smart : epel  builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-ircd : epel       builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-ldap : epel       builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-load : epel       builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-log : epel        builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-mailq : epel      builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-mrtg : epel       builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-mrtgtraf : epel   builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-mysql : epel      builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-nagios : epel     builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-nrpe : epel       builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-nt : epel         builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-ntp : epel        builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-nwstat : epel     builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-oracle : epel     builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-overcr : epel     builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-perl : epel       builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-pgsql : epel      builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-ping : epel       builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-procs : epel      builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-real : epel       builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-rpc : epel        builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-sensors : epel    builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-smtp : epel       builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-snmp : epel       builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-ssh : epel        builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-swap : epel       builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-tcp : epel        builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-time : epel       builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-ups : epel        builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-users : epel      builder foopy master mozpool puppetmaster signing slaveapi
> nagios-plugins-wave : epel       builder foopy master mozpool puppetmaster signing slaveapi
> nrpe : epel                      builder foopy master mozpool puppetmaster signing slaveapi
> nut-client : epel                builder foopy master mozpool              signing slaveapi
> perl-LockFile-Simple : epel                                   puppetmaster
> perl-Regexp-Common : epel        builder foopy master mozpool              signing slaveapi
> pigz : epel                      builder
> protobuf : epel                  builder foopy master mozpool              signing slaveapi
> protobuf-c : epel                builder foopy master mozpool              signing slaveapi
> protobuf-compiler : epel         builder foopy master mozpool              signing slaveapi
> python-meld3 : epel              builder              mozpool
> qstat : epel                     builder foopy master mozpool puppetmaster signing slaveapi
> ruby-augeas : epel                                                         signing
> ruby-shadow : epel                                                         signing
> rubygem-daemon_controller : epel                              puppetmaster
> rubygem-fastthread : epel                                     puppetmaster
> rubygem-rack : epel                                           puppetmaster
> varnish-libs : epel              builder foopy master mozpool              signing slaveapi

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

I'm not at all worried about the nagios/nrpe stuff (including, oddly, qstat), nor the puppetmaster-only stuff.

pigz is a parallel gzip implementation used for builders.  Assuming hte new EPEL version isn't broken, I don't see a reason not to upgrade that.  Similarly with the protobuf stuff

I have no idea why nut-client is installed.  rpm -q --whatrequires doesn't show anything requiring it.  At a guess, it might be collectd (which is probably also responsible for varnish-libs).

libgdiplus and the mono stuff could potentially be important to signing, so we'll need to test that.  Ben, do you have an idea how critical that is, how we might go about testing it, and/or who might know better?

Comment 3

[[e-mail] (:bhearsum)]

(In reply to Dustin J. Mitchell [:dustin] from comment #2)
> libgdiplus and the mono stuff could potentially be important to signing, so
> we'll need to test that.  Ben, do you have an idea how critical that is, how
> we might go about testing it, and/or who might know better?

I'm not sure about libgdiplus, but Mono is used as part of Authenticode signing. How major of an upgrade are we talking about here? One thing we could look at doing is manually upgrading mono on one signing server and seeing how that fares.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Mono goes from 2.4.3.1-3 to 2.10.8-4.  libgdiplus is a requirement of mono, and similarly goes from 2.4.2-3 to 2.10-1.

So how about:
 - I kill the signing processes on signing4.srv.releng.scl3, upgrade manually, and reboot
   (and disable puppet)
 - You re-enable the dep and nightly signers (but not release, just in case)
 - ..monitor the results..
 - Revert by re-imaging if there are any issues

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

(see comment 4)

Comment 6

[[e-mail] (:bhearsum)]

(In reply to Dustin J. Mitchell [:dustin] from comment #4)
> Mono goes from 2.4.3.1-3 to 2.10.8-4.  libgdiplus is a requirement of mono,
> and similarly goes from 2.4.2-3 to 2.10-1.
> 
> So how about:
>  - I kill the signing processes on signing4.srv.releng.scl3, upgrade
> manually, and reboot
>    (and disable puppet)
>  - You re-enable the dep and nightly signers (but not release, just in case)
>  - ..monitor the results..
>  - Revert by re-imaging if there are any issues

Sounds fine to me, as long as it's not today.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

We currently have two versions of EPEL - 2012-03-07 and 2014-05-06.  latest/ points to 2012-03-07, and that's what puppet uses.  But grepping the access logs on the puppet masters shows *something* installing some python packages from 2014-05-06 a few times a day.  MXR finds http://mxr.mozilla.org/build/source/cloud-tools/ami_configs/releng-public.repo

Rail made this change in http://hg.mozilla.org/build/cloud-tools/rev/2aff440e33b0#l3.13 but can't recall why.  Ideally that would point to latest/, so that it floats when that symlink changes.  So I'll put in a patch to revert it to latest/ and we'll see what happens with the AMIs tomorrow.

I'm also mirroring EPEL to the relabs puppetmasters now, so we can try the signing upgrade tomorrow.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug1060516-cloud-tools.patch

Comment 9

[Rail Aliiev [:rail]]

Comment on attachment 8500493 [details] [diff] [review]
bug1060516-cloud-tools.patch

https://hg.mozilla.org/build/cloud-tools/rev/5ec409babbac

I had to backout this patch because cloud-init in that repo is too old.

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

OK, we'll re-land once latest/ points to a 2014 repo :)

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

Ugh, I hate packages and repos.

Error: Package: libgdiplus-2.10-1.el6.x86_64 (epel)
           Requires: libjpeg.so.62(LIBJPEG_6.2)(64bit)

[[e-mail] ~]# rpm -q --provides libjpeg
libjpeg.so.62()(64bit)
libjpeg = 6b-46.el6
libjpeg(x86-64) = 6b-46.el6

so we have the right version of libjpeg installed, it's just not tagged correctly.  I assume a newer version of the CentOS 6.2 repos would have an updated libjpeg, but we can't use that..

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

I think the conclusion is, we can never upgrade EPEL, and in fact we shouldn't be using it at all -- we should be using custom repos instead.  So, I'd like to make it a non-default repo.  We should also make a custom cloud-init repo for cloud-tools (comment 7 through comment 10).

Comment 13

[:kanban-engops]

A Pivotal Tracker story has been created for this Bug: https://www.pivotaltracker.com/story/show/80405330

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

The fun part is going to be testing all of those, as they all require reinstalls to verify the correct packages are installed.

I'd like to get repos and patches worked up for all of them, then put them in my environment and reinstall some relabs boxes with each flavor of machine - that should test everything at once.

Comment 15

[Amy Rich [:arr] [:arich]]

Unfortunately we don't have the cycles to work on this.