1067586 - Install extra packages on supportadm to allow package compilation during SUMO deploy.

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Community Platform, task)

Description

[Michael Cooper [:mythmon]]

SUMO is currently in the midst of changing our deployments to use a Python virtualenv and use peep [0] to install things. This is a large improvement in developer workflow, without compromising security. (bug 1035319).

One of the challenges of this is trying to get an environment that uses both local packages in the virtualenv, and system packages in the form of RPM-installed. We have been fighting this for about two weeks now, and have decided we don't want to use this anymore.

Because of this, I would like to change the deployment process of SUMO so that we build and install the compiled packages (the ones formerly installed via RPMs) within our own deploy scripts via peep.

To achieve this, we need a few things installed on the build server (supportadm.private.phx1.mozilla.com). On our Ubuntu development VMs, the list of packages we need to install are:

python-dev
libssl-dev
libmysqlclient-dev
libxml2-dev
libxslt1-dev

I'm not sure what the RHEL packages names are. These are probably build dependencies for the RPM versions of the python packages. The Python packages we want to install via peep/pip are:

MySQL-python==1.2.3c1
Jinja2==2.5.2
PIL==1.1.7
lxml==2.2.6
pyOpenSSL==0.13

There should already be one or more RPMs for these python packages on supportadm, as well as all the webheads and celery workers in the SUMO cluster.

[0]: https://github.com/erikrose/peep

Comment 1

[Chris Turra [:cturra]]

(In reply to Mike Cooper [:mythmon] from comment #0)
> I'm not sure what the RHEL packages names are. These are probably build
> dependencies for the RPM versions of the python packages. The Python
> packages we want to install via peep/pip are:
> 
> MySQL-python==1.2.3c1

already present.

 [[e-mail]1 support-dev.allizom.org]# yum list installed | grep -i mysql-python
 MySQL-python.x86_64               1.2.3-0.3.c1.1.el6       @rhel-x86_64-server-6


> Jinja2==2.5.2

already present.

 [[e-mail]1 support-dev.allizom.org]# yum list installed | grep -i jinja2
 Jinja2.x86_64                     2.5.5-2                  @mozilla


> PIL==1.1.7

updated from 1.1.6

 [[e-mail]1 support-dev.allizom.org]# yum list installed | grep -i python-imaging
 python-imaging.x86_64             1.1.7-19.el6             @mozilla


> lxml==2.2.6

newest version available through rhel (redhat enterprise linux) is 2.2.3-1. 

 [[e-mail]1 support-dev.allizom.org]# yum list installed | grep -i python-lxml
 python-lxml.x86_64                2.2.3-1.1.el6            @rhel-x86_64-server-6


> pyOpenSSL==0.13

we had to wrap a custom pyOpenSSL package b/c verson 0.10-2 is required for the rhel package system. more details on how to use the pyOpenSSLmoz package can be found => https://bugzilla.mozilla.org/show_bug.cgi?id=830496#c38

 [[e-mail]1 support-dev.allizom.org]# yum list installed | grep -i pyOpenSSL
 pyOpenSSL.x86_64      0.10-2.el6   @anaconda-RedHatEnterpriseLinux-201009221801.x86_64/6.0
 pyOpenSSLmoz.x86_64   1:0.13-1     @mozilla


:mythmon - please let me know if these packages and their versions all work out for you.

Comment 2

[Dean Johnson [:deanj]]

Did you happen to install the RHEL equivalent of these packages as well?

> python-dev
> libssl-dev
> libmysqlclient-dev
> libxml2-dev
> libxslt1-dev

The problem wasn't that the compiled packages weren't up to date, our problem was that we couldn't peep install the listed packages due to missing header files from the above system packages.

Comment 3

[Chris Turra [:cturra]]

oh, sorry, i miss understood that part.

here is a list of what i believe are the equivalent packages. can you please versify the description matches your expectations?

> python-dev
python-devel.x86_64 : The libraries and header files needed for Python development

> libssl-dev
openssl-devel.x86_64 : Files for development of applications which will use OpenSSL

> libmysqlclient-dev
mysql-devel.x86_64 : Files for development of MySQL applications

> libxml2-dev
libxml2-devel.x86_64 : Libraries, includes, etc. to develop XML and HTML applications

> libxslt1-dev
libxslt-python.x86_64 : Python bindings for the libxslt library

Comment 4

[Dean Johnson [:deanj]]

Looks good to me! :)

Comment 5

[Chris Turra [:cturra]]

sweet. packages installed on the last puppet run :)

 Notice: /Stage[main]/Webapp::Admin::Support/Package[openssl-devel]/ensure: created
 Notice: /Stage[main]/Webapp::Admin::Support/Package[python-devel]/ensure: created
 Notice: /Stage[main]/Webapp::Admin::Support/Package[libxslt-python]/ensure: created
 Notice: /Stage[main]/Webapp::Admin::Support/Package[mysql-devel]/ensure: created
 Notice: /Stage[main]/Webapp::Admin::Support/Package[libxml2-devel]/ensure: created

Comment 6

[Michael Cooper [:mythmon]]

Sorry cturra, it looks like one of the packages wasn't quite right. libxslt-python.x86_64 is Python bindings for libxslt, which would be the python package we are trying to build. What we need is development headers for libxslt. Maybe something like libxslt-devel ? I'm not sure what the right package is, but I can say it's not libxslt-python. We should have caught this when you asked about it.

These packages did help, most of the pytohn dependencies installed without error in my test environment (in my homedir), and it's just failing on the python package that requires libxslt.

Comment 7

[Chris Turra [:cturra]]

i have swapped libxslt-python with libxslt-devel as requested. let me know how you make out?

  Notice: /Stage[main]/Webapp::Admin::Support/Package[libxslt-devel]/ensure: created

Comment 8

[Michael Cooper [:mythmon]]

cturra: thanks, this fixed the problem in my small scale test. To be honest, I'm surprised we didn't have any follow up packages we needed. Hopefully it works well with the full site too.

dean: go ahead and deploy again with the deploy script that installs everything and doesn't use system packages. Remember to deploy twice! If it works, you should mark this bug as VERIFIED/FIXED.

Comment 9

[Dean Johnson [:deanj]]

Works great and the celery queue is now clearing itself out! Thanks again cturra!