Install extra packages on supportadm to allow package compilation during SUMO deploy.

VERIFIED FIXED

Status

VERIFIED FIXED
4 years ago
2 months ago

People

(Reporter: mythmon, Assigned: cturra)

Tracking

Details

(Whiteboard: [kanban:https://kanbanize.com/ctrl_board/4/1276] )

(Reporter)

Description

4 years ago
SUMO is currently in the midst of changing our deployments to use a Python virtualenv and use peep [0] to install things. This is a large improvement in developer workflow, without compromising security. (bug 1035319).

One of the challenges of this is trying to get an environment that uses both local packages in the virtualenv, and system packages in the form of RPM-installed. We have been fighting this for about two weeks now, and have decided we don't want to use this anymore.

Because of this, I would like to change the deployment process of SUMO so that we build and install the compiled packages (the ones formerly installed via RPMs) within our own deploy scripts via peep.

To achieve this, we need a few things installed on the build server (supportadm.private.phx1.mozilla.com). On our Ubuntu development VMs, the list of packages we need to install are:

python-dev
libssl-dev
libmysqlclient-dev
libxml2-dev
libxslt1-dev

I'm not sure what the RHEL packages names are. These are probably build dependencies for the RPM versions of the python packages. The Python packages we want to install via peep/pip are:

MySQL-python==1.2.3c1
Jinja2==2.5.2
PIL==1.1.7
lxml==2.2.6
pyOpenSSL==0.13

There should already be one or more RPMs for these python packages on supportadm, as well as all the webheads and celery workers in the SUMO cluster.

[0]: https://github.com/erikrose/peep

Updated

4 years ago
Whiteboard: [kanban:https://kanbanize.com/ctrl_board/4/1276]

Updated

4 years ago
Assignee: server-ops-webops → cturra
(Assignee)

Comment 1

4 years ago
(In reply to Mike Cooper [:mythmon] from comment #0)
> I'm not sure what the RHEL packages names are. These are probably build
> dependencies for the RPM versions of the python packages. The Python
> packages we want to install via peep/pip are:
> 
> MySQL-python==1.2.3c1

already present.

 [root@supportadm.private.phx1 support-dev.allizom.org]# yum list installed | grep -i mysql-python
 MySQL-python.x86_64               1.2.3-0.3.c1.1.el6       @rhel-x86_64-server-6


> Jinja2==2.5.2

already present.

 [root@supportadm.private.phx1 support-dev.allizom.org]# yum list installed | grep -i jinja2
 Jinja2.x86_64                     2.5.5-2                  @mozilla


> PIL==1.1.7

updated from 1.1.6

 [root@supportadm.private.phx1 support-dev.allizom.org]# yum list installed | grep -i python-imaging
 python-imaging.x86_64             1.1.7-19.el6             @mozilla


> lxml==2.2.6

newest version available through rhel (redhat enterprise linux) is 2.2.3-1. 

 [root@supportadm.private.phx1 support-dev.allizom.org]# yum list installed | grep -i python-lxml
 python-lxml.x86_64                2.2.3-1.1.el6            @rhel-x86_64-server-6


> pyOpenSSL==0.13

we had to wrap a custom pyOpenSSL package b/c verson 0.10-2 is required for the rhel package system. more details on how to use the pyOpenSSLmoz package can be found => https://bugzilla.mozilla.org/show_bug.cgi?id=830496#c38

 [root@supportadm.private.phx1 support-dev.allizom.org]# yum list installed | grep -i pyOpenSSL
 pyOpenSSL.x86_64      0.10-2.el6   @anaconda-RedHatEnterpriseLinux-201009221801.x86_64/6.0
 pyOpenSSLmoz.x86_64   1:0.13-1     @mozilla


:mythmon - please let me know if these packages and their versions all work out for you.
Flags: needinfo?(mcooper)
Did you happen to install the RHEL equivalent of these packages as well?

> python-dev
> libssl-dev
> libmysqlclient-dev
> libxml2-dev
> libxslt1-dev

The problem wasn't that the compiled packages weren't up to date, our problem was that we couldn't peep install the listed packages due to missing header files from the above system packages.
(Assignee)

Comment 3

4 years ago
oh, sorry, i miss understood that part.

here is a list of what i believe are the equivalent packages. can you please versify the description matches your expectations?

> python-dev
python-devel.x86_64 : The libraries and header files needed for Python development

> libssl-dev
openssl-devel.x86_64 : Files for development of applications which will use OpenSSL

> libmysqlclient-dev
mysql-devel.x86_64 : Files for development of MySQL applications

> libxml2-dev
libxml2-devel.x86_64 : Libraries, includes, etc. to develop XML and HTML applications

> libxslt1-dev
libxslt-python.x86_64 : Python bindings for the libxslt library
Flags: needinfo?(mcooper) → needinfo?(djohnson)
Looks good to me! :)
Flags: needinfo?(djohnson)
(Assignee)

Comment 5

4 years ago
sweet. packages installed on the last puppet run :)

 Notice: /Stage[main]/Webapp::Admin::Support/Package[openssl-devel]/ensure: created
 Notice: /Stage[main]/Webapp::Admin::Support/Package[python-devel]/ensure: created
 Notice: /Stage[main]/Webapp::Admin::Support/Package[libxslt-python]/ensure: created
 Notice: /Stage[main]/Webapp::Admin::Support/Package[mysql-devel]/ensure: created
 Notice: /Stage[main]/Webapp::Admin::Support/Package[libxml2-devel]/ensure: created
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
(Reporter)

Comment 6

4 years ago
Sorry cturra, it looks like one of the packages wasn't quite right. libxslt-python.x86_64 is Python bindings for libxslt, which would be the python package we are trying to build. What we need is development headers for libxslt. Maybe something like libxslt-devel ? I'm not sure what the right package is, but I can say it's not libxslt-python. We should have caught this when you asked about it.

These packages did help, most of the pytohn dependencies installed without error in my test environment (in my homedir), and it's just failing on the python package that requires libxslt.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 7

4 years ago
i have swapped libxslt-python with libxslt-devel as requested. let me know how you make out?

  Notice: /Stage[main]/Webapp::Admin::Support/Package[libxslt-devel]/ensure: created
Status: REOPENED → RESOLVED
Last Resolved: 4 years ago4 years ago
Resolution: --- → FIXED
(Reporter)

Comment 8

4 years ago
cturra: thanks, this fixed the problem in my small scale test. To be honest, I'm surprised we didn't have any follow up packages we needed. Hopefully it works well with the full site too.

dean: go ahead and deploy again with the deploy script that installs everything and doesn't use system packages. Remember to deploy twice! If it works, you should mark this bug as VERIFIED/FIXED.
Works great and the celery queue is now clearing itself out! Thanks again cturra!
Status: RESOLVED → VERIFIED
(Reporter)

Updated

4 years ago
Blocks: 1070020

Updated

2 months ago
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.