SUMO is currently in the midst of changing our deployments to use a Python virtualenv and use peep  to install things. This is a large improvement in developer workflow, without compromising security. (bug 1035319). One of the challenges of this is trying to get an environment that uses both local packages in the virtualenv, and system packages in the form of RPM-installed. We have been fighting this for about two weeks now, and have decided we don't want to use this anymore. Because of this, I would like to change the deployment process of SUMO so that we build and install the compiled packages (the ones formerly installed via RPMs) within our own deploy scripts via peep. To achieve this, we need a few things installed on the build server (supportadm.private.phx1.mozilla.com). On our Ubuntu development VMs, the list of packages we need to install are: python-dev libssl-dev libmysqlclient-dev libxml2-dev libxslt1-dev I'm not sure what the RHEL packages names are. These are probably build dependencies for the RPM versions of the python packages. The Python packages we want to install via peep/pip are: MySQL-python==1.2.3c1 Jinja2==2.5.2 PIL==1.1.7 lxml==2.2.6 pyOpenSSL==0.13 There should already be one or more RPMs for these python packages on supportadm, as well as all the webheads and celery workers in the SUMO cluster. : https://github.com/erikrose/peep
(In reply to Mike Cooper [:mythmon] from comment #0) > I'm not sure what the RHEL packages names are. These are probably build > dependencies for the RPM versions of the python packages. The Python > packages we want to install via peep/pip are: > > MySQL-python==1.2.3c1 already present. [email@example.com support-dev.allizom.org]# yum list installed | grep -i mysql-python MySQL-python.x86_64 1.2.3-0.3.c1.1.el6 @rhel-x86_64-server-6 > Jinja2==2.5.2 already present. [firstname.lastname@example.org support-dev.allizom.org]# yum list installed | grep -i jinja2 Jinja2.x86_64 2.5.5-2 @mozilla > PIL==1.1.7 updated from 1.1.6 [email@example.com support-dev.allizom.org]# yum list installed | grep -i python-imaging python-imaging.x86_64 1.1.7-19.el6 @mozilla > lxml==2.2.6 newest version available through rhel (redhat enterprise linux) is 2.2.3-1. [firstname.lastname@example.org support-dev.allizom.org]# yum list installed | grep -i python-lxml python-lxml.x86_64 2.2.3-1.1.el6 @rhel-x86_64-server-6 > pyOpenSSL==0.13 we had to wrap a custom pyOpenSSL package b/c verson 0.10-2 is required for the rhel package system. more details on how to use the pyOpenSSLmoz package can be found => https://bugzilla.mozilla.org/show_bug.cgi?id=830496#c38 [email@example.com support-dev.allizom.org]# yum list installed | grep -i pyOpenSSL pyOpenSSL.x86_64 0.10-2.el6 @anaconda-RedHatEnterpriseLinux-201009221801.x86_64/6.0 pyOpenSSLmoz.x86_64 1:0.13-1 @mozilla :mythmon - please let me know if these packages and their versions all work out for you.
Did you happen to install the RHEL equivalent of these packages as well? > python-dev > libssl-dev > libmysqlclient-dev > libxml2-dev > libxslt1-dev The problem wasn't that the compiled packages weren't up to date, our problem was that we couldn't peep install the listed packages due to missing header files from the above system packages.
oh, sorry, i miss understood that part. here is a list of what i believe are the equivalent packages. can you please versify the description matches your expectations? > python-dev python-devel.x86_64 : The libraries and header files needed for Python development > libssl-dev openssl-devel.x86_64 : Files for development of applications which will use OpenSSL > libmysqlclient-dev mysql-devel.x86_64 : Files for development of MySQL applications > libxml2-dev libxml2-devel.x86_64 : Libraries, includes, etc. to develop XML and HTML applications > libxslt1-dev libxslt-python.x86_64 : Python bindings for the libxslt library
Flags: needinfo?(mcooper) → needinfo?(djohnson)
Looks good to me! :)
sweet. packages installed on the last puppet run :) Notice: /Stage[main]/Webapp::Admin::Support/Package[openssl-devel]/ensure: created Notice: /Stage[main]/Webapp::Admin::Support/Package[python-devel]/ensure: created Notice: /Stage[main]/Webapp::Admin::Support/Package[libxslt-python]/ensure: created Notice: /Stage[main]/Webapp::Admin::Support/Package[mysql-devel]/ensure: created Notice: /Stage[main]/Webapp::Admin::Support/Package[libxml2-devel]/ensure: created
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Sorry cturra, it looks like one of the packages wasn't quite right. libxslt-python.x86_64 is Python bindings for libxslt, which would be the python package we are trying to build. What we need is development headers for libxslt. Maybe something like libxslt-devel ? I'm not sure what the right package is, but I can say it's not libxslt-python. We should have caught this when you asked about it. These packages did help, most of the pytohn dependencies installed without error in my test environment (in my homedir), and it's just failing on the python package that requires libxslt.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
i have swapped libxslt-python with libxslt-devel as requested. let me know how you make out? Notice: /Stage[main]/Webapp::Admin::Support/Package[libxslt-devel]/ensure: created
Status: REOPENED → RESOLVED
Last Resolved: 4 years ago → 4 years ago
Resolution: --- → FIXED
cturra: thanks, this fixed the problem in my small scale test. To be honest, I'm surprised we didn't have any follow up packages we needed. Hopefully it works well with the full site too. dean: go ahead and deploy again with the deploy script that installs everything and doesn't use system packages. Remember to deploy twice! If it works, you should mark this bug as VERIFIED/FIXED.
Works great and the celery queue is now clearing itself out! Thanks again cturra!
Status: RESOLVED → VERIFIED
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.