Closed Bug 830496 Opened 11 years ago Closed 11 years ago

Install PyOpenSSL on SUMO servers

Categories

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Product:
Infrastructure & Operations Graveyard
Component:
WebOps: Other
Platform:
All
Other
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: rrosario, Assigned: cturra)

References

Details

(Whiteboard: [triaged 20130118][push interrupt]) 

We are about to start using Google APIs Client Library for python and the oauth2client library it comes with requires PyOpenSSL (https://launchpad.net/pyopenssl) for server to server auth (Service Account API client). This is a compiled dependency so we can't just add it to vendor like most of our libraries.

This would need to be installed on the following servers for -dev, -stage and -prod:
* Servers where cronjobs run (admin node?)
* Celery servers
* Web app servers

Thanks!
Blocks: 825610
:r1cky - as requested, i have pushed PyOpenSSL to all the sumo web/celery nodes.
Assignee: server-ops-webops → cturra
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Whiteboard: [triaged 20130118][push interrupt]
(In reply to Chris Turra [:cturra] from comment #1)
> :r1cky - as requested, i have pushed PyOpenSSL to all the sumo web/celery
> nodes.

YAY! Thanks! I'll verify once I land my code :-).
As I was discussing over in bug 832416, something isn't right with the PyOpenSSL that was installed.

The error I am getting is:

Traceback (most recent call last):
  File "manage.py", line 49, in <module>
    execute_manager(settings)
  File "/data/support-dev/www/support-dev.allizom.org/kitsune/vendor/src/django/django/core/management/__init__.py", line 459, in execute_manager
    utility.execute()
  File "/data/support-dev/www/support-dev.allizom.org/kitsune/vendor/src/django/django/core/management/__init__.py", line 382, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/data/support-dev/www/support-dev.allizom.org/kitsune/vendor/src/django/django/core/management/base.py", line 196, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/data/support-dev/www/support-dev.allizom.org/kitsune/vendor/src/django/django/core/management/base.py", line 232, in execute
    output = self.handle(*args, **options)
  File "/data/support-dev/www/support-dev.allizom.org/kitsune/vendor/src/django-cronjobs/cronjobs/management/commands/cron.py", line 38, in handle
    registered[script](*args)
  File "/data/support-dev/www/support-dev.allizom.org/kitsune/apps/kpi/cron.py", line 85, in update_l10n_metric
    locale_visits = googleanalytics.visitors_by_locale(start, end)
  File "/data/support-dev/www/support-dev.allizom.org/kitsune/apps/sumo/googleanalytics.py", line 56, in visitors_by_locale
    request = _build_request()
  File "/data/support-dev/www/support-dev.allizom.org/kitsune/apps/sumo/googleanalytics.py", line 21, in _build_request
    service = build('analytics', 'v3', request)
  File "/data/support-dev/www/support-dev.allizom.org/kitsune/vendor/packages/google-api-python-client/oauth2client/util.py", line 120, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/data/support-dev/www/support-dev.allizom.org/kitsune/vendor/packages/google-api-python-client/apiclient/discovery.py", line 193, in build
    resp, content = http.request(requested_url)
  File "/data/support-dev/www/support-dev.allizom.org/kitsune/vendor/packages/google-api-python-client/oauth2client/util.py", line 120, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/data/support-dev/www/support-dev.allizom.org/kitsune/vendor/packages/google-api-python-client/oauth2client/client.py", line 405, in new_request
    self._refresh(request_orig)
  File "/data/support-dev/www/support-dev.allizom.org/kitsune/vendor/packages/google-api-python-client/oauth2client/client.py", line 573, in _refresh
    self._do_refresh_request(http_request)
  File "/data/support-dev/www/support-dev.allizom.org/kitsune/vendor/packages/google-api-python-client/oauth2client/client.py", line 597, in _do_refresh_request
    body = self._generate_refresh_request_body()
  File "/data/support-dev/www/support-dev.allizom.org/kitsune/vendor/packages/google-api-python-client/oauth2client/client.py", line 729, in _generate_refresh_request_body
    assertion = self._generate_assertion()
  File "/data/support-dev/www/support-dev.allizom.org/kitsune/vendor/packages/google-api-python-client/oauth2client/client.py", line 835, in _generate_assertion
    Signer.from_string(private_key, self.private_key_password), payload)
  File "/data/support-dev/www/support-dev.allizom.org/kitsune/vendor/packages/google-api-python-client/oauth2client/crypt.py", line 163, in make_signed_jwt
    signature = signer.sign(signing_input)
  File "/data/support-dev/www/support-dev.allizom.org/kitsune/vendor/packages/google-api-python-client/oauth2client/crypt.py", line 108, in sign
    return crypto.sign(self._key, message, 'sha256')
AttributeError: 'module' object has no attribute 'sign'


This is in my local shell:

    In [1]: from OpenSSL import crypto
    In [2]: crypto.sign
    Out[2]: <function OpenSSL.crypto.sign>


I suspect if you run that on the machines, you will get the AttributeError I am seeing. Which is why I think we have the wrong version of PyOpenSSL installed.


Also, to check the version you can do:

    In [5]: import OpenSSL
    In [6]: OpenSSL.__version__
    Out[6]: '0.13'
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
:r1cky - this all makes more sense now. pyOpenSSL 0.10-2 is installed on these nodes and i am looking into the best path to 0.13 for this. due to our security requirements, we cannot use pip/gem installs on these serves yet, so we will likely have to build a custom rpm to resolve this. at first glace, that might be easier said than done because it's not packing just a binary. additionally, it has a number of dependencies we're going to have to ensure are not messed up with this. 

i will report back when i have a better action plan in place.
(In reply to Chris Turra [:cturra] from comment #4)
> i will report back when i have a better action plan in place.

OK, I figured this would take time and a custom RPM was needed. I was surprised it was so quick initially :). Thanks.
:r1cky - i have sorted through some of the requirements with our systems team and have created a bug for our security assurance group to get approval for this new pyOpenSSL package. once that is complete, we can do some testing to ensure this doesn't break any RHEL core dependencies (since pyOpenSSL is a RHEL core package).
(In reply to Chris Turra [:cturra] from comment #6)
> :r1cky - i have sorted through some of the requirements with our systems
> team and have created a bug for our security assurance group to get approval
> for this new pyOpenSSL package. once that is complete, we can do some
> testing to ensure this doesn't break any RHEL core dependencies (since
> pyOpenSSL is a RHEL core package).

I can't see the blocking bug but I do see it is fixed. Does that mean we are good to go?
:r1cky - you're correct, we have been given the green light from opsec. i will try to get this packaged and tested for you asap. but it will take a bit of testing to ensure that pyOpenSSL 0.13 does not cause other dependencies to get grumpy.

i can't commit to an eta on this, but promise it's on my short list of priority items :)
We tried --provides --requires to get pyOpenSSL 0.13 to replace the 0.10 rpm, but it never could do so.

If manually installed, RHN worked fine.

An app-specific prefix would probably be best, unless there's a master-level yum/rpm person online to help.
alright, i have made some progress here. to install this side-by-side the core pyOpenSSL (0.10) package, i had to create a new pyOpenSSLmoz rpm. i have it installed and running on support-dev:

here it is in action in a python shell:

  >>> import OpenSSLmoz
  >>> OpenSSLmoz.__version__
  '0.13'

  >>> from OpenSSLmoz import crypto
  >>> crypto.sign
  <built-in function sign>


*the only downside i can see to this approach is if the library you're using calls OpenSSL, it will still use the existing 0.10 version. to get the 0.13 fixes you were looking for, your application is going to have to import/use OpenSSLmoz instead of OpenSSL.
(In reply to Chris Turra [:cturra] from comment #10)
> *the only downside i can see to this approach is if the library you're using
> calls OpenSSL, it will still use the existing 0.10 version. to get the 0.13
> fixes you were looking for, your application is going to have to import/use
> OpenSSLmoz instead of OpenSSL.

Well, that... is kind of a problem. The library that requires OpenSSL isn't our code, it's Google's:

http://code.google.com/p/google-api-python-client/source/browse/oauth2client/crypt.py

Since we can't change the library code without maintaining our own fork forever, I guess we need to do something like PyMySQL's install_as_MySQLdb path hack?
(In reply to James Socol [:jsocol, :james] from comment #11)
> Since we can't change the library code without maintaining our own fork
> forever, I guess we need to do something like PyMySQL's install_as_MySQLdb
> path hack?

Which, for the record, is fairly straightforward, though I've had issues with it actually working in practice.

http://code.google.com/p/pymysql/source/browse/trunk/pymysql/__init__.py#110
Does the python-based webserver component or whatever have PYTHON_PATH= or equivalent?

There's something to be said for prefixing /opt/mozilla-python to the search path of whatever it is.
(In reply to Richard Soderberg [:atoll] from comment #13)
> Does the python-based webserver component or whatever have PYTHON_PATH= or
> equivalent?

There's a WSGIPythonPath setting, yes.

http://code.google.com/p/modwsgi/wiki/ConfigurationDirectives#WSGIPythonPath

That's better than changing the name of the Python module, if we can do it.
:atoll/:jsocol - i agree and was an approach i had been trying with limited success. i will have to go back to the rpm building drawing board to see if i can get this package to build/deploy to another directory.
good news! i have managed to build the 0.13 rpm to alive along side the 0.13 package. it's installed to a different path, so you will need to set the python path accordingly. note, you will want to add it to the beginning of your path, not the end because the first package that is found will be used.

>>> import sys
>>> sys.path.insert(0,"/usr/local/lib64/python2.6/site-packages")
>>> print sys.path
['/usr/local/lib64/python2.6/site-packages', '', '/usr/lib64/python26.zip', '/usr/lib64/python2.6', '/usr/lib64/python2.6/plat-linux2', '/usr/lib64/python2.6/lib-tk', '/usr/lib64/python2.6/lib-old', '/usr/lib64/python2.6/lib-dynload', '/usr/lib64/python2.6/site-packages', '/usr/lib64/python2.6/site-packages/PIL', '/usr/lib64/python2.6/site-packages/gtk-2.0', '/usr/lib/python2.6/site-packages', '/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg-info']
>>> import OpenSSL
0.13
>>> OpenSSL.__version__
'0.13'
(In reply to Chris Turra [:cturra] from comment #16)
> note, you will want to add it to the beginning of
> your path, not the end because the first package that is found will be used.

I assume this is something that is done in some config (wsgi?) on your end?
Also yay! \o/ :-D
(In reply to Ricky Rosario [:rrosario, :r1cky] from comment #18)
> Also yay! \o/ :-D

Agreed, yay! You rock, cturra :D

(In reply to Chris Turra [:cturra] from comment #16)
> good news! i have managed to build the 0.13 rpm to alive along side the 0.13
> package. it's installed to a different path, so you will need to set the
> python path accordingly.

We do some of this in manage.py already, but it's all relative paths to get the vendor library set up.

We can do this in manage.py (does it matter if you add a path that doesn't exist on most systems?) or we might be able to do it with WSGIPythonPath. WSGIPythonPath is probably a better option because it lets us manage the path juggling with puppet, alongside the RPM (so if the patch ever changes, we don't need to coordinate with code changes).
i have un-hard-hatted sumo-dev so we can begin to play with this and added WSGIPythonPath to the apache config as suggested by :jsocol. 

+# bug 830496 -> added to support pyOpenSSLmoz pkg
+WSGIPythonPath /usr/local/lib64/python2.6/site-packages:/usr/lib64/python2.6/site-packages
(In reply to Chris Turra [:cturra] from comment #20)
> i have un-hard-hatted sumo-dev so we can begin to play with this and added
> WSGIPythonPath to the apache config as suggested by :jsocol. 

Is it hard to get this on -stage as well? The thing is... I can get code onto -stage through chief from a branch. To get code onto -dev I have to push to master and then potentially roll it back and I already did that once. And we're talking about 9 commits each time.

... or is it easy to get -dev pointing to a different branch?

... or?
:r1cky - it shouldn't be *and* will give me a chance to test out how to manage this from start -> finish with puppet. i will report back when stage is ready.
pyOpenSSL 0.13 has now been pushed (with ONLY puppet) to stage. time for some testing :)
(In reply to Chris Turra [:cturra] from comment #23)
> pyOpenSSL 0.13 has now been pushed (with ONLY puppet) to stage. time for
> some testing :)

woot. I'll report back.
hmm, no luck yet. I just realized the python path needs to be correct for cron and celery as well. And on whatever box is running cron and celery:

Traceback (most recent call last):
  File "manage.py", line 49, in <module>
    execute_manager(settings)
  File "/data/support-stage/www/support.allizom.org/kitsune/vendor/src/django/django/core/management/__init__.py", line 459, in execute_manager
    utility.execute()
  File "/data/support-stage/www/support.allizom.org/kitsune/vendor/src/django/django/core/management/__init__.py", line 382, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/data/support-stage/www/support.allizom.org/kitsune/vendor/src/django/django/core/management/base.py", line 196, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/data/support-stage/www/support.allizom.org/kitsune/vendor/src/django/django/core/management/base.py", line 232, in execute
    output = self.handle(*args, **options)
  File "/data/support-stage/www/support.allizom.org/kitsune/vendor/src/django-cronjobs/cronjobs/management/commands/cron.py", line 38, in handle
    registered[script](*args)
  File "/data/support-stage/www/support.allizom.org/kitsune/apps/kpi/cron.py", line 39, in update_visitors_metric
    visitors = googleanalytics.visitors(start, end)
  File "/data/support-stage/www/support.allizom.org/kitsune/apps/sumo/googleanalytics.py", line 34, in visitors
    request = _build_request()
  File "/data/support-stage/www/support.allizom.org/kitsune/apps/sumo/googleanalytics.py", line 21, in _build_request
    service = build('analytics', 'v3', request)
  File "/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/google-api-python-client/oauth2client/util.py", line 120, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/google-api-python-client/apiclient/discovery.py", line 193, in build
    resp, content = http.request(requested_url)
  File "/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/google-api-python-client/oauth2client/util.py", line 120, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/google-api-python-client/oauth2client/client.py", line 405, in new_request
    self._refresh(request_orig)
  File "/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/google-api-python-client/oauth2client/client.py", line 573, in _refresh
    self._do_refresh_request(http_request)
  File "/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/google-api-python-client/oauth2client/client.py", line 597, in _do_refresh_request
    body = self._generate_refresh_request_body()
  File "/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/google-api-python-client/oauth2client/client.py", line 729, in _generate_refresh_request_body
    assertion = self._generate_assertion()
  File "/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/google-api-python-client/oauth2client/client.py", line 835, in _generate_assertion
    Signer.from_string(private_key, self.private_key_password), payload)
  File "/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/google-api-python-client/oauth2client/crypt.py", line 163, in make_signed_jwt
    signature = signer.sign(signing_input)
  File "/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/google-api-python-client/oauth2client/crypt.py", line 108, in sign
    return crypto.sign(self._key, message, 'sha256')
AttributeError: 'module' object has no attribute 'sign'
:cturra cron runs from the admin box, correct? Does that have the correct pyOpenSSL too? Sorry for not realizing this before.
:r1cky - the celery nodes automatically got this for dev and stage. i also pushed this to the supportadm puppet module for the crons.
:cturra - cool. the next cron job is going to run in two minutes. *fingers crossed*
:cturra - still no luck. How do we fix the python path for cron jobs?
:r1cky - i suspect it's going to need to be baked into manage.py. 

i know it's overly simplified, but my example does accomplish this within a python path:

>>> import sys
>>> sys.path.insert(0,"/usr/local/lib64/python2.6/site-packages")
>>> print sys.path
['/usr/local/lib64/python2.6/site-packages', '', '/usr/lib64/python26.zip', '/usr/lib64/python2.6', '/usr/lib64/python2.6/plat-linux2', '/usr/lib64/python2.6/lib-tk', '/usr/lib64/python2.6/lib-old', '/usr/lib64/python2.6/lib-dynload', '/usr/lib64/python2.6/site-packages', '/usr/lib64/python2.6/site-packages/PIL', '/usr/lib64/python2.6/site-packages/gtk-2.0', '/usr/lib/python2.6/site-packages', '/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg-info']
(In reply to Chris Turra [:cturra] from comment #30)
> :r1cky - i suspect it's going to need to be baked into manage.py. 

cron supports adding environment variables prior to a cron entry, such as PYTHON_PATH=xyz, in case that's preferred. Also there's "env PYTHON_PATH=xyz /usr/bin/python ...".
i just tested and that should also be an option.

:r1cky - you actually have full control over these crons (they're auto generated from scripts/crontab/crontab.tpl) in the kitsune project:

  https://github.com/mozilla/kitsune/tree/master/scripts/crontab


[root@supportadm.private.phx1 ~]# export PYTHONPATH=/usr/local/lib64/python2.6/site-packages/
[root@supportadm.private.phx1 ~]# python
Python 2.6.6 (r266:84292, Aug 28 2012, 10:55:56) 
[GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import OpenSSL
>>> OpenSSL.__version__
'0.13'
>>> quit()
OK, I haven't had luck doing it in manage.py. I try to print out the version after changing the path and I get:

Traceback (most recent call last):
  File "manage.py", line 27, in <module>
    print OpenSSL.__version__
AttributeError: 'module' object has no attribute '__version__'


Here is the result of `print sys.path`:
['/usr/local/lib64/python2.6/site-packages/', '/data/support-stage/www/support.allizom.org/kitsune/apps', '/data/support-stage/www/support.allizom.org/kitsune/lib', '/data/support-stage/www/support.allizom.org/kitsune/vendor', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/pytz', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/coverage', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/sqlparse', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/GitPython/lib', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/Werkzeug', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/Babel', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/python-dateutil', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/pylint', '/data/support-stage/www/support.allizom.org/kits
 une/vendor/packages/pyflakes', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/amqplib', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/django-taggit', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/anyjson', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/carrot', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/pyquery', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/setuptools', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/pep8', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/logilab-astng', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/importlib', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/python-memcached', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/selenium', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packag
 es/mock', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/html5lib/src', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/logilab-common', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/billiard', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/pyparsing', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/sqlalchemy', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/translate-toolkit', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/nose', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/httplib2/python2', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/pyes', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/mimeparse', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/django-cronjobs', '/data/s
 upport-stage/www/support.allizom.org/kitsune/vendor/src/django-cache-machine', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/django-nose', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/django-authority', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/django-picklefield/src', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/django-timezones', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/commonware', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/jingo', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/cache-panel', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/jingo-minify', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/check', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/django-mozilla-product-details', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/test-utils', '/data/support-stage/www/
 support.allizom.org/kitsune/vendor/src/schematic', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/py-wikimarkup', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/django-debug-toolbar', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/django-multidb-router', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/bleach', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/tower', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/django-extensions', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/django', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/tweepy', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/django-csp', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/django-waffle', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/sphinxapi', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/django-mobi
 lity', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/django-qunit', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/django-adminplus', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/celery', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/kombu', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/django-celery', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/django-tidings', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/jstestnetlib', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/pystatsd', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/django-session-csrf', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/redis-py', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/django-ratelimit', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/django-recaptcha', '/data/support-stage/www/support.
 allizom.org/kitsune/vendor/src/python-recaptcha', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/oedipus', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/elasticutils', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/django-tastypie', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/zendesk', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/ordereddict', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/django-eadred', '/data/support-stage/www/support.allizom.org/kitsune/vendor/src/raven-python', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/google-api-python-client', '/data/support-stage/www/support.allizom.org/kitsune/vendor/packages/python-gflags', '/data/support-stage/www/support.allizom.org/kitsune', '/usr/lib/python2.6/site-packages/pip-1.0.2-py2.6.egg', '/usr/lib64/python26.zip', '/usr/lib64/python2.6', '/usr/lib64/python2.6/plat-linux2', '/usr/lib
 64/python2.6/lib-tk', '/usr/lib64/python2.6/lib-old', '/usr/lib64/python2.6/lib-dynload', '/usr/lib64/python2.6/site-packages', '/usr/lib64/python2.6/site-packages/PIL', '/usr/lib64/python2.6/site-packages/gtk-2.0', '/usr/lib/python2.6/site-packages', '/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg-info']
OK, something weird is happening as soon as I make put that in front of the path.

I added this to manage.py:
import OpenSSL
print OpenSSL.__version__

Then the cron command runs:
Cron <apache@supportadm> cd /data/support-stage/www/support.allizom.org/kitsune; PYTHONPATH=/usr/local/lib64/python2.6/site-packages/ /usr/bin/python2.6 manage.py cron collect_tweets

And I get the traceback:
Traceback (most recent call last):
  File "manage.py", line 27, in <module>
    print OpenSSL.__version__
AttributeError: 'module' object has no attribute '__version__'


It is importing OpenSSL fine but it seems screwed up. If I run the cron command without the PYTHONPATH I get:
Email subject:
Cron <apache@supportadm> cd /data/support-stage/www/support.allizom.org/kitsune;  /usr/bin/python2.6 manage.py cron collect_tweets
Email body:
0.10


Wat? Weird.
Oh, I just realized I can log into stage :). I get the same result playing there:

[rrosario@support1.stage.webapp.phx1 ~]$ PYTHONPATH=/usr/local/lib64/python2.6/site-packages/ python
Python 2.6.6 (r266:84292, Aug 28 2012, 10:55:56) 
[GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import OpenSSL
>>> OpenSSL.__version__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute '__version__'
Is it a permissions issue? I notice you tried as root, that is cheating! :-D

[rrosario@support1.stage.webapp.phx1 OpenSSL]$ pwd
/usr/local/lib64/python2.6/site-packages/OpenSSL
[rrosario@support1.stage.webapp.phx1 OpenSSL]$ ls -al
total 656
drwxr-xr-x 3 root root   4096 Feb  6 14:07 .
drwxr-xr-x 3 root root   4096 Feb  6 13:37 ..
-rwxr-x--- 1 root root 363155 Feb  6 10:54 crypto.so
-rw-r----- 1 root root    965 Feb  6 10:54 __init__.py
-rw-r----- 1 root root    965 Feb  6 14:07 __init__.pyc
-rwxr-x--- 1 root root  36378 Feb  6 10:54 rand.so
-rwxr-x--- 1 root root 221202 Feb  6 10:54 SSL.so
drwxr-xr-x 2 root root   4096 Feb  6 13:37 test
-rw-r----- 1 root root   1010 Feb  6 10:54 tsafe.py
-rw-r----- 1 root root   1786 Feb  6 10:54 tsafe.pyc
-rw-r----- 1 root root    176 Feb  6 10:54 version.py
-rw-r----- 1 root root    259 Feb  6 14:07 version.pyc
:r1cky - i think you're onto something there. i am going to have to play with the package some more to sort out those permission. 

two steps forward, one step back... but we're making /some/ progress :)
sorted! there was actually two issues here: (1) the build/ source had the incorrect permissions and (2) umask seemed not be set correctly on the server i was creating the rpm from.

to resolve (1) i updated the permission to match those of pyOpenSSL 0.10 and (2) reset the umask manually before rebuilding the rpm.

i have rolled this out to the supportadm node and the tests turned out as expected (NOT as root ;) i will get this onto each of the dev/stage nodes also.

[cturra@supportadm.private.phx1 site-packages]$ PYTHONPATH=/usr/local/lib64/python2.6/site-packages/ python
Python 2.6.6 (r266:84292, Aug 28 2012, 10:55:56) 
[GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import OpenSSL
>>> OpenSSL.__version__
'0.13'
:cturra - It works!!! \o/

From my end, we are ready for this and Bug 832416 to be rolled out to -prod.
:r1cky - i have pushed this out to prod, which should now have everything needed to test this. i am going to mark this bug as r/fixed, but please reopen if you see anything odd.

[cturra@support5.webapp.phx1 ~]$ PYTHONPATH=/usr/local/lib64/python2.6/site-packages/ python
Python 2.6.6 (r266:84292, Aug 28 2012, 10:55:56) 
[GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import OpenSSL
>>> OpenSSL.__version__
'0.13'
>>> quit()
Status: REOPENED → RESOLVED
Closed: 11 years ago → 11 years ago
Resolution: --- → FIXED
:cturra yay! THANK YOU! As soon as Bug 832416 is fixed, I'll land this and deploy.
Everything seems to be running great. Thanks :cturra!!
Status: RESOLVED → VERIFIED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.