1065152 - Switch celery scripts from using vendor for libraries to virtualenv

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Community Platform, task)

Description

[Dean Johnson [:deanj]]

+++ This bug was initially created as a clone of Bug #1064457 +++

Blocking: bug 1064457

We were able to track down the issues in bug 1064457 to our deployment script that runs the following:

ctx.remote(settings.REMOTE_UPDATE_SCRIPT)
ctx.remote('/sbin/service %s restart' % settings.CELERY_SERVICE)

We changed our deployment strategy from using vendor libraries to a virtual environment and need to update the CELERY_SERVICE and/or the REMOTE_UPDATE_SCRIPT to reflect those changes. Basically, we want to source the virtualenv (which is located in the TLD of kitsune) before we run any celery tasks, so we can ensure that the tasks all run properly. 

These changes need to be made on the dev server.

Comment 1

[Dean Johnson [:deanj]]

Celery tasks seem to be queuing up on dev.

According to the logs at http://supportadm.private.phx1.mozilla.com/chief/support.dev/logs/ad03ce3b05c97d11e9125e2f8a88a3bb4e9d0deb.1410556899 celery doesn't seem to be starting correctly, which I believe has to do with it not activating the virtualenv, and instead trying to use vendor packages.

Comment 2

[Chris Turra [:cturra]]

(In reply to Dean Johnson [:deanj] from comment #1)
> Celery tasks seem to be queuing up on dev.
> 
> According to the logs at
> http://supportadm.private.phx1.mozilla.com/chief/support.dev/logs/
> ad03ce3b05c97d11e9125e2f8a88a3bb4e9d0deb.1410556899 celery doesn't seem to
> be starting correctly, which I believe has to do with it not activating the
> virtualenv, and instead trying to use vendor packages.

looks like you're exactly right. from the logs i see:

Traceback (most recent call last):
  File "/data/www/support-dev.allizom.org/kitsune/manage.py", line 9, in <module>
    from django.conf import settings
ImportError: No module named django.conf


i have read over your bug a couple times, but it's not explicitly clear to me what you need done on the celery server? can you please provide more details? feel free to ping me on irc if you'd like to work through it together.

Comment 3

[Dean Johnson [:deanj]]

The problem ended up being that when python looked for virtualenv/lib64, it was symlinked to an incorrect absolute path for virtualenv/lib. This happens because we rsync our virtualenv around, and the paths get a little confused. The paths on the admin node where the virtualenv is created are a bit different from where celery is started. We're not quite sure why this doesn't affect the web-heads at the moment.

The fix is in commit 85fc1063926e658261af7b32e71aa355d8c81c9a.

We still have more errors relating to celery accessing packages it needs, so we're going to leave the bug open for now.

Comment 4

[Dean Johnson [:deanj]]

We're now running into issues using virtualenv/bin/activate_this.py, which seem to be caused by the system package abrt_addon_python. 

The issue seems to come from version 2.x for abrt_addon_python on the webheads and celery node, because version 1.x is installed on the admin node and using virtualenv/bin/activate_this.py in our deploy script runs without issues.

Here's a reference to a similar issue on redhat's bug tracker that may be of help: https://bugzilla.redhat.com/show_bug.cgi?id=907449.

Would it be possible to downgrade abrt_addon_python on the webheads and celery node to match the version on the admin node?

Comment 5

[Chris Turra [:cturra]]

while downgrading would technically be possible, abrt is used for a lot of other things on this host and as a result i expect would be a rather large effort. :mythmon has indicated that addressing bug 1067586 should get us around needing this work done. i'd like to go that route if it's indeed an option.

Comment 6

[Dean Johnson [:deanj]]

Because of the fixes in #1067586, we no longer need the downgrade. Fixed.