966410 - [mozillians-stage] Some celery tasks not being processed

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Engagement, task)

Description

[Andrei Hajdukewycz [:sancus]]

We have a couple of tasks that send emails, and they seem to be working on dev, but not on stage. Dev and stage are using the same code, and the periodic tasks are configured the same way in the database.

Namely https://github.com/mozilla/mozillians/blob/master/mozillians/groups/tasks.py#L28 and the one below that.

Maybe check the celery log and see if something is blowing up?

Comment 1

[Andrei Hajdukewycz [:sancus]]

This is blocking a launch of this feature for us, because we have no way of knowing if prod will be broken in the same way as stage, despite dev seeming to be okay.

Comment 2

[Chris Turra [:cturra]]

i didn't see any obvious errors on the stage celery node, but i did restart the celery process. after doing this, i noticed there are some tracebacks being returned in the celery.log that look like the following. thoughts?

[2014-02-04 10:36:12,837: ERROR/MainProcess] Task mozillians.groups.tasks.send_pending_membership_emails[16a8c7c0-3291-463e-85f3-c2bc74501426] raised exception: ImportError('No module named urls',)
Traceback (most recent call last):
  File "/data/www/mozillians.allizom.org/mozillians/vendor-local/lib/python/celery/execute/trace.py", line 181, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/data/www/mozillians.allizom.org/mozillians/mozillians/groups/tasks.py", line 57, in send_pending_membership_emails
    'count': count,
  File "/data/www/mozillians.allizom.org/mozillians/vendor/lib/python/django/template/loader.py", line 171, in render_to_string
    return t.render(Context(dictionary))
  File "/data/www/mozillians.allizom.org/mozillians/vendor/src/jingo/jingo/__init__.py", line 189, in render
    return super(Template, self).render(context_dict)
  File "/usr/lib64/python2.6/site-packages/jinja2/environment.py", line 891, in render
    return self.environment.handle_exception(exc_info, True)
  File "/data/www/mozillians.allizom.org/mozillians/mozillians/templates/groups/email/memberships_pending.txt", line 8, in top-level template code
    {{ group.get_absolute_url() }}
  File "/data/www/mozillians.allizom.org/mozillians/vendor/lib/python/django/utils/functional.py", line 11, in _curried
    return _curried_func(*(args+moreargs), **dict(kwargs, **morekwargs))
  File "/data/www/mozillians.allizom.org/mozillians/vendor/lib/python/django/db/models/base.py", line 883, in get_absolute_url
    return settings.ABSOLUTE_URL_OVERRIDES.get('%s.%s' % (opts.app_label, opts.module_name), func)(self, *args, **kwargs)
  File "/data/www/mozillians.allizom.org/mozillians/mozillians/groups/models.py", line 209, in get_absolute_url
    return absolutify(reverse('groups:show_group', args=[self.url]))
  File "/data/www/mozillians.allizom.org/mozillians/vendor/src/funfactory/funfactory/urlresolvers.py", line 30, in reverse
    url = django_reverse(viewname, urlconf, args, kwargs, prefix)
  File "/data/www/mozillians.allizom.org/mozillians/vendor/lib/python/django/core/urlresolvers.py", line 447, in reverse
    app_list = resolver.app_dict[ns]
  File "/data/www/mozillians.allizom.org/mozillians/vendor/lib/python/django/core/urlresolvers.py", line 290, in app_dict
    self._populate()
  File "/data/www/mozillians.allizom.org/mozillians/vendor/lib/python/django/core/urlresolvers.py", line 242, in _populate
    for pattern in reversed(self.url_patterns):
  File "/data/www/mozillians.allizom.org/mozillians/vendor/lib/python/django/core/urlresolvers.py", line 328, in url_patterns
    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
  File "/data/www/mozillians.allizom.org/mozillians/vendor/lib/python/django/core/urlresolvers.py", line 323, in urlconf_module
    self._urlconf_module = import_module(self.urlconf_name)
  File "/data/www/mozillians.allizom.org/mozillians/vendor/lib/python/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/data/www/mozillians.allizom.org/mozillians/mozillians/urls.py", line 57, in <module>
    url(r'^browserid/', include('django_browserid.urls')),
  File "/data/www/mozillians.allizom.org/mozillians/vendor/lib/python/django/conf/urls/__init__.py", line 24, in include
    urlconf_module = import_module(urlconf_module)
  File "/data/www/mozillians.allizom.org/mozillians/vendor/lib/python/django/utils/importlib.py", line 35, in import_module
    __import__(name)
ImportError: No module named urls
[2014-02-04 10:36:40,013: INFO/Beat] Scheduler: Sending due task groups.tasks.remove_empty_groups


[2014-02-04 10:36:40,022: ERROR/MainProcess] Received unregistered task of type u'groups.tasks.remove_empty_groups'.
The message has been ignored and discarded.

Did you remember to import the module containing this task?
Or maybe you are using relative imports?
Please see http://bit.ly/gLye1c for more information.

The full contents of the message body was:
{'retries': 0, 'task': u'groups.tasks.remove_empty_groups', 'utc': False, 'args': [], 'expires': None, 'eta': None, 'kwargs': {}, 'id': 'bda4069a-8b88-47f7-a808-d34dd89d6608'}
Traceback (most recent call last):
  File "/data/www/mozillians.allizom.org/mozillians/vendor-local/lib/python/celery/worker/consumer.py", line 442, in receive_message
    self.strategies[name](message, body, message.ack_log_error)
KeyError: u'groups.tasks.remove_empty_groups'


[2014-02-04 10:37:13,056: ERROR/MainProcess] Task mozillians.groups.tasks.send_pending_membership_emails[3db0c995-f3ae-4b49-a380-b44392e3742d] raised exception: ImportError('No module named urls',)
Traceback (most recent call last):
  File "/data/www/mozillians.allizom.org/mozillians/vendor-local/lib/python/celery/execute/trace.py", line 181, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/data/www/mozillians.allizom.org/mozillians/mozillians/groups/tasks.py", line 57, in send_pending_membership_emails
    'count': count,
  File "/data/www/mozillians.allizom.org/mozillians/vendor/lib/python/django/template/loader.py", line 171, in render_to_string
    return t.render(Context(dictionary))
  File "/data/www/mozillians.allizom.org/mozillians/vendor/src/jingo/jingo/__init__.py", line 189, in render
    return super(Template, self).render(context_dict)
  File "/usr/lib64/python2.6/site-packages/jinja2/environment.py", line 891, in render
    return self.environment.handle_exception(exc_info, True)
  File "/data/www/mozillians.allizom.org/mozillians/mozillians/templates/groups/email/memberships_pending.txt", line 8, in top-level template code
    {{ group.get_absolute_url() }}
  File "/data/www/mozillians.allizom.org/mozillians/vendor/lib/python/django/utils/functional.py", line 11, in _curried
    return _curried_func(*(args+moreargs), **dict(kwargs, **morekwargs))
  File "/data/www/mozillians.allizom.org/mozillians/vendor/lib/python/django/db/models/base.py", line 883, in get_absolute_url
    return settings.ABSOLUTE_URL_OVERRIDES.get('%s.%s' % (opts.app_label, opts.module_name), func)(self, *args, **kwargs)
  File "/data/www/mozillians.allizom.org/mozillians/mozillians/groups/models.py", line 209, in get_absolute_url
    return absolutify(reverse('groups:show_group', args=[self.url]))
  File "/data/www/mozillians.allizom.org/mozillians/vendor/src/funfactory/funfactory/urlresolvers.py", line 30, in reverse
    url = django_reverse(viewname, urlconf, args, kwargs, prefix)
  File "/data/www/mozillians.allizom.org/mozillians/vendor/lib/python/django/core/urlresolvers.py", line 447, in reverse
    app_list = resolver.app_dict[ns]
  File "/data/www/mozillians.allizom.org/mozillians/vendor/lib/python/django/core/urlresolvers.py", line 290, in app_dict
    self._populate()
  File "/data/www/mozillians.allizom.org/mozillians/vendor/lib/python/django/core/urlresolvers.py", line 242, in _populate
    for pattern in reversed(self.url_patterns):
  File "/data/www/mozillians.allizom.org/mozillians/vendor/lib/python/django/core/urlresolvers.py", line 328, in url_patterns
    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
  File "/data/www/mozillians.allizom.org/mozillians/vendor/lib/python/django/core/urlresolvers.py", line 323, in urlconf_module
    self._urlconf_module = import_module(self.urlconf_name)
  File "/data/www/mozillians.allizom.org/mozillians/vendor/lib/python/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/data/www/mozillians.allizom.org/mozillians/mozillians/urls.py", line 57, in <module>
    url(r'^browserid/', include('django_browserid.urls')),
  File "/data/www/mozillians.allizom.org/mozillians/vendor/lib/python/django/conf/urls/__init__.py", line 24, in include
    urlconf_module = import_module(urlconf_module)
  File "/data/www/mozillians.allizom.org/mozillians/vendor/lib/python/django/utils/importlib.py", line 35, in import_module
    __import__(name)
ImportError: No module named urls

Comment 3

[Andrei Hajdukewycz [:sancus]]

can you do a restart on dev and check the celery logs there, too? I'm deeply puzzled as to why this is erroring that way on stage and yet not on dev.

Comment 4

[Chris Turra [:cturra]]

after further investigation, i found that there were some git hash errors in some of the previous chief pushes, which led me to believe we had a bad checkout of the genericadm git repo. to resolve, i did the following:

 - stopped the celery worker on generic-celery1.stage
 - removed the /data/www/mozillians.allizom.org/mozillians directory
 - recloned from the repo (/data/bin/update-www.sh mozillians.allizom.org)
 - started the celery worker again.

i confirmed with :sancus, everything seems to have returned to functioning as expected :)

Comment 5

[Andrei Hajdukewycz [:sancus]]

Thanks cturra!