804797 - [traceback] error: [Errno 111] Connection refused when trying to log in to Mozillians dev

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

When I try to log in [e-mail] (through Persona/BrowserID) on dev, I get a 500, which is this exception:

Traceback (most recent call last):

 File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/src/django/django/core/handlers/base.py", line 111, in get_response
   response = callback(request, *callback_args, **callback_kwargs)

 File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/src/django/django/views/generic/base.py", line 48, in view
   return self.dispatch(request, *args, **kwargs)

 File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/src/django/django/views/generic/base.py", line 69, in dispatch
   return handler(request, *args, **kwargs)

 File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/src/django/django/views/generic/edit.py", line 138, in post
   return self.form_valid(form)

 File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/src/django-browserid/django_browserid/views.py", line 53, in form_valid
   return self.login_success()

 File "/data/www/mozillians-dev.allizom.org/mozillians/apps/users/views.py", line 41, in login_success
   auth.login(self.request, self.user)

 File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/src/django/django/contrib/auth/__init__.py", line 76, in login
   user_logged_in.send(sender=user.__class__, request=request, user=user)

 File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/src/django/django/dispatch/dispatcher.py", line 172, in send
   response = receiver(signal=self, sender=sender, **named)

 File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/src/django/django/contrib/auth/models.py", line 26, in update_last_login
   user.save()

 File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/src/django/django/db/models/base.py", line 463, in save
   self.save_base(using=using, force_insert=force_insert, force_update=force_update)

 File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/src/django/django/db/models/base.py", line 565, in save_base
   created=(not record_exists), raw=raw, using=using)

 File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/src/django/django/dispatch/dispatcher.py", line 172, in send
   response = receiver(signal=self, sender=sender, **named)

 File "/data/www/mozillians-dev.allizom.org/mozillians/apps/users/models.py", line 282, in create_user_profile
   up.save()

 File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/src/django/django/db/models/base.py", line 463, in save
   self.save_base(using=using, force_insert=force_insert, force_update=force_update)

 File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/src/django/django/db/models/base.py", line 565, in save_base
   created=(not record_exists), raw=raw, using=using)

 File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/src/django/django/dispatch/dispatcher.py", line 172, in send
   response = receiver(signal=self, sender=sender, **named)

 File "/data/www/mozillians-dev.allizom.org/mozillians/apps/users/models.py", line 321, in update_search_index
   tasks.index_objects.delay(sender, [instance.id])

 File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/celery/task/base.py", line 348, in delay
   return self.apply_async(args, kwargs)

 File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/celery/task/base.py", line 364, in apply_async
   return apply_async(self, args, kwargs, **options)

 File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/celery/messaging.py", line 294, in _inner
   return fun(*args, **kwargs)

 File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/celery/execute/__init__.py", line 102, in apply_async
   exchange_type=exchange_type)

 File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/celery/task/base.py", line 316, in get_publisher
   routing_key=self.routing_key)

 File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/celery/messaging.py", line 46, in __init__
   consumers = get_consumer_set(self.connection)

 File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/celery/messaging.py", line 314, in get_consumer_set
   backend=cset.backend, **queue_options)

 File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/carrot/messaging.py", line 233, in __init__
   self.declare()

 File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/carrot/messaging.py", line 271, in declare
   warn_if_exists=self.warn_if_exists)

 File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/carrot/backends/pyamqplib.py", line 245, in queue_declare
   return self.channel.queue_declare(queue=queue,

 File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/carrot/backends/pyamqplib.py", line 187, in channel
   connection = self.connection.connection

 File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/carrot/connection.py", line 135, in connection
   self._connection = self._establish_connection()

 File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/carrot/connection.py", line 148, in _establish_connection
   return self.create_backend().establish_connection()

 File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/carrot/backends/pyamqplib.py", line 208, in establish_connection
   connect_timeout=conninfo.connect_timeout)

 File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/amqplib/client_0_8/connection.py", line 125, in __init__
   self.transport = create_transport(host, connect_timeout, ssl)

 File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/amqplib/client_0_8/transport.py", line 220, in create_transport
   return TCPTransport(host, connect_timeout)

 File "/data/www/mozillians-dev.allizom.org/mozillians/vendor/lib/python/amqplib/client_0_8/transport.py", line 58, in __init__
   self.sock.connect((host, port))

 File "<string>", line 1, in connect

error: [Errno 111] Connection refused

Comment 1

[Ben (:bensternthal)]

More info:

http://qa-selenium.mv.mozilla.com:8080/job/mozillians.dev/2323/HTML_Report/?

Note you must be connected to vpn mozilla-mv

Comment 2

[Stephen Donner [:stephend] Not actively reading bugmail]

/test_edit_profile_information and test_profile_creation are currently failing

[1] https://github.com/mozilla/mozillians-tests/blob/master/tests/test_profile.py

Comment 3

[Giorgos Logiotatidis [:giorgos]]

Hello IT,

can you please check connections between:

1. mozillians-dev server and celery dev server
2. celery dev server and elastic search dev server 
3. celery dev server and the dev broker server?

Thanks!

Comment 4

[Shyam Mani [:fox2mike]]

This is paging oncall, dropping severity till someone can check this from webops when they're up.

Comment 5

[Peter Radcliffe [:pir]]

Still paging, dropping to normal.

Comment 6

[Ben (:bensternthal)]

Once this is resolved can you give us some forensics on what/why this happened?

Comment 7

[Jake Maul [:jakem]]

This is a side-effect of bug 804565.

I've stopped puppet again, and cleared up some disk space.

Comment 8

[Giorgos Logiotatidis [:giorgos]]

We still get the error :/

Comment 9

[Brandon Burton [:solarce]]

This should be stable now, but may blip later today as we move to a new celery server, the current one has been running short on disk

I'll close the bug once the permanent fix is in place

Comment 10

[Brandon Burton [:solarce]]

This will bleed over into tomorrow for the permanent fix

Comment 11

[Rebecca Billings [:rbillings]]

I wasn't able to repro the error on dev 10/29 - but I'll check back to see if others have the same result.

Comment 12

[Stephen Donner [:stephend] Not actively reading bugmail]

Looks like automation is happy with the (temporary?) changes (comment 9) too: http://qa-selenium.mv.mozilla.com:8080/view/Mozillians/job/mozillians.dev/2342/HTML_Report/?

Comment 13

[Ben (:bensternthal)]

closing given comment 11,12

Comment 14

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED, still. :-)