[pulsetranslator] Listening stops after publish failure

NEW
Unassigned

Status

4 years ago
3 years ago

People

(Reporter: jgriffin, Unassigned)

Tracking

Trunk
Points:
---

Firefox Tracking Flags

(firefox43 affected)

Details

(Reporter)

Description

4 years ago
It looks like when pulsetranslator loses a connection to pulse while publishing a message, it stop listening for messages, even though the publish attempt is retried until it succeeds.

This morning pulsetranslator stopped reading from its queue after this event:

 Sep 02 09:27:40 pulsetranslator app/worker.1: Traceback (most recent call last): 
Sep 02 09:27:40 pulsetranslator app/worker.1:   File "/app/pulsetranslator/translatorqueues.py", line 25, in publish_message 
Sep 02 09:27:40 pulsetranslator app/worker.1:     publisher.publish(msg) 
Sep 02 09:27:40 pulsetranslator app/worker.1:   File "/app/.heroku/python/lib/python2.7/site-packages/mozillapulse/publishers.py", line 62, in publish 
Sep 02 09:27:40 pulsetranslator app/worker.1:     routing_key=message.routing_key) 
Sep 02 09:27:40 pulsetranslator app/worker.1:   File "/app/.heroku/python/lib/python2.7/site-packages/kombu/messaging.py", line 85, in __init__ 
Sep 02 09:27:40 pulsetranslator app/worker.1:     self.revive(self._channel) 
Sep 02 09:27:40 pulsetranslator app/worker.1:   File "/app/.heroku/python/lib/python2.7/site-packages/kombu/messaging.py", line 222, in revive 
Sep 02 09:27:40 pulsetranslator app/worker.1:     self.declare() 
Sep 02 09:27:40 pulsetranslator app/worker.1:   File "/app/.heroku/python/lib/python2.7/site-packages/kombu/messaging.py", line 105, in declare 
Sep 02 09:27:40 pulsetranslator app/worker.1:     self.exchange.declare() 
Sep 02 09:27:40 pulsetranslator app/worker.1:   File "/app/.heroku/python/lib/python2.7/site-packages/kombu/entity.py", line 171, in declare 
Sep 02 09:27:40 pulsetranslator app/worker.1:     return self.channel.exchange_declare( 
Sep 02 09:27:40 pulsetranslator app/worker.1:   File "/app/.heroku/python/lib/python2.7/site-packages/kombu/abstract.py", line 115, in channel 
Sep 02 09:27:40 pulsetranslator app/worker.1:     channel = self._channel = channel() 
Sep 02 09:27:40 pulsetranslator app/worker.1:   File "/app/.heroku/python/lib/python2.7/site-packages/kombu/utils/__init__.py", line 422, in __call__ 
Sep 02 09:27:40 pulsetranslator app/worker.1:     value = self.__value__ = self.__contract__() 
Sep 02 09:27:40 pulsetranslator app/worker.1:   File "/app/.heroku/python/lib/python2.7/site-packages/kombu/messaging.py", line 209, in <lambda> 
Sep 02 09:27:40 pulsetranslator app/worker.1:     channel = ChannelPromise(lambda: connection.default_channel) 
Sep 02 09:27:40 pulsetranslator app/worker.1:   File "/app/.heroku/python/lib/python2.7/site-packages/kombu/connection.py", line 756, in default_channel 
Sep 02 09:27:40 pulsetranslator app/worker.1:     self.connection 
Sep 02 09:27:40 pulsetranslator app/worker.1:   File "/app/.heroku/python/lib/python2.7/site-packages/kombu/connection.py", line 741, in connection 
Sep 02 09:27:40 pulsetranslator app/worker.1:     self._connection = self._establish_connection() 
Sep 02 09:27:40 pulsetranslator app/worker.1:   File "/app/.heroku/python/lib/python2.7/site-packages/kombu/connection.py", line 696, in _establish_connection 
Sep 02 09:27:40 pulsetranslator app/worker.1:     conn = self.transport.establish_connection() 
Sep 02 09:27:40 pulsetranslator app/worker.1:   File "/app/.heroku/python/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection 
Sep 02 09:27:40 pulsetranslator app/worker.1:     conn = self.Connection(**opts) 
Sep 02 09:27:40 pulsetranslator app/worker.1:   File "/app/.heroku/python/lib/python2.7/site-packages/amqp/connection.py", line 183, in __init__ 
Sep 02 09:27:40 pulsetranslator app/worker.1:     return self._x_open(virtual_host) 
Sep 02 09:27:40 pulsetranslator app/worker.1:   File "/app/.heroku/python/lib/python2.7/site-packages/amqp/connection.py", line 627, in _x_open 
Sep 02 09:27:40 pulsetranslator app/worker.1:     (10, 41),    # Connection.open_ok 
Sep 02 09:27:40 pulsetranslator app/worker.1:   File "/app/.heroku/python/lib/python2.7/site-packages/amqp/abstract_channel.py", line 67, in wait 
Sep 02 09:27:40 pulsetranslator app/worker.1:     self.channel_id, allowed_methods) 
Sep 02 09:27:40 pulsetranslator app/worker.1:   File "/app/.heroku/python/lib/python2.7/site-packages/amqp/connection.py", line 240, in _wait_method 
Sep 02 09:27:40 pulsetranslator app/worker.1:     self.method_reader.read_method() 
Sep 02 09:27:40 pulsetranslator app/worker.1:   File "/app/.heroku/python/lib/python2.7/site-packages/amqp/method_framing.py", line 189, in read_method 
Sep 02 09:27:40 pulsetranslator app/worker.1:     raise m 
Sep 02 09:27:40 pulsetranslator app/worker.1: IOError: Socket closed 
Sep 02 09:27:40 pulsetranslator app/worker.1: Sleeping for 5 seconds. 
Sep 02 09:27:45 pulsetranslator app/worker.1: Retrying...
(Reporter)

Updated

4 years ago
Assignee: nobody → jgriffin
Status: NEW → ASSIGNED
(Reporter)

Updated

3 years ago
Assignee: jgriffin → nobody
Status: ASSIGNED → NEW
You need to log in before you can comment on or make changes to this bug.