1215464 - mozmill-ci Pulse listeners are no longer able to connect to broker running on pulse.mozilla.org

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Infrastructure, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Our Pulse listeners are timing out since yesterday due to a broken connection to the broker. Sadly they do not re-connect automatically. Not sure yet if it is related to some changes on the Pulse side lately. As of now more than 3000 jobs are in the queue...

2015-10-15T00:08:58Z WARNING kombu.mixins: Connection to broker lost. Trying to re-establish the connection...
2015-10-15T00:09:13Z WARNING kombu.mixins: Broker connection error: error(timeout('timed out',),). Trying again in 2.0 seconds.
2015-10-15T00:09:30Z WARNING kombu.mixins: Broker connection error: error(timeout('timed out',),). Trying again in 4.0 seconds.
2015-10-15T00:09:49Z WARNING kombu.mixins: Broker connection error: error(timeout('timed out',),). Trying again in 6.0 seconds.
2015-10-15T00:10:10Z WARNING kombu.mixins: Broker connection error: error(timeout('timed out',),). Trying again in 8.0 seconds.
2015-10-15T00:10:33Z WARNING kombu.mixins: Broker connection error: error(timeout('timed out',),). Trying again in 10.0 seconds.
2015-10-15T00:10:58Z WARNING kombu.mixins: Broker connection error: error(timeout('timed out',),). Trying again in 12.0 seconds.
2015-10-15T00:11:25Z WARNING kombu.mixins: Broker connection error: error(timeout('timed out',),). Trying again in 14.0 seconds.
2015-10-15T00:11:54Z WARNING kombu.mixins: Broker connection error: error(timeout('timed out',),). Trying again in 16.0 seconds.
2015-10-15T00:12:25Z WARNING kombu.mixins: Broker connection error: error(timeout('timed out',),). Trying again in 18.0 seconds.
2015-10-15T00:12:58Z WARNING kombu.mixins: Broker connection error: error(timeout('timed out',),). Trying again in 20.0 seconds.
2015-10-15T00:13:33Z WARNING kombu.mixins: Broker connection error: error(timeout('timed out',),). Trying again in 22.0 seconds.
2015-10-15T00:14:10Z WARNING kombu.mixins: Broker connection error: error(timeout('timed out',),). Trying again in 24.0 seconds.
2015-10-15T00:14:49Z WARNING kombu.mixins: Broker connection error: error(timeout('timed out',),). Trying again in 26.0 seconds.
2015-10-15T00:15:30Z WARNING kombu.mixins: Broker connection error: error(timeout('timed out',),). Trying again in 28.0 seconds.
2015-10-15T00:16:13Z WARNING kombu.mixins: Broker connection error: error(timeout('timed out',),). Trying again in 30.0 seconds.
2015-10-15T00:16:58Z WARNING kombu.mixins: Broker connection error: error(timeout('timed out',),). Trying again in 32.0 seconds.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Interestingly it started to happen at midnight from Oct 14th to Oct 15th.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I can successfully connect with those credentials from my local host. So I would assume there is a problem with the network in qa.scl3.mozilla.org which doesn't let us connect to Pulse.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

As mentioned on bug 1094272 comment 30 this is most likely due to the move to CloudAMQP and leaving us behind because we no longer can reach the new hosts which are restricted by ACL entries for our network. 

I will have to check which hosts exist now and get new ACL entries added.

In generally I wish we would do better transitions like that. :(

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Currently we have the following ACL setting:

source: * ; port: * --> dest: pulse.mozilla.org ; port 5671, 5672

Trying to connect to those ports times out:

XYZ@mm-ci-production:/data/mozmill-ci$ telnet pulse.mozilla.org 5671
Trying 54.215.253.142...
^C
XYZ@mm-ci-production:/data/mozmill-ci$ telnet pulse.mozilla.org 5672
Trying 54.215.254.97...
^C

I can ping the host but traceroute also fails to show any valuable data.

Here the current DNS settings:

$ nslookup pulse.mozilla.org
Server:         127.0.1.1
Address:        127.0.1.1#53

Name:   pulse.mozilla.org
Address: 54.215.253.142
Name:   pulse.mozilla.org
Address: 54.215.223.66
Name:   pulse.mozilla.org
Address: 54.215.254.97

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

So this actually got busted most likely by the work on bug 1205867.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

It's working again. But interestingly I do not see any drop of the messages in the queues.

queue/mozauto/mm-ci-production.qa.scl3.mozilla.com/production_update 51 messages
queue/mozauto/mm-ci-production.qa.scl3.mozilla.com/production_build 3397 messages 42%
queue/mozauto/mm-ci-staging.qa.scl3.mozilla.com/staging_update 51 messages
queue/mozauto/mm-ci-staging.qa.scl3.mozilla.com/staging_build 3397 messages 42%

How often is the number of messages in the queues updated on PulseGuardian? Marc, can you have a look at it please?

Comment 7

[Mark Côté [:mcote]]

Sorry, PulseGuardian has been having issues.  It's sorted out now, so your queue lengths should now be valid.

And sorry about the net-flow confusion.  We had updated the Buildbot flows, but I neglected to mention in my migration announcements that other data-centre apps would have to have theirs updated as well.  Note that the IPs will change occasionally (very infrequently).  We're coming up with a policy for that in bug 1208600.  It's worded just for Buildbot right now, but we'll expand it to include other data-centre apps.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Thanks Marc! All looks good again. So closing this bug out.