1297560 - retry pulse connection establishment

Status: RESOLVED FIXED

(Taskcluster :: Services, defect)

Description

[Jonas Finnemann Jensen (:jonasfj)]

This took us down because 2 / 3 servers when offline.

We can do this at our level, or do it upstream which seems ideal if we can get some buy in, see:
https://github.com/squaremo/amqp.node/issues/268

Comment 1

[Jonas Finnemann Jensen (:jonasfj)]

Solution here is probably just to retry connection establishment, rather than trying to be smart with DNS.

Comment 2

[Jonas Finnemann Jensen (:jonasfj)]

Fixed when we land:

https://github.com/taskcluster/taskcluster-client/pull/57
https://github.com/taskcluster/pulse-publisher/pull/13

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

(without looking at the bug) did you confirm that this will cause systems to perform a new DNS lookup on every retry?

Comment 4

[Jonas Finnemann Jensen (:jonasfj)]

nope... and I don't know how we would force a new DNS lookup. I think it depends on the kernel, which probably relies on TTL.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

It depends on the resolver lib being used -- the kernel doesn't do DNS.

Anyway, that's OK, just something to be aware of.  It seems CloudAMQP keeps their TTL low, so this shouldn't be a huge issue:

orange-antelope.rmq.cloudamqp.com. 30 IN CNAME  ec2-52-53-240-221.us-west-1.compute.amazonaws.com.