1467146 - cloud-mirror: REDIS_URL is automatically updated by heroku, but not used by cloud-mirror

Status: RESOLVED WONTFIX

(Taskcluster :: Services, enhancement)

Description

[Jonas Finnemann Jensen (:jonasfj)]

heroku will automatically update redis and switch to a fallback redis instance if the first one fails. When this happens the REDIS_URL env var is updated.

But cloud-mirror uses REDIS_{HOST,PORT,PASS}, so cloud-mirror just stops working.
Ideally we would:
 A) update cloud-mirror frontend to read REDIS_URL
 B) update cloud-mirror copiers to fetch the REDIS_URL from the frontend through
    an authenticated API request.

Yesterday, heroku updated redis, cloud-mirror crashed and we had to update env vars in the heroku app and password-store followed by a redeployment with terraform. This is very slow
and error prone. Deploying with terraform to fix tree closure isn't nice :)

Note:
  In theory I suspect that heroku might change the REDIS_URL env var whenever. I'm not
  convinced they only do it for crashes, I think it happens when they upgrade redis
  deployment, but I could be wrong. That's just how I read the messages on heroku.

Comment 1

[Jonas Finnemann Jensen (:jonasfj)]

@jhford, I'm not sure that (B) is a good solution. Maybe you have some better ideas?

Comment 2

[John Ford [:jhford] CET/CEST Berlin Time]

We don't hit this very frequently at all, and we're aiming to replace cloud-mirror's role with the new object service.  It is also an efficiency issue ($ & time) to have cloud-mirror down, rather than a tree-closing event.  I think we should consider not doing this work unless it begins happening frequently.

That said, if we're going to do it at all, we should do a & b at the same time to avoid the situation where a credential change results in queueing up a bunch of requests which we know aren't going to be fulfilled in time.

If you agree, let's close this bug in favour of working on the object service.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

I'll agree for jonas ;)