1168418 - all heroku apps: queue, scheduler, auth, index, mozilla-taskcluster, events

Status: RESOLVED WONTFIX

(Taskcluster :: General, defect)

Description

[James Lal [:lightsofapollo]]

Use "forever" or something similar to manage app crashes to bypass heroku exponential retries.

Comment 1

[James Lal [:lightsofapollo]]

We should also alert when we are down for N minutes (N should be configurable).

Comment 2

[Jonas Finnemann Jensen (:jonasfj)]

I'm not sure this works for anything but background processes, see:
https://devcenter.heroku.com/articles/error-codes#r10-boot-timeout

Basically, heroku will crash an app if it doesn't start listen to $PORT within 60s.
Sure we could setup a reverse proxy manager too, but they heroku router can't guide requests around
the dynos that are restarting. Which dynos do every 24 hours.
The point having multiple dynos is that restarts don't necessarily drop connections.

At the end of the day, I think doing this is fighting heroku, which is probably the wrong thing to do.

Comment 3

[Jonas Finnemann Jensen (:jonasfj)]

I believe we agreed to close this because it can't be done right, See Comment 2.

Some day we'll probably move heroku apps to a docker based platform and this won't be an issue.

Comment 4

[Pete Moore [:pmoore][:pete]]

Resetting Version and Target Milestone that accidentally got changed...