1181508 - Decide whether to enable the Heroku preboot feature

Status: RESOLVED WONTFIX

(Tree Management :: Treeherder: Infrastructure, defect, P3)

Description

[Ed Morley [:emorley]]

[~/src/treeherder]$ heroku features
=== App Features (treeherder-heroku)
[ ] preboot  Provide seamless web dyno deploys [general]

https://devcenter.heroku.com/articles/preboot

"""
Preboot changes the standard dyno start behavior for web dynos. Instead of stopping the existing set of web dynos before starting the new ones, preboot ensures that the new web dynos are started (and receive traffic) before the existing ones are terminated. This can contribute to zero downtime deployments.

This feature should be used carefully - it could lead to situations where you have two versions of your code bases running side by side, where your web dynos are running an old release while your workers are running a new release - and your code and database configuration should support this behavior.
"""

Comment 1

[Ed Morley [:emorley]]

More info:

"""
Once preboot is enabled, you can deploy as usual, but you’ll see a different behavior:

* Logs will show new dynos starting soon after the slug compile is finished. The output of heroku ps will immediately show the state (for example, starting or up) of the new dynos; old dynos are still running but won’t appear in heroku ps.
* Approximately 3 minutes after the deploy completes, HTTP requests will start routing to the new dynos and simultaneously stop routing to the old dynos.
* Shortly after the new dynos are fully live and serving user requests, the old dynos will be killed. You’ll see them killed in the logs as usual.

This behavior change only applies to the web process type. Other dynos (such as workers) will continue to be killed and restarted normally.

Preboot applies any time the web dynos get restarted, regardless of the cause. So for example, any kind of deployment, config var change, app restart or automatic dyno restart will make use of this behavior.


Caveats:
* Whoever is doing the deployment will have to wait a few minutes before the new code starts serving user requests; this happens later than it would without preboot (but in the meanwhile, user requests are still served by old dynos).
* There will be a short period (a minute or two) where heroku ps shows the status of the new dynos, but user requests are still being served by old dynos.
* You will have two versions of your code running at the same time, overlapping for up to 2 minutes, although only one will be serving user requests. You should be careful to make sure they interact gracefully. Some add-ons and other services limit the number of simultaneous connections; running twice as many dynos makes you more likely to hit such a limit. 
"""

So sounds like there are pros/cons.

I wonder if we even need to use it? I mean all of our tasks _should_ be made to retry if a request fails (though not yet the case: bug 1113235) - so the web dynos being unavailable for a few seconds (we need to time it admittedly) shouldn't be a big deal. I wonder how the UI handles it? Perhaps we could throw a few retries/bit of error handling in the UI (which would help for slow connections etc anyway) instead?

Comment 2

[Ed Morley [:emorley]]

This can both wait until after the main move, and will likely be easier to assess either way, once we've already transitioned.

Comment 3

[Ed Morley [:emorley]]

I don't really think we need to do this.