1176491 - Stress-test the Heroku instance & check for correctness

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P2)

Description

[Ed Morley [:emorley]]

Before moving to Heroku, we need to:
* ensure that it can handle the full load of prod, and not just the low load when few people are using it.
* ensure that people can successfully submit to its API (since at the moment it's only handling buildbot data)
* check that the timeouts we were seeing previously are now all gone
* check for correctness
* check that eg deploying doesn't result in lost data (due to workers not shutting down gracefully etc). this is important since Heroku also results dynos once every 24 hrs, so we need to handle it with no loss.
* check that things like cycle-data are running correctly

Comment 1

[Ed Morley [:emorley]]

(In reply to Ed Morley [:emorley] from comment #0)
> * check that eg deploying doesn't result in lost data (due to workers not
> shutting down gracefully etc). this is important since Heroku also results
> dynos once every 24 hrs, so we need to handle it with no loss.

Sections worth reading:
https://devcenter.heroku.com/articles/dynos#automatic-dyno-restarts
https://devcenter.heroku.com/articles/dynos#shutdown

Worth emphasising:
"""
The application processes have ten seconds to shut down cleanly (ideally, they will do so more quickly than that). During this time they should stop accepting new requests or jobs and attempt to finish their current requests, or put jobs back on the queue for other worker processes to handle. If any processes remain after ten seconds, the dyno manager will terminate them forcefully with SIGKILL.
"""

It will be possible to see if we are hitting some of these cases by looking at any error codes shown on:
https://dashboard.heroku.com/apps/treeherder-heroku/metrics/web

The list of error codes is at:
https://devcenter.heroku.com/articles/error-codes

Comment 2

[Ed Morley [:emorley]]

All looks good :-)