Stress-test the Heroku instance & check for correctness

RESOLVED FIXED

Status

Tree Management
Treeherder: Infrastructure
P2
normal
RESOLVED FIXED
3 years ago
a year ago

People

(Reporter: emorley, Assigned: emorley)

Tracking

Details

(Assignee)

Description

3 years ago
Before moving to Heroku, we need to:
* ensure that it can handle the full load of prod, and not just the low load when few people are using it.
* ensure that people can successfully submit to its API (since at the moment it's only handling buildbot data)
* check that the timeouts we were seeing previously are now all gone
* check for correctness
* check that eg deploying doesn't result in lost data (due to workers not shutting down gracefully etc). this is important since Heroku also results dynos once every 24 hrs, so we need to handle it with no loss.
* check that things like cycle-data are running correctly
(Assignee)

Updated

3 years ago
Component: Treeherder → Treeherder: Infrastructure
QA Contact: laura
(Assignee)

Comment 1

2 years ago
(In reply to Ed Morley [:emorley] from comment #0)
> * check that eg deploying doesn't result in lost data (due to workers not
> shutting down gracefully etc). this is important since Heroku also results
> dynos once every 24 hrs, so we need to handle it with no loss.

Sections worth reading:
https://devcenter.heroku.com/articles/dynos#automatic-dyno-restarts
https://devcenter.heroku.com/articles/dynos#shutdown

Worth emphasising:
"""
The application processes have ten seconds to shut down cleanly (ideally, they will do so more quickly than that). During this time they should stop accepting new requests or jobs and attempt to finish their current requests, or put jobs back on the queue for other worker processes to handle. If any processes remain after ten seconds, the dyno manager will terminate them forcefully with SIGKILL.
"""

It will be possible to see if we are hitting some of these cases by looking at any error codes shown on:
https://dashboard.heroku.com/apps/treeherder-heroku/metrics/web

The list of error codes is at:
https://devcenter.heroku.com/articles/error-codes
(Assignee)

Updated

2 years ago
Priority: P3 → P2
(Assignee)

Updated

2 years ago
Depends on: 1176487, 1249146
(Assignee)

Comment 2

a year ago
All looks good :-)
Assignee: nobody → emorley
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.