1150986 - Set up a MYSQL DB on RDS for the prototype Heroku instance

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P2)

Description

[Kendall Libby [:fubar] (he/him)]

background: we'd like to follow in taskcluster's footsteps and get treeherder running on Heroku (ideally for realz, but at least as a proof of concept) as a Q2 goal. Converting treeherder from mysql to postgres is not under consideration, but that means using an RDS instance (or other external-to-heroku mysql server). 

I know IT's working on a brand spankin' new AWS greenfield, so I thought I'd get a converstation started.

* any general recommendations on RDS? 
* should this be in IT's AWS acct, and/or is this quarter even reasonable?
* heroku no longer gives out its AWS acct ID and security group name to folks, so access control is somewhat less than ideal (see https://devcenter.heroku.com/articles/amazon-rds) 

I'm sure I had more questions, but *friday afternoon*. CC'ing Ed and Cam as I'm sure they'll have their own questions.

Thanks!

K.

Comment 1

[Kendall Libby [:fubar] (he/him)]

stood up an rds instance in the dev services acct, just so we have something to poke at. ssl is required, but 3306 is otherwise open to all. no databases created yet; msater user is th_admin, see me for pw.

rds ca cert is at http://s3.amazonaws.com/rds-downloads/rds-combined-ca-bundle.pem

relevant django config:
DATABASES['default']['OPTIONS'] = {'ssl': {'ca': '/path/to/rds-combined-ca-bundle.pem'}}`

and for heroku:
heroku config:add DATABASE_URL="mysql2://th_admin:[e-mail]:3306/db_name?sslca=path/to/rds-combined-ca-bundle.pem" -a app_id

Comment 2

[Mauro Doglio [:mdoglio]]

This is used now by http://treeherder-heroku.herokuapp.com .
I think we still need to understand what's the story for replication, failover, etc
:sheeri any ideas?

Comment 3

[Sheeri Cabral [:sheeri]]

Well, RDS is DBaaS - database as a service - so it should already be redundant, with any failovers transparent to us.

Thus, there should be no need to have replication or failover.

The exception to this is if you want redundancy among multiple Availability Zones - which would be the equivalent of multiple data centers - which you currently do not have today (treeherder is only in the scl3 data center, with no redundancy in the phx1 data center).

You also want to keep in mind: monitoring and backups.

Comment 4

[Mauro Doglio [:mdoglio]]

Thanks :sheeri, I was looking for something comparable to the master-slave replication we have on the current prod db. It looks like we need to explicitly enable read replicas as it says here that http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html

Comment 5

[Sheeri Cabral [:sheeri]]

Right, you can do that, but your case of having a read replica is because the master cannot handle all of the traffic. Do you know that RDS cannot handle all of your traffic, even on the largest instance? 

If you want to just put the treeherder dbs in the cloud and not change the architecture for optimal cloud use, you could set up EC2 instances and run MySQL from there. Part of the cool part about RDS is they make some things easier, like perhaps not having to have a read replica.

Comment 6

[Mauro Doglio [:mdoglio]]

I don't know yet how the RDS instance will perform for real, we have very low read traffic there for now. Maybe I should avoid early optimizations and see how it behaves when we have more traffic.

Comment 7

[Ed Morley [:emorley]]

Seems like we're all done here - for this instance at least. Once we start thinking in terms of an official stage/prod Heroku setup we'll need to file additional bugs for creating instances for them.

Moving back to the treeherder component, since we did this in-house.

Comment 8

[Ed Morley [:emorley]]

Which RDS instance are we on at the moment?

Comment 9

[Ed Morley [:emorley]]

instance type that is

Comment 10

[Kendall Libby [:fubar] (he/him)]

m3.xlarge w/ 100gb storage, atm

Comment 11

[Ed Morley [:emorley]]

Thank you :-)