1080602 - Treeherder staging DBs

Status: RESOLVED FIXED

(Data & BI Services Team :: DB: MySQL, task)

Description

[Kendall Libby [:fubar] (he/him)]

We're setting up a staging environment for treeherder in SCL3 that closely matches production (and will handle comparable load). We'll need new DBs for it. I believe the plan is to only store 1-2 months of data, unlike the 6+ months for prod (and are getting increased). Prod db set up is in bug 998491.

I expect you'll have questions that I can't answer, so I've CCed the treeherder team. :-)

Thanks!

Comment 1

[Kendall Libby [:fubar] (he/him)]

ping?

Comment 2

[Brandon Johnson [:cyborgshadow]]

So we're realistically looking at 300GB disks and db machines identical to prod otherwise.

I'm in talks with the virt team about seeing if this can be a reality now.

Comment 3

[Brandon Johnson [:cyborgshadow]]

These exist but haven't been setup prior to my departure. Please set them up and communicate with the folks in #treeherder for it's use. I believe we won't be transferring any data to it but I'm not 100% certain.

Comment 4

[Kendall Libby [:fubar] (he/him)]

ping? need to have these set up soon, please. no data needs to be xferred.

Comment 5

[Sheeri Cabral [:sheeri]]

Started building them, got the MySQL packages on it, need to change the root pw to get puppet running properly, then will make a user and pass.

Comment 6

[Kendall Libby [:fubar] (he/him)]

checking in to see where we are. do we have VIPs yet so I can get flows set up? thx, sheeri1

Comment 7

[Sheeri Cabral [:sheeri]]

Got replication working, still need to set up monitoring and backups. bug for VIP is bug 1104797.

Comment 8

[Kendall Libby [:fubar] (he/him)]

The th_admin user is missing privs, I think. IIRC, it actually needs to create a database *per tree*, in addition to the treeherder_stage db. Looking at prod, I see a whole bunch of dbs, which seems to bear this out.

Can the staging db users have their privs set to match prod, please? Thanks!

Comment 9

[Sheeri Cabral [:sheeri]]

I've matched privileges (not passwords) with production. 

th_admin - GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, SHOW DATABASES, CREATE TEMPORARY TABLES, LOCK TABLES, EVENT ON *.*

th_user - GRANT SELECT, INSERT, UPDATE, DELETE ON *.*

Comment 10

[Sheeri Cabral [:sheeri]]

added both treeherder machines to nagios monitoring groups:
                'virtual',
                'mysql2-puppetized-servers',
                'db-swap-high',
                'mysql-masters',
                'mysql-slaves',
                'mysql-checksum',
                'mysql-rw'

MacBook-Air:hosts scabral$ svn commit -m "adding treeherder machines to nagios stage"
Sending        scl3.pp
Transmitting file data .
Committed revision 97966.

pushed to Nagios, all is green.

Left to do-  backups.

Comment 11

[Sheeri Cabral [:sheeri]]

backups are complete. Resolving.