813317 - Reporting replica for Socorro

Status: RESOLVED FIXED

(Data & BI Services Team :: DB: MySQL, task)

Description

[Selena Deckelmann :selenamarie :selena]

We've got a replica for Socorro's primary database, and have been directing some users to run read-only queries against it. 

If this service becomes more popular, the chance that a long-running query causes the replica to go out of sync with the master database increases. Replication in Postgres can be delayed by certain read-only queries because of lock contention.

To avoid a production problem, it seems wise to have an additional replica dedicated to running queries against, that is configured to recover from prolonged replication delay and is not considered a primary failover target.

Comment 1

[Sheeri Cabral [:sheeri]]

What are the desired specs for the box? disk size, SSD vs. spinning disks, RAM size, CPU speed / # CPUs?

Comment 2

[Sheeri Cabral [:sheeri]]

2.5 Tb spinning disk after RAIDing, whatever CPUs come with it (quad or 8-core is plenty), at least 12 Gb RAM but 24 or higher would be nice. This is an 800Gb database, so more RAM=better.

Comment 3

[Shyam Mani [:fox2mike]]

(In reply to Sheeri Cabral [:sheeri] from comment #2)
> 2.5 Tb spinning disk after RAIDing, whatever CPUs come with it (quad or
> 8-core is plenty), at least 12 Gb RAM but 24 or higher would be nice. This
> is an 800Gb database, so more RAM=better.

The max you can get for blades is 1TB disk. Is that sufficient?

Comment 4

[Sheeri Cabral [:sheeri]]

Well, the db itself is 800 Gb right now, so I don't think limiting ourselves to 1Tb is a good idea.

What's the alternatives...seamicros? Have we ever tested postgres on seamicros (we haven't in the year or so I've been here, to my knowledge).

Comment 5

[Sheeri Cabral [:sheeri]]

Corey will order an HP system for this.

Comment 6

[Corey Shields [:cshields]]

We ordered 2 "sandbox" pg nodes for socorro, we can use one of those for this (should be installed this week) and I am ordering a third to backfill that one.

Comment 7

[Rich Pomper]

Bacfill shipped to Phoenix.  here is tracking

Storage Blade Fed Ex 017136230767311 ETA 1/25
Hard drives (need to be installed) shipped via On Trac ETA 1/22
Server Blade shipped via Fed Ex 013840237703237 ETA 1/24

Comment 8

[Vinh Hua [:vinh]]

The blade server and storage blade has been installed here:  

https://inventory.mozilla.org/en-US/systems/racks/?location=12&status=&rack=280&allocation=


Asset #'s:

15536
15535

Comment 9

[Corey Shields [:cshields]]

closing this out as we have them being kicked in another bug.  Thanks all!