929690 - Quote for replacement hardware for crash-stats systems

Status: RESOLVED FIXED

(Infrastructure & Operations :: DCOps, task)

Description

[Selena Deckelmann :selenamarie :selena]

Here are our existing systems that need replacing:

41 - HP SL170zG6, each with 6 TB of local storage (1TBx6) and 48 GB ram
6 - HP BL460c (6 with 500GB SSD local storage)
4 - HP DL360 G7 (with 2.5 TB of RAID 10 storage on each)

We'd need to replace all of these with like systems.

Our low I/O systems are:
21 - HP BL2x220c G6 (somewhat high peak CPU utilization)

Plus 24 additional nodes that are all virtualizable.

I'd say the 45 "low I/O" systems are largely virtualizable. I'd prefer that we spread these across two blade chassis in two separate racks.  I don't know what density we can go for here, so sorry for waving my hands around a bit.

This is an investigation, rather than a request at this point.

Comment 1

[Derek Moore [:dmoore]]

Selena: assuming we can virtualize your 45 low-I/O systems, that leaves us with 51 physical nodes to be upgraded. Do you have any perspective on whether you actually need 51 distinct servers, or could they be replaced by fewer, more-powerful platforms?

Comment 2

[Selena Deckelmann :selenamarie :selena]

(In reply to Derek Moore [:dmoore] from comment #1)
> Selena: assuming we can virtualize your 45 low-I/O systems, that leaves us
> with 51 physical nodes to be upgraded. Do you have any perspective on
> whether you actually need 51 distinct servers, or could they be replaced by
> fewer, more-powerful platforms?

This is a hard question to answer right now. 

The Postgres and Elastic Search nodes are probably over-powered CPU-wise, but can use all the IO they can possibly ever get. We're also considering splitting up the Postgres databases, in anticipation of 100% processing. So, if anything, these nodes will multiply (postgres x2) rather than be reduced.

The HBase nodes are not necessarily a match for our workload, which is extremely write-heavy, and mostly only read-heavy for very recent data. :tmary would be the right person to weigh in on appropriate hardware and sizing for our current use.

Comment 3

[T [:tmary] Meyarivan]

(In reply to Selena Deckelmann :selenamarie :selena from comment #3)
> 41 - HP SL170zG6, each with 6 TB of local storage (1TBx6) and 48 GB ram

If these are the same as Hadoop/HBase nodes, please use the list of hosts at https://bugzilla.mozilla.org/show_bug.cgi?id=929056#c15 

Re replacing them with fewer, more-powerful platforms, it might be possible. Capacity requirements for next 2-3 years, monthly recurring costs and list of platforms being considered, would be useful in evaluation

--

Comment 4

[Selena Deckelmann :selenamarie :selena]

Resolved and budget submitted to laura.