1031419 - Make Angela very happy by doing another round of cost savings on aws

Status: RESOLVED INCOMPLETE

(Webmaker Graveyard :: DevOps, defect)

Description

[JP Schneider :jp]

Last year we cut our costs by 25%, but over the course of a year, we've added a bunch of new apps and infrastructure.  We're now back to the same place, cost-wise, as we were.

A few things to do here:

1) Delete unused RDS instances
2) Consider resizing DB instances to new classes
3) redo our reservations for db's (big up front cost, big savings)
4) redo our reservations for ec2 scaling groups (big up front cost, big savings)
5) resize autoscale groups based on utilization and new instance sizing analysis

For reference, since May, we average about $ 18.68 an hour.  Lets see how low we can get that!

Comment 1

[JP Schneider :jp]

Backed up and removed the following db instances:
csplogger-production-a
csplogger-staging-a
monitoring-production-b
source-staging-b
comemessusup
deletemeyo
wmprofile-production-a
wmprofile-staging-a
devdb-a

Comment 2

[JP Schneider :jp]

This leaves us with the following # of instances on RDS:

  17 db.t1.micro
  11 db.m1.small

Comment 3

[JP Schneider :jp]

OK, to reserve our current production instances:

9 multi-az t1.micros (0.05 per hour), for $3942/year
7 multi-az m1.smalls (0.11 per hour), for $6745/year

Reserved light usage:
Micro: $39 up front per instance, then $0.027/hr 
Smalls: $104 up front per instance, then $0.063/hr

For us to take advantage:
Micros:  $351 up front, then $2128 for a year, saving $1463 per year
Smalls: $728 up front, then $3863 for a year, saving $2153 per year


So then, my total ask for today is $1079.

Comment 4

[JP Schneider :jp]

EC2 TLDR:
If we spend $6572 today, we save $8740 for the next year.  


Stuff we have reserved for 8 more weeks:
26 m1.mediums in each AZ

Usage right now by AZ/Size
20 t1.micros in us-east-1d
7 t1.micros in us-east-1c
3 t1.micros in us-east-1b
11 m1.smalls in us-east-1d
16 m1.smalls in us-east-1c
10 m1.smalls in us-east-1b
37 m1.mediums in us-east-1d
23 m1.mediums in us-east-1c
30 m1.mediums in us-east-1b
4 m1.larges in us-east-1c
2 m1.larges in us-east-1d
2 m3.2xlarge in us-east-1c
1 m3.xlarge in us-east-1b

t1.micro unreserved pricing: $0.020 / hr
m1.small unreserved pricing: $0.044 / hr
m1.medium unreserved pricing: $0.087 / hr
m3.medium unreserved pricing: $0.070 / hr
m1.large unreserved pricing: $0.175 / hr
m3.large unreserved pricing: $0.140 / hr
m3.xlarge unreserved pricing: $0.28 / hr
m3.2xlarge unresreved pricing: $0.560 / hr


Reserved light usage pricing
t1.micro: $23 up front, then $0.012 / hr
m1.small: $61 up front, then $0.027 / hr
m1.medium: $122 up front, then $0.054 / hr
m3.medium: $110 up front, then $0.05 / hr
m3.xlarge: $439 up front, then $0.201 / hr
m3.2xlarge: $879 up front, then $0.401 / hr

For t1.micros, action plan:
Reserve 15 in us-east-1d, 5 in us-east-1c, 3 in us-east-1b
23 total t1.micro's: $529 up front, then $2418 for the next year for $1082 in yearly savings

For m1.smalls, action plan:
Reserve 12 in each AZ: $2196 up front, then $8514 for the next year, for $3165 in yearly savings

For m1.mediums, we'll revisit more in 8 weeks when our likely-unsellable reservations expire.  For now, I'll plan to move a couple AS groups to these cheaper instances.
Reserve 15 m3.mediums (5 in each AZ): $1650 up front, then $6570 for the rest of the year, saving $3211 a year

For m1.larges, action plan:
I'll need to work on resizing some mongo and elasticsearch servers due to this.  We'll hold off touching this until we update our makeapi elasticsearch.

For m3.xlarge, action plan:
Reserve 1 instance in us-east-1b: $439 up front, then $1760 for the next year, for a savings of $254 a year.

For m3.2xlarge, action plan:
Reserve 2 instances in us-east-1c: $1758 up front, then $7025 for the next year, for a savings of $1028 for the next year

Comment 5

[JP Schneider :jp]

I removed six autoscale groups, three security groups, released two unused elastic ips, deleted the following iam-users:

app-aestimia-cem-produciton
app-aestimia-cem-staging
app-aestimia-development
app-aestimia-ob-production
app-aestimia-ob-staging
app-aestimia-production
app-aestimia-staging
app-collusion-staging
app-csol-development
app-huboard-staging
app-huboard-production
app-http-manipulator-staging
app-http-manipulator-production
app-huboard-test
app-openbadger-wm-development
app-openbadger-ob-staging
app-openbadger-ob-production
app-openbadger-csol-development
app-webpagemaker-staging
app-webpagemaker-test
app-webpagemaker-production
asgard-production

Comment 6

[JP Schneider :jp]

I have submitted the RDS reservations, but am stuck on EC2.  We want to move to t2. instances, but htose don't seem to have any available reservations. :/

Comment 7

[JP Schneider :jp]

To move to t2's, we have to migrate our stuff to VPC's.  I've got an open issue on that.

Comment 8

[Chris DeCairos (:cade)]

Closing this out as we move to deprecate the Webmaker Component.

Issues can be re-filed at https://github.com/mozillafoundation/mofo-devops/issues/new if required.