767022 - Puppetize dependencies for Graphite and Redis servers?

Status: RESOLVED WONTFIX

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Comment 1

[Brandon Burton [:solarce]]

Review currently installed stuff on briarpatch1-2.private.scl3 and start a briarpatch puppet module for setting these two servers up

Review currently installed python deps on briarpatch-processor1-2.seamicro.phx1 that are appropriate to install via Puppet, vs :bear packaging in vendor/

Comment 2

[Brandon Burton [:solarce]]

Starting by puppetizing stage using our graphite and redis modules

Comment 3

[Brandon Burton [:solarce]]

As :Aj finished the VMs this afternoon, I proceeded to Puppetize the stage VMs first

briarpatch-graphite-stage1.private.scl3.mozilla.com has our current Graphite module, which includes some basic Apache and statsd configs as well
 * we may need to work on the module to abstract/override stuff you do/don't need :)

briarpatch-redis-stage1.private.scl3.mozilla.com has our current Redis module, with node definition stuff to tell Redis to use 10GB of RAM and persist the DB

I should be able to work on dev first thing tomorrow

Comment 4

[Lonnen :lonnen]

Thanks!

Comment 5

[Brandon Burton [:solarce]]

(In reply to Chris Lonnen :lonnen from comment #4)
> Thanks!

In working on putting our Graphite module on briarpatch1.private.scl3, :bear noticed some differences in your previous Graphite configs, vs what we ship with Puppet. (admittedly our Graphite module is extremely stock and only used on box before briarpatch)

The main differences being in storage-schemas.conf, diff below

[[e-mail]3 opt]# diff /opt/archive/graphite/conf/storage-schemas.conf /opt/graphite/conf/storage-schemas.conf
1,18c1,3
< # Schema definitions for Whisper files. Entries are scanned in order,
< # and first match wins. This file is scanned for changes every 60 seconds.
< #
< #  [name]
< #  pattern = regex
< #  retentions = timePerPoint:timeToStore, timePerPoint:timeToStore, ...
< 
< # Carbon's internal metrics. This entry should match what is specified in
< # CARBON_METRIC_PREFIX and CARBON_METRIC_INTERVAL settings
< [carbon]
< pattern = ^carbon\.
< retentions = 60:90d
< 
< #[default_1min_for_1day]
< #pattern = .*
< #retentions = 60s:1d
< 
< [stats]
---
> # whisper file size is ~1.7mb/file
> [longterm]
> priority = 100
20,22c5
< aggregationMethod = sum
< retentions = 60s:12h,1h:60d,1d:5y
< #retentions = 10s:6h,1h:60d
---
> retentions = 1m:30d,15m:3y

The main concern, I believe is 'aggregationMethod = sum', do you know offhand if having/not having that makes a big difference for existing Whisper files?

If adding to an existing Graphite install is harmless, I can add it to Puppet, but that'd effect our graphite1.dmz.phx1 server

If it does affect existing Whisper files, but you really need it, I'll start a process to make it a tunable option in Puppet, but that will require a little work

Comment 6

[Melissa O'Connor [:melissa]]

Solarce, are you still waiting for updates on the following to move forward?

The main concern, I believe is 'aggregationMethod = sum', do you know offhand if having/not having that makes a big difference for existing Whisper files?

If adding to an existing Graphite install is harmless, I can add it to Puppet, but that'd effect our graphite1.dmz.phx1 server

If it does affect existing Whisper files, but you really need it, I'll start a process to make it a tunable option in Puppet, but that will require a little work

Comment 7

[Brandon Burton [:solarce]]

(In reply to Melissa O'Connor [:moconnor] from comment #6)
> Solarce, are you still waiting for updates on the following to move forward?
> 
> The main concern, I believe is 'aggregationMethod = sum', do you know
> offhand if having/not having that makes a big difference for existing
> Whisper files?
> 
> If adding to an existing Graphite install is harmless, I can add it to
> Puppet, but that'd effect our graphite1.dmz.phx1 server
> 
> If it does affect existing Whisper files, but you really need it, I'll start
> a process to make it a tunable option in Puppet, but that will require a
> little work

Whoops, I didn't realize I hadn't updated this after Friday's meeting.

I've been working on this on and off this week, but :bear, :lonnen, and I spent time on Friday on Vidyo discussing this and they've made the progress they need to and we're gonna circle back next week to test out the work I've been doing on their new puppet module, as I'm at a conference tomorrow/fri and they are busy with b2g and git this week too

Comment 8

[Mike Taylor [:bear]]

the supervisor config for statsd was referencing /usr/bin/node but the node install had the binary at /usr/local/bin/node

added a symlink to work around this until puppet module tweaks are don