984660 - Store diamond logStore diamond logs as hostname.log in s3. Only report some aggregation to hosted-graphite in s3

Status: RESOLVED WONTFIX

(Infrastructure & Operations Graveyard :: CIDuty, task)

Comment 1

[(dormant account)]

Tools like graphite do not work well with a cloud architecture where instances can only last a couple of hours and you have 1000s of them. In case of hosted-graphite, they could cope with that. It could cost us 10s of thousands of dollars per month yet aggregation of 1000s of logs would still be slow.

However graphite is still pretty useful for aggregated data, we should keep using it for that. 

So the way I envision this working is:
* During node lifetime diamond should save data to a log on disk
* Each node should upload the diamond log to s3 as ec2-instanceid.log.gz before shutdown(could do it every X minutes too to deal with getting killed by spot)
* Each node should also submit stats to an aggregation node which would forward them to hosted-graphite(I think hosted graphite can host it for us via hosted statsd).

Then whenever we need to do analysis, we can have a script pull relevant diamond logs from s3 and either do analysis locally or feed it to some graphite instance.

This will make it cheaper and easier to correlate diamond perf info with the following logs: build, spot price, aws cloudwatch. Main downside is a little bit of diy and possibly having to deal with timestamp drift on each node(atm we probably use hosted-graphite timestamps).