1224780 - Deploy Push Endpoint server Redshift derived stream nightly job

Status: RESOLVED FIXED

(Cloud Services :: Operations: Metrics/Monitoring, task, P1)

Description

[Rob Miller [:rmiller]]

Attachment: push_load_bundle.tar.gz

The attached tarball contains everything needed to perform the nightly processing of Push Endpoint server logs (from the `heka-logs` S3 bucket) in order to populate a Redshift derived stream from which usage metrics can be generated.

The tarball is meant to be untarred in the `ec2-user` home folder on a fresh Centos 7 instance (tested with ami-bb2b238b). This will extract a `push_load` folder, which contains a `push_load.sh` script that, when run, will install all needed RPMs and run the hindsight_cli job that performs the desired operation.

The only thing not covered by the script is permissions. Access to the `heka-logs` AWS S3 bucket and the Push production Redshift db must be specified. Instructions on where to set the Redshift credentials are included in the comments of the `push_load.sh` script.

This job should be run once per day. It presumes that the *previous* day's autopush log files have been processed by the Heka aggregator and are in place in the S3 bucket.

Comment 1

[Daniel Thorn [:relud]]

Attachment: push_load.tar.gz

I created an rpm from this modified tarball where everything is in "/opt/push_load". The redshift info is written by puppet. spec file used:

> Name:           push_load
> Version:        0.1
> Release:        1
> Summary:        push_load
> 
> License:        MPL 2.0
> Source0:        push_load.tar.gz
> 
> Requires:       postgresql-libs
> Requires:       luasandbox-core
> Requires:       hindsight
> 
> %pre
> ldconfig
> 
> %description
> Job for load push metrics into redshift
> 
> %install
> tar -xf '%{SOURCE0}' -C '%{buildroot}'
> 
> %clean
> rm -rf '%{buildroot}'
> 
> %files
> %defattr(-,root,root,-)
> /opt/push_load

Comment 2

[Daniel Thorn [:relud]]

this is deployed in prod, and will run at 0:30 UTC daily.