1272524 - Deploy push connection server data loading and backfill

Status: RESOLVED FIXED

(Cloud Services :: Operations: Metrics/Monitoring, task)

Description

[Rob Miller [:rmiller]]

The Push log loading process is being changed so that it correctly loads the connection server logs in addition to the endpoint server logs that are already captured. Also, the nightly email report generated by the `report.lua` script is going away, replaced by a Python script that accesses the loaded data and uses it to generate roll up tables that will populate re:dash dashboards.

The first step is to perform the backfill. I've created an 'initial_conn_backfill' branch of the https://github.com/mozilla-services/push_derived repo (note it's new home in the mozilla-services organization). This is all set up to load and generate the roll up tables for the last 35 days of connection server activity. It should not impact the endpoint tables at all. NOTE: This will likely take a while, and will also consume a lot of disk space. I've been using an EC2 instance w 256GiB of storage, and even then I had to delete some of the queue files in the hs_output folder after the cursor advanced past them to ensure that the machine didn't run out of space. Suboptimal, I know. :(

After the backfill has happened, then the nightly job can be changed to use the latest commit from master (d71928320795984c5c2775433741d37a2c984deb). This works just like before, installing all needed dependencies onto a fresh EC2 instance, running the hindsight job, and then installing and running the Python script that performs the rollups. There are 4 cfg files in the hindsight/hs_run/output file and one db_creds.sh file where the Redshift data credentials should be specified.