820510 - utilities to push JSON logs from metlog up to HDFS

Status: VERIFIED WONTFIX

(Cloud Services Graveyard :: Server: Sync, defect, P3)

Description

[Victor Ng [:vng] [:crankycoder]]

Attachment: sample config file for upload_logs.py

We currently push out the scripts to upload metlog generated JSON logs to HDFS by bundling the admin scripts with the logstash-metlog RPM.

Talking to ops, it's probably a better idea to ship those admin scripts separately from the logstash-metlog RPM as the scripts have little to no value for people who are using logstash/metlog outside of Mozilla. 

I'm attaching the configuration file and the latest upload_logs.py script from logstash-metlog (0.8.8)

Comment 1

[Victor Ng [:vng] [:crankycoder]]

Attachment: latest upload_log.py from logstash-metlog 0.8.8

Comment 2

[Victor Ng [:vng] [:crankycoder]]

merged into the server-syncstorage repo on the rfk/management-scripts branch as of  c07ba26e6c

https://github.com/mozilla-services/server-syncstorage/commit/c07ba26e6c

Comment 3

[James Bonacci [:jbonacci]]

I think these changes have either come and gone or where never merged.
:rfkelly can you confirm?

Comment 4

[Ryan Kelly [:rfkelly]]

IIRC this was never merged, and is no longer needed in the new heka-based universe

Comment 5

[James Bonacci [:jbonacci]]

Yea!