Closed Bug 1339273 Opened 9 years ago Closed 9 years ago

Make Autopush S3 heka log bucket available to a.t.m.o.

Categories

(Cloud Services Graveyard :: Metrics: Pipeline, defect)

Product:
Cloud Services Graveyard
Component:
Metrics: Pipeline
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: benbangert, Unassigned)

Details

:relud mentioned that since Autopush is still using Logging 1.0, the scrubbed heka protobuf logs are stored in a S3 bucket (s3://heka-logs/shared/YYYY-MM/autopush-app.log-YYYYMMDD_<hostname>.gz). This bucket should be made available to atmo so that we can setup data analysis for our data.
Unfortunately that s3 bucket does not contain scrubbed logs. There is however a bucket that contains the scrubbed data, which is s3://net-mozaws-prod-us-west-2-autopush-logs. The files are uploaded minutely so you will need to pull down ~1400 files to process a day's logs. I've updated the bucket policy to allow access from ATMO to that bucket, and also filed and applied https://github.com/mozilla/emr-bootstrap-spark/pull/63 to add it to the IAM role permissions. "aws s3 ls s3://net-mozaws-prod-us-west-2-autopush-logs/autopush/YYYYMMDD" will list the files for a particular day from an ATMO cluster. Let me know if you run into any issues.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Product: Cloud Services → Cloud Services Graveyard
You need to log in before you can comment on or make changes to this bug.