1312006 - Upload crash_reports.json to S3

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Peter Bengtsson [:peterbe]]

When the Telemetry team ingest crashes we've saved in S3, they need to download the JSON Schema. Every morning. 
The URL they download is https://raw.githubusercontent.com/mozilla/socorro/master/socorro/schemas/crash_report.json and even thought GitHub is awesome it's not as reliable as S3 for reads. And if S3 isn't working, no ingestion can be done anyway. 

The task is to take that .json file and upload it to the root of the S3 bucket used for telemetry. 

No versioning needed. It can be uploaded on an hourly basis. In the "root" of the bucket.

Comment 1

[[DEACTIVATED] Adrian Gaudebert]

How would you handle updates to the schema file? Does it add a manual step every time we change it, or can think of a way to automate that?

Comment 2

[Peter Bengtsson [:peterbe]]

We'll just replace the previously uploaded one. No versioning.

Comment 3

[[DEACTIVATED] Adrian Gaudebert]

Yeah, but do we have to do that manually, or will you automate that?

Comment 4

[Peter Bengtsson [:peterbe]]

(In reply to Adrian Gaudebert [:adrian] from comment #3)
> Yeah, but do we have to do that manually, or will you automate that?

The plan is to write a crontabber app (which will have access to the socorro.schemas package) that just makes an S3 post/put every day.

Comment 5

[Peter Bengtsson [:peterbe]]

Adrian had a smart idea...
In a couple of months when people forget how this all works, they might find the JSON Schema there on S3 and wonder if that's the source of truth. What we should do is write a $description text that explains where this file belongs and comes from etc.

Comment 6

[Peter Bengtsson [:peterbe]]

Attachment: Link to Github pull-request: https://github.com/mozilla/socorro/pull/3564

Comment 7

[Peter Bengtsson [:peterbe]]

Once this lands, we need to upgrade the admin nodes. 

For stage:

1) Add `consulate kv add socorro/crontabber/class-UploadCrashReportJSONSchemaCronApp.bucket_name "org-allizom-telemetry-crashes"`
2) The of course we need to add  `socorro.cron.jobs.upload_crash_report_json_schema.UploadCrashReportJSONSchemaCronApp|1h`  to `socorro/crontabber/crontabber.jobs`

For prod:

1) Add `consulate kv add socorro/crontabber/class-UploadCrashReportJSONSchemaCronApp.bucket_name "org-mozilla-telemetry-crashes"`
2) The of course we need to add  `socorro.cron.jobs.upload_crash_report_json_schema.UploadCrashReportJSONSchemaCronApp|1h`  to `socorro/crontabber/crontabber.jobs`

Comment 8

[[github robot]]

Commit pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/304dd1704f8e2f21f7c112cb7a5d66cca4d9d1af
bug 1312006 - Upload crash_reports.json to S3 (#3564)

* bug 1312006 - Upload crash_reports.json to S3

* changing the description

Comment 9

[Peter Bengtsson [:peterbe]]

This has now been applied on stage. The URI is https://s3-us-west-2.amazonaws.com/org-allizom-telemetry-crashes/crash_report.json

Comment 10

[Peter Bengtsson [:peterbe]]

This has now been applied on prod too. The URI is https://s3-us-west-2.amazonaws.com/org-mozilla-telemetry-crashes/crash_report.json