Automate JSON schema validation

NEW
Unassigned

Status

2 years ago
2 years ago

People

(Reporter: peterbe, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

2 years ago
At the moment we have validate_and_test.py [0]
The idea is that you run that script from the command line from whence you are doing your intended changes to crash_report.json [1].

Also, you're ideally expected to try to really "push the envelope" with this script. If you don't supply a URL to a Super Search API query, e.g. `python validate_and_test.py` it will just test the 100 most recent crashes. Likely that most of them are Firefox Release crashes. 
So if you're adding a new field called `"foo_bar":{"type": "integer"}` you're expected to pass in a URL where Super Search makes sure crashes that have this are included. 

Would be nice to automate all of this in some efficient manner. 



[0] https://github.com/mozilla/socorro/blob/master/socorro/schemas/validate_and_test.py
[1] https://github.com/mozilla/socorro/blob/master/socorro/schemas/crash_report.json
Do we have any idea how often this comes up? Is it a once-a-week sort of thing or once-a-month or something else?
(Reporter)

Comment 2

2 years ago
(In reply to Will Kahn-Greene [:willkg] ET needinfo? me from comment #1)
> Do we have any idea how often this comes up? Is it a once-a-week sort of
> thing or once-a-month or something else?

About once a month. But the whole system just landed in production in Dec 2016.
(Reporter)

Comment 3

2 years ago
That frequency might increase. I suspect it will but then maybe it'll become (at most) once a week.

Running validate_and_test.py is slow. It's because it has to do a LOT of network IO. I.e. download 100 raw crashes and 100 processed crashes. (Mind you, validate_and_test.py does exercise that TelemetryBotoS3CrashStorage [0] class nicely)

We could potentially check the content of the git branch's difference and if "crash_report.json" is mentioned, then execute validate_and_test.py

Another idea is to run validate_and_test.py frequentlyish using cron on stage (perhaps against a mix of stage and prod data). 
That means, if we too hastily land something in crash_report.json, land it in master and break crontabber on stage, we'd be alerted through monitoring. 

More crazy ideas?


[0] https://github.com/mozilla/socorro/blob/5e1d5585ca03256043ae83832da552cde328db84/socorro/external/boto/crashstorage.py#L294
(Reporter)

Comment 4

2 years ago
For what it's worth at least we have decent documentation about editing crash_report.json AND how to test your changes.
https://github.com/mozilla/socorro/blob/master/socorro/schemas/README.md#testing-schema-changes
You need to log in before you can comment on or make changes to this bug.