1343623 - Why didn't validate_and_test.py catch type mismatches

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Peter Bengtsson [:peterbe]]

See https://bugzilla.mozilla.org/show_bug.cgi?id=1343619
We added some fields to JSON Schema, ran the validate_and_test.py but it didn't notice the type difference. It should. 

The resolve this, take the original from https://github.com/mozilla/socorro/pull/3682 and run validate_and_test.py and it should find flaws on some types.

Comment 1

[Peter Bengtsson [:peterbe]]

One immediate thought is that doesn't ProcessedCrash filter out fields if you view the crash without an access token or cookie?

Comment 2

[Peter Bengtsson [:peterbe]]

(In reply to Peter Bengtsson [:peterbe] from comment #1)
> One immediate thought is that doesn't ProcessedCrash filter out fields if
> you view the crash without an access token or cookie?

Scratch that.

Comment 3

[Peter Bengtsson [:peterbe]]

The reason it can't work is both simple and sad. 

What the processor does is that it takes the processed crash, merges it with the raw crash and renames fields according to Super Search Fields. 
E.g.

 processed_crash = {'foo': 'Foo'}
 raw_crash = {'DOMIPCEnabled': '1'}
 merged = {'foo': 'Foo', 'DOMIPCEnabled': '1'}
 renamed_according_to_supersearch_fields = {'foo': 'Foo', 'dom_ipc_enabled': '1'}
 and send that to the Telemetry S3 bucket. 

When validate_and_test.py, I think, was originally written we only used fields from the processed crash. It was our gentle start. 

The solution to this must be to try to simulate what https://github.com/mozilla/socorro/blob/3d63f17aabc6a10875d2d23507077df42970bbf8/socorro/external/boto/crashstorage.py#L338-L377 does. 

As a side-note, either this is a lie (as in, it's not doing any validation) or it's buggy: https://github.com/mozilla/socorro/blob/3d63f17aabc6a10875d2d23507077df42970bbf8/socorro/external/boto/crashstorage.py#L373

Comment 4

[Peter Bengtsson [:peterbe]]

A possible solution is to change validate_and_test.py to import `from socorro.external.boto.crashstorage import TelemetryBotoS3CrashStorage` and using some Auth Token or something try to pull down the SuperSearchFields and do the merge locally and *then* validate.

Comment 5

[Peter Bengtsson [:peterbe]]

Attachment: Link to Github pull-request: https://github.com/mozilla/socorro/pull/3683

Comment 6

[[github robot]]

Commit pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/b378c36cada9fd85d6e8b8fff8b32941a62237da
fixes bug 1343623 - validate_and_test.py should test more realistically (#3683)