1311647 - Add fields that are available in SuperSearch to the crash report data available on Telemetry

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Marco Castelluccio [:marco]]

I've noticed a few fields are missing from https://raw.githubusercontent.com/mozilla/socorro/master/socorro/schemas/crash_report.json, compared to what is available via SuperSearch.

To move my correlation tool to use data from Telemetry instead of SuperSearch, I'd need a few fields that are currently missing.

For example, all the 'adapter_*' fields are missing, 'graphics_critical_error' is missing, etc.

If you don't want to add everything that is available on SuperSearch, I can build a list of the fields that I'm currently using.

Comment 1

[[DEACTIVATED] Adrian Gaudebert]

All of these are part of the raw_crash, which have decided not to send just yet. It will come a bit later, after we have confirmed that our system is in working order. 

However, given how we built it, adding any new field is just a matter of adding that field to the crash_report.json schema. I think. Peter can confirm that nothing is required on the Telemetry side.

So, if you have a list of the fields you want to see added, I can then add these to our schema.

Comment 2

[Peter Bengtsson [:peterbe]]

(In reply to Adrian Gaudebert [:adrian] from comment #1)
> All of these are part of the raw_crash, which have decided not to send just
> yet. It will come a bit later, after we have confirmed that our system is in
> working order. 
> 
> However, given how we built it, adding any new field is just a matter of
> adding that field to the crash_report.json schema. I think. Peter can
> confirm that nothing is required on the Telemetry side.
> 

Correct. And I have that confirmed by mreid. 

> So, if you have a list of the fields you want to see added, I can then add
> these to our schema.

Up to you Adrian, but perhaps best to wait till the dust settles a bit. I.e. we start sending prod crash data to telemetry.

Comment 3

[Marco Castelluccio [:marco]]

The fields that I'm currently using are: ['abort_message', 'accessibility', 'adapter_device_id', 'adapter_driver_version', 'adapter_subsys_id', 'adapter_vendor_id', 'addons', 'address', 'app_init_dlls', 'app_notes', 'available_virtual_memory', 'available_physical_memory', 'bios_manufacturer', 'build_id', 'contains_memory_report', 'cpu_arch', 'cpu_info', 'cpu_microcode_version', 'date', 'dom_ipc_enabled', 'gmp_plugin', 'graphics_critical_error', 'graphics_startup_test', 'ipc_channel_error', 'ipc_fatal_error_msg', 'ipc_fatal_error_protocol', 'ipc_message_name', 'ipc_system_error', 'is_garbage_collecting', 'jit_category', 'moz_crash_reason', 'number_of_processors', 'oom_allocation_size', 'platform', 'platform_pretty_version', 'platform_version', 'plugin_version', 'reason', 'safe_mode', 'shutdown_progress', 'signature', 'submitted_from_infobar', 'startup_crash', 'theme', 'total_physical_memory', 'total_virtual_memory', 'useragent_locale'].

Comment 4

[[DEACTIVATED] Adrian Gaudebert]

Marco, we are still working on sending all our data there. Once that is settled and we have proof that our system works well, I'll get to adding more fields to our document. Please bear with us! :)

Comment 5

[Marco Castelluccio [:marco]]

Adrian, do you think this can be done now?

Comment 6

[Lonnen :lonnen]

This is unlikely to be done before Hawaii

Comment 7

[Marco Castelluccio [:marco]]

Attachment: Link to Github pull-request: https://github.com/mozilla/socorro/pull/3653

Comment 8

[Lonnen :lonnen]

Hey Marco -- thank you for the pull request. Marshall wanted to check and approve the initial list, so I'm going to take these changes to Privacy for sign off before we can merge these.

Comment 9

[Lonnen :lonnen]

I'll take the bug for now since it's blocking on me.

Comment 10

[Marco Castelluccio [:marco]]

Attachment: Link to Github pull-request: https://github.com/mozilla/socorro/pull/3663

This is another batch, quite smaller, of (Android-specific) fields. If the privacy sign off hasn't started yet, perhaps we can add these too now. Anyway, they shouldn't be privacy sensitive at all, as they are all attributes of the Android device and they are all publicly available via SuperSearch.

Comment 11

[Lonnen :lonnen]

They're reviewing the document in the PR, but probably won't leave comments on github. I'm going to use this as an opportunity to update and remind them I need an answer about this.

Comment 12

[Lonnen :lonnen]

I've got a +1 to exercise my discretion and approve this and similar myself for now. I'll have a look at the open PRs in the morning.

Comment 13

[Lonnen :lonnen]

ok, r+ across the board. My work is done here.

Comment 14

[[github robot]]

Commit pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/543cbc06659b7bf3796d43575c3f2452e5475a13
[DO NOT MERGE] Bug 1311647 - Add some android specific fields to telemetry crash reports (#3663)

* Bug 1311647 - Add more fields to Telemetry crash reports

* Set _facets_size to 0 as we don't need facets

* Bug 1311647 - Add some Android-specific fields to Telemetry crash reports

Comment 15

[Marco Castelluccio [:marco]]

Marking as fixed, as the two PRs landed, even though bug 1340105 is needed before the fields are actually exposed.

Comment 16

[Peter Bengtsson [:peterbe]]

It is speculative that https://github.com/mozilla/socorro/commit/543cbc06659b7bf3796d43575c3f2452e5475a13#diff-82e3b30a9c5737b0a5e278a3d095bf27 broke the JSON Schema validation. 

Also, see https://bugzilla.mozilla.org/show_bug.cgi?id=1343623

Comment 17

[Marco Castelluccio [:marco]]

https://github.com/mozilla/socorro/commit/1c56ef48db6cddf9d9d885a8c6a1977c575cb37f
Fixes bug 1291173 and bug 1311647 by readding the new fields back with correct
types (#3688)