2052944 - Add all missing Crash Ping fields to Socorro's raw and processed crash schema

Status: ASSIGNED

(Socorro :: General, task, P1)

Description

[Bianca Danforth [:bdanforth]]

Crash Pings contain some fields in CrashAnnotations.yaml that are not present in Socorro's raw and processed crash schema or are classified as protected in the current schema.

In both cases, this causes those fields to appear in the "Protected data" section of /report/index for Crash Pings, rather than being surfaced correctly as public data. It also blocks correct SuperSearch behavior for Crash Ping queries, since SUPERSEARCH_FIELDS permissions are derived from the schemas.

This ticket aligns Socorro's schemas with CrashAnnotations.yaml for Crash Pings.

Acceptance Criteria

• Every annotation in CrashAnnotations.yaml with scope: ping is present in socorro/schemas/raw_crash.schema.yaml
• Every annotation in CrashAnnotations.yaml with scope: ping is present in socorro/schemas/processed_crash.schema.yaml
• Every scope:ping field in both schemas has permissions: ["public"]
  • Note: there are at least 3 fields in Crash Pings that are currently listed as "protected" that should be moved to "public": MozCrashReason, RemoteType, and JavaException.
• Unit tests added or updated for the schema changes.
  • We could make use of validate_raw_crash.py here for the raw crash schema changes.

Comment 1

[Bianca Danforth [:bdanforth]]

We should also make the recommended uptime changes as described in https://bugzilla.mozilla.org/show_bug.cgi?id=2055875#c12.

Comment 2

[Bianca Danforth [:bdanforth]]

We should also add the Glean client ID to both the raw and processed crash schema as a private field per Bug 2052950.

Comment 3

[Bianca Danforth [:bdanforth]]

We should cross-reference the fields being added to Crash Pings in Bug 2049749 in case any are added outside of those already in Socorro.

Comment 4

[Bianca Danforth [:bdanforth]]

I've spent a good amount of time thinking about this, and to completely do this, it represents about 23 (or more) tickets worth of work, since there are 23 of 58 scope: "ping" crash annotations present in Crash Pings today but missing from Socorro (with an additional five being added in Bug 2049749 which is in review), and about half of them don't have an explicit data review and approval by a data steward. Those that do limit access/visibility to Firefox telemetry (visible to all MoCo but not the general public in BigQuery etc.)[1]. In our case, making a field public in Socorro means making it public to the world on our website. Because of this, we should really add these fields carefully with a new data review, in many cases on a field-by-field basis. This, among other reasons, is why we typically handle adding a single new field to Socorro as its own separate ticket.

Additionally, by definition, these are fields that are not in Socorro. It's therefore guaranteed that no one is querying them in Crash Stats today, and we will not break any existing workflows by deferring adding them to follow-up work as needed/requested. As with all fields Socorro doesn't know about, they will be viewable in the raw crash document only and classified as "protected" until then.

Given this, and in discussion with Sven and Alex Franchuk, I have significantly reduced the scope of this ticket to cover the absolute minimum changes needed for this project:

• Ensure that critical path consumers can query existing Crash Report fields that are currently derived from a minidump (Crash Pings won't have a minidump to derive the fields from)
  • These are fields that I previously identified in Bug 2055875 Comment 10 that are being added as new crash annotations in Bug 2049749).
  • I've just confirmed the exact fields with Alex: CPUArchitecture, CPUInfo, OS, OSVersion, LinuxLSBDescription
• Ensure we have the information we need to handle Firefox telemetry data deletion requests.

Revised acceptance criteria:

• CPUArchitecture, CPUInfo, OS, OSVersion, LinuxLSBDescription are added to the raw crash schema
• Existing processor rules to determine the processed crash equivalents of these fields (cpu_arch, cpu_info, os_name, os_version and os_pretty_version, respectively) need to be updated to use the derived value from the minidump and fallback to the value provided in the crash annotation.
• The processed crash schema description for these fields should be updated to mention that we conditionally use the provided source annotation (or add a comment if possible to that effect).
• Copy and index the Glean client ID as a protected field into the processed crash report for handling data deletion requests as mentioned in Comment 2.
• Unit tests added or updated for the schema and processor changes.

[1]: The crash-pings.mozilla.org website does surface some crash ping data publicly, but it doesn't provide a comprehensive view of each crash ping and is quite limited in terms of search/filtering functionality relative to Socorro, in addition to being a much newer website that many fewer people know about.

Comment 5

[Bianca Danforth [:bdanforth]]

Attachment: bug-2052944: fall back to crash annotation values for platform info when there's no minidump - #7249

Comment 6

[Bianca Danforth [:bdanforth]]

I merged bug-2052944: fall back to crash annotation values for platform info when there's no minidump - #7249.

Hoping to have the second of two PRs up later today.

Comment 7

[Bianca Danforth [:bdanforth]]

Attachment: bug-2052944-data-review-request.txt

The second patch (in progress) for this bug is to add support for the Glean client ID in Socorro as a protected field. This is a data review request for adding this field to Socorro.

While the actual data collection will not occur until we start submitting crash pings to Socorro much later on in the larger effort of Bug 2049499, we require a data review link at the point of adding a new field to Socorro's schema, which is what this ticket covers.

Comment 8

[Chris H-C :chutten]

Comment on attachment 9628472 [details]
bug-2052944-data-review-request.txt

DATA COLLECTION REVIEW RESPONSE:

Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes.

Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection can be controlled through the product's preferences.

If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, Bianca Danforth is responsible.

Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 1, Technical, but also Category 4 Highly Sensitive (persistent identifiers are identified this way as being both benignly technical but also worth scrutiny)

Is the data collection request for default-on or default-off?

Default on for all channels.

Does the instrumentation include the addition of any new identifiers?

No. The Glean client_id is already reported, just not persisted.

Is the data collection covered by the existing Firefox privacy notice?

Yes.

Does the data collection use a third-party collection tool?

No.

---

Result: datareview+

Comment 9

[Bianca Danforth [:bdanforth]]

Attachment: bug-2052944: add Glean client ID as a protected field - #7251

Comment 10

[Bianca Danforth [:bdanforth]]

I merged bug-2052944: add Glean client ID as a protected field - #7251.