I've spent a good amount of time thinking about this, and to completely do this, it represents about 23 (or more) tickets worth of work, since there are 23 of 58 `scope: "ping"` crash annotations present in Crash Pings but missing from Socorro (excluding those being added in Bug 2049749), and most of them don't have an explicit data review and aproval by a data steward. Those that do limit access/visiblity to Firefox telemetry. In our case, making a field public in Socorro means making it public to the world on our website. Because of this, we should really add these fields carefully with a new data review, in many cases on a field-by-field basis. This, among other reasons, is why we typically handle adding a single new field to Socorro as its own separate ticket. Additionally, by definition, these are fields that are not in Socorro. It's therefore guaranteed that no one is querying them in Crash Stats today, and we will not break any existing workflows by deferring adding them to follow-up work as needed/requested. As with all fields Socorro doesn't know about, they will be viewable in the raw crash document only and classified as "protected" until then. Given this, and in discussion with Sven and Alex Franchuk, I have signficantly reduced the scope of this ticket to cover the absolute minimum changes needed for this project: * Ensure that critical path consumers can query existing Crash Report fields that are currently derived from a minidump * These are fields that I previously identified in [Bug 2055875 Comment 10](https://bugzilla.mozilla.org/show_bug.cgi?id=2055875#c10) that are being added as new crash annotations in Bug 2049749). * I've just confirmed the exact fields with Alex: `CPUArchitecture`, `CPUInfo`, `OS`, `OSVersion`, `LinuxLSBDescription` Revised acceptance criteria: * `CPUArchitecture`, `CPUInfo`, `OS`, `OSVersion`, `LinuxLSBDescription` are added to the raw crash schema * Existing processor rules to determine the processed crash equivalents of these fields (`cpu_arch`, `cpu_info`, `os_name`, `os_version` and `os_pretty_version`, respectively) need to be updated to use the derived value from the minidump and fallback to the value provided in the crash annotation. * The processed crash schema description for these fields should be updated to mention that we conditionally use the provided source annotation (or add a comment if possible to that effect). * Copy and index the Glean client ID as a protected field into the processed crash report for handling data deletion requests as mentioned in [Comment 2](https://bugzilla.mozilla.org/show_bug.cgi?id=2052944#c2). * Unit tests added or updated for the schema and processor changes.
Bug 2052944 Comment 4 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
I've spent a good amount of time thinking about this, and to completely do this, it represents about 23 (or more) tickets worth of work, since there are 23 of 58 `scope: "ping"` crash annotations present in Crash Pings but missing from Socorro (excluding the five being added in Bug 2049749), and most of them don't have an explicit data review and aproval by a data steward. Those that do limit access/visiblity to Firefox telemetry. In our case, making a field public in Socorro means making it public to the world on our website. Because of this, we should really add these fields carefully with a new data review, in many cases on a field-by-field basis. This, among other reasons, is why we typically handle adding a single new field to Socorro as its own separate ticket. Additionally, by definition, these are fields that are not in Socorro. It's therefore guaranteed that no one is querying them in Crash Stats today, and we will not break any existing workflows by deferring adding them to follow-up work as needed/requested. As with all fields Socorro doesn't know about, they will be viewable in the raw crash document only and classified as "protected" until then. Given this, and in discussion with Sven and Alex Franchuk, I have signficantly reduced the scope of this ticket to cover the absolute minimum changes needed for this project: * Ensure that critical path consumers can query existing Crash Report fields that are currently derived from a minidump * These are fields that I previously identified in [Bug 2055875 Comment 10](https://bugzilla.mozilla.org/show_bug.cgi?id=2055875#c10) that are being added as new crash annotations in Bug 2049749). * I've just confirmed the exact fields with Alex: `CPUArchitecture`, `CPUInfo`, `OS`, `OSVersion`, `LinuxLSBDescription` Revised acceptance criteria: * `CPUArchitecture`, `CPUInfo`, `OS`, `OSVersion`, `LinuxLSBDescription` are added to the raw crash schema * Existing processor rules to determine the processed crash equivalents of these fields (`cpu_arch`, `cpu_info`, `os_name`, `os_version` and `os_pretty_version`, respectively) need to be updated to use the derived value from the minidump and fallback to the value provided in the crash annotation. * The processed crash schema description for these fields should be updated to mention that we conditionally use the provided source annotation (or add a comment if possible to that effect). * Copy and index the Glean client ID as a protected field into the processed crash report for handling data deletion requests as mentioned in [Comment 2](https://bugzilla.mozilla.org/show_bug.cgi?id=2052944#c2). * Unit tests added or updated for the schema and processor changes.
I've spent a good amount of time thinking about this, and to completely do this, it represents about 23 (or more) tickets worth of work, since there are 23 of 58 `scope: "ping"` crash annotations present in Crash Pings but missing from Socorro (excluding the five being added in Bug 2049749), and most of them don't have an explicit data review and approval by a data steward. Those that do limit access/visiblity to Firefox telemetry. In our case, making a field public in Socorro means making it public to the world on our website. Because of this, we should really add these fields carefully with a new data review, in many cases on a field-by-field basis. This, among other reasons, is why we typically handle adding a single new field to Socorro as its own separate ticket. Additionally, by definition, these are fields that are not in Socorro. It's therefore guaranteed that no one is querying them in Crash Stats today, and we will not break any existing workflows by deferring adding them to follow-up work as needed/requested. As with all fields Socorro doesn't know about, they will be viewable in the raw crash document only and classified as "protected" until then. Given this, and in discussion with Sven and Alex Franchuk, I have signficantly reduced the scope of this ticket to cover the absolute minimum changes needed for this project: * Ensure that critical path consumers can query existing Crash Report fields that are currently derived from a minidump * These are fields that I previously identified in [Bug 2055875 Comment 10](https://bugzilla.mozilla.org/show_bug.cgi?id=2055875#c10) that are being added as new crash annotations in Bug 2049749). * I've just confirmed the exact fields with Alex: `CPUArchitecture`, `CPUInfo`, `OS`, `OSVersion`, `LinuxLSBDescription` Revised acceptance criteria: * `CPUArchitecture`, `CPUInfo`, `OS`, `OSVersion`, `LinuxLSBDescription` are added to the raw crash schema * Existing processor rules to determine the processed crash equivalents of these fields (`cpu_arch`, `cpu_info`, `os_name`, `os_version` and `os_pretty_version`, respectively) need to be updated to use the derived value from the minidump and fallback to the value provided in the crash annotation. * The processed crash schema description for these fields should be updated to mention that we conditionally use the provided source annotation (or add a comment if possible to that effect). * Copy and index the Glean client ID as a protected field into the processed crash report for handling data deletion requests as mentioned in [Comment 2](https://bugzilla.mozilla.org/show_bug.cgi?id=2052944#c2). * Unit tests added or updated for the schema and processor changes.
I've spent a good amount of time thinking about this, and to completely do this, it represents about 23 (or more) tickets worth of work, since there are 23 of 58 `scope: "ping"` crash annotations present in Crash Pings but missing from Socorro (excluding the five being added in Bug 2049749), and about half of them don't have an explicit data review and approval by a data steward. Those that do limit access/visiblity to Firefox telemetry. In our case, making a field public in Socorro means making it public to the world on our website. Because of this, we should really add these fields carefully with a new data review, in many cases on a field-by-field basis. This, among other reasons, is why we typically handle adding a single new field to Socorro as its own separate ticket. Additionally, by definition, these are fields that are not in Socorro. It's therefore guaranteed that no one is querying them in Crash Stats today, and we will not break any existing workflows by deferring adding them to follow-up work as needed/requested. As with all fields Socorro doesn't know about, they will be viewable in the raw crash document only and classified as "protected" until then. Given this, and in discussion with Sven and Alex Franchuk, I have signficantly reduced the scope of this ticket to cover the absolute minimum changes needed for this project: * Ensure that critical path consumers can query existing Crash Report fields that are currently derived from a minidump * These are fields that I previously identified in [Bug 2055875 Comment 10](https://bugzilla.mozilla.org/show_bug.cgi?id=2055875#c10) that are being added as new crash annotations in Bug 2049749). * I've just confirmed the exact fields with Alex: `CPUArchitecture`, `CPUInfo`, `OS`, `OSVersion`, `LinuxLSBDescription` Revised acceptance criteria: * `CPUArchitecture`, `CPUInfo`, `OS`, `OSVersion`, `LinuxLSBDescription` are added to the raw crash schema * Existing processor rules to determine the processed crash equivalents of these fields (`cpu_arch`, `cpu_info`, `os_name`, `os_version` and `os_pretty_version`, respectively) need to be updated to use the derived value from the minidump and fallback to the value provided in the crash annotation. * The processed crash schema description for these fields should be updated to mention that we conditionally use the provided source annotation (or add a comment if possible to that effect). * Copy and index the Glean client ID as a protected field into the processed crash report for handling data deletion requests as mentioned in [Comment 2](https://bugzilla.mozilla.org/show_bug.cgi?id=2052944#c2). * Unit tests added or updated for the schema and processor changes.
I've spent a good amount of time thinking about this, and to completely do this, it represents about 23 (or more) tickets worth of work, since there are 23 of 58 [`scope: "ping"` crash annotations present in Crash Pings today](https://searchfox.org/firefox-main/rev/ee8e457ae20efb27313476fed0bed50571c1199f/toolkit/crashreporter/CrashAnnotations.yaml#1) but missing from Socorro (excluding the five being added in Bug 2049749), and about half of them don't have an explicit data review and approval by a data steward. Those that do limit access/visiblity to Firefox telemetry. In our case, making a field public in Socorro means making it public to the world on our website. Because of this, we should really add these fields carefully with a new data review, in many cases on a field-by-field basis. This, among other reasons, is why we typically handle adding a single new field to Socorro as its own separate ticket. Additionally, by definition, these are fields that are not in Socorro. It's therefore guaranteed that no one is querying them in Crash Stats today, and we will not break any existing workflows by deferring adding them to follow-up work as needed/requested. As with all fields Socorro doesn't know about, they will be viewable in the raw crash document only and classified as "protected" until then. Given this, and in discussion with Sven and Alex Franchuk, I have signficantly reduced the scope of this ticket to cover the absolute minimum changes needed for this project: * Ensure that critical path consumers can query existing Crash Report fields that are currently derived from a minidump * These are fields that I previously identified in [Bug 2055875 Comment 10](https://bugzilla.mozilla.org/show_bug.cgi?id=2055875#c10) that are being added as new crash annotations in Bug 2049749). * I've just confirmed the exact fields with Alex: `CPUArchitecture`, `CPUInfo`, `OS`, `OSVersion`, `LinuxLSBDescription` Revised acceptance criteria: * `CPUArchitecture`, `CPUInfo`, `OS`, `OSVersion`, `LinuxLSBDescription` are added to the raw crash schema * Existing processor rules to determine the processed crash equivalents of these fields (`cpu_arch`, `cpu_info`, `os_name`, `os_version` and `os_pretty_version`, respectively) need to be updated to use the derived value from the minidump and fallback to the value provided in the crash annotation. * The processed crash schema description for these fields should be updated to mention that we conditionally use the provided source annotation (or add a comment if possible to that effect). * Copy and index the Glean client ID as a protected field into the processed crash report for handling data deletion requests as mentioned in [Comment 2](https://bugzilla.mozilla.org/show_bug.cgi?id=2052944#c2). * Unit tests added or updated for the schema and processor changes.
I've spent a good amount of time thinking about this, and to completely do this, it represents about 23 (or more) tickets worth of work, since there are 23 of 58 [`scope: "ping"` crash annotations present in Crash Pings today](https://searchfox.org/firefox-main/rev/ee8e457ae20efb27313476fed0bed50571c1199f/toolkit/crashreporter/CrashAnnotations.yaml#1) but missing from Socorro (with an additional five being added in Bug 2049749 which is in review), and about half of them don't have an explicit data review and approval by a data steward. Those that do limit access/visiblity to Firefox telemetry. In our case, making a field public in Socorro means making it public to the world on our website. Because of this, we should really add these fields carefully with a new data review, in many cases on a field-by-field basis. This, among other reasons, is why we typically handle adding a single new field to Socorro as its own separate ticket. Additionally, by definition, these are fields that are not in Socorro. It's therefore guaranteed that no one is querying them in Crash Stats today, and we will not break any existing workflows by deferring adding them to follow-up work as needed/requested. As with all fields Socorro doesn't know about, they will be viewable in the raw crash document only and classified as "protected" until then. Given this, and in discussion with Sven and Alex Franchuk, I have signficantly reduced the scope of this ticket to cover the absolute minimum changes needed for this project: * Ensure that critical path consumers can query existing Crash Report fields that are currently derived from a minidump * These are fields that I previously identified in [Bug 2055875 Comment 10](https://bugzilla.mozilla.org/show_bug.cgi?id=2055875#c10) that are being added as new crash annotations in Bug 2049749). * I've just confirmed the exact fields with Alex: `CPUArchitecture`, `CPUInfo`, `OS`, `OSVersion`, `LinuxLSBDescription` Revised acceptance criteria: * `CPUArchitecture`, `CPUInfo`, `OS`, `OSVersion`, `LinuxLSBDescription` are added to the raw crash schema * Existing processor rules to determine the processed crash equivalents of these fields (`cpu_arch`, `cpu_info`, `os_name`, `os_version` and `os_pretty_version`, respectively) need to be updated to use the derived value from the minidump and fallback to the value provided in the crash annotation. * The processed crash schema description for these fields should be updated to mention that we conditionally use the provided source annotation (or add a comment if possible to that effect). * Copy and index the Glean client ID as a protected field into the processed crash report for handling data deletion requests as mentioned in [Comment 2](https://bugzilla.mozilla.org/show_bug.cgi?id=2052944#c2). * Unit tests added or updated for the schema and processor changes.
I've spent a good amount of time thinking about this, and to completely do this, it represents about 23 (or more) tickets worth of work, since there are 23 of 58 [`scope: "ping"` crash annotations present in Crash Pings today](https://searchfox.org/firefox-main/rev/ee8e457ae20efb27313476fed0bed50571c1199f/toolkit/crashreporter/CrashAnnotations.yaml#1) but missing from Socorro (with an additional five being added in Bug 2049749 which is in review), and about half of them don't have an explicit data review and approval by a data steward. Those that do limit access/visiblity to Firefox telemetry. In our case, making a field public in Socorro means making it public to the world on our website. Because of this, we should really add these fields carefully with a new data review, in many cases on a field-by-field basis. This, among other reasons, is why we typically handle adding a single new field to Socorro as its own separate ticket. Additionally, by definition, these are fields that are not in Socorro. It's therefore guaranteed that no one is querying them in Crash Stats today, and we will not break any existing workflows by deferring adding them to follow-up work as needed/requested. As with all fields Socorro doesn't know about, they will be viewable in the raw crash document only and classified as "protected" until then. Given this, and in discussion with Sven and Alex Franchuk, I have signficantly reduced the scope of this ticket to cover the absolute minimum changes needed for this project: * Ensure that critical path consumers can query existing Crash Report fields that are currently derived from a minidump (Crash Pings won't have a minidump to derive the fields from) * These are fields that I previously identified in [Bug 2055875 Comment 10](https://bugzilla.mozilla.org/show_bug.cgi?id=2055875#c10) that are being added as new crash annotations in Bug 2049749). * I've just confirmed the exact fields with Alex: `CPUArchitecture`, `CPUInfo`, `OS`, `OSVersion`, `LinuxLSBDescription` Revised acceptance criteria: * `CPUArchitecture`, `CPUInfo`, `OS`, `OSVersion`, `LinuxLSBDescription` are added to the raw crash schema * Existing processor rules to determine the processed crash equivalents of these fields (`cpu_arch`, `cpu_info`, `os_name`, `os_version` and `os_pretty_version`, respectively) need to be updated to use the derived value from the minidump and fallback to the value provided in the crash annotation. * The processed crash schema description for these fields should be updated to mention that we conditionally use the provided source annotation (or add a comment if possible to that effect). * Copy and index the Glean client ID as a protected field into the processed crash report for handling data deletion requests as mentioned in [Comment 2](https://bugzilla.mozilla.org/show_bug.cgi?id=2052944#c2). * Unit tests added or updated for the schema and processor changes.
I've spent a good amount of time thinking about this, and to completely do this, it represents about 23 (or more) tickets worth of work, since there are 23 of 58 [`scope: "ping"` crash annotations present in Crash Pings today](https://searchfox.org/firefox-main/rev/ee8e457ae20efb27313476fed0bed50571c1199f/toolkit/crashreporter/CrashAnnotations.yaml#1) but missing from Socorro (with an additional five being added in Bug 2049749 which is in review), and about half of them don't have an explicit data review and approval by a data steward. Those that do limit access/visiblity to Firefox telemetry. In our case, making a field public in Socorro means making it public to the world on our website. Because of this, we should really add these fields carefully with a new data review, in many cases on a field-by-field basis. This, among other reasons, is why we typically handle adding a single new field to Socorro as its own separate ticket. Additionally, by definition, these are fields that are not in Socorro. It's therefore guaranteed that no one is querying them in Crash Stats today, and we will not break any existing workflows by deferring adding them to follow-up work as needed/requested. As with all fields Socorro doesn't know about, they will be viewable in the raw crash document only and classified as "protected" until then. Given this, and in discussion with Sven and Alex Franchuk, I have signficantly reduced the scope of this ticket to cover the absolute minimum changes needed for this project: * Ensure that critical path consumers can query existing Crash Report fields that are currently derived from a minidump (Crash Pings won't have a minidump to derive the fields from) * These are fields that I previously identified in [Bug 2055875 Comment 10](https://bugzilla.mozilla.org/show_bug.cgi?id=2055875#c10) that are being added as new crash annotations in Bug 2049749). * I've just confirmed the exact fields with Alex: `CPUArchitecture`, `CPUInfo`, `OS`, `OSVersion`, `LinuxLSBDescription` * Ensure we have the information we need to handle Firefox telemetry data deletion requests. Revised acceptance criteria: * `CPUArchitecture`, `CPUInfo`, `OS`, `OSVersion`, `LinuxLSBDescription` are added to the raw crash schema * Existing processor rules to determine the processed crash equivalents of these fields (`cpu_arch`, `cpu_info`, `os_name`, `os_version` and `os_pretty_version`, respectively) need to be updated to use the derived value from the minidump and fallback to the value provided in the crash annotation. * The processed crash schema description for these fields should be updated to mention that we conditionally use the provided source annotation (or add a comment if possible to that effect). * Copy and index the Glean client ID as a protected field into the processed crash report for handling data deletion requests as mentioned in [Comment 2](https://bugzilla.mozilla.org/show_bug.cgi?id=2052944#c2). * Unit tests added or updated for the schema and processor changes.
I've spent a good amount of time thinking about this, and to completely do this, it represents about 23 (or more) tickets worth of work, since there are 23 of 58 [`scope: "ping"` crash annotations present in Crash Pings today](https://searchfox.org/firefox-main/rev/ee8e457ae20efb27313476fed0bed50571c1199f/toolkit/crashreporter/CrashAnnotations.yaml#1) but missing from Socorro (with an additional five being added in Bug 2049749 which is in review), and about half of them don't have an explicit data review and approval by a data steward. Those that do limit access/visiblity to Firefox telemetry (visible to all MoCo but not the general public in BigQuery etc.)[1]. In our case, making a field public in Socorro means making it public to the world on our website. Because of this, we should really add these fields carefully with a new data review, in many cases on a field-by-field basis. This, among other reasons, is why we typically handle adding a single new field to Socorro as its own separate ticket. Additionally, by definition, these are fields that are not in Socorro. It's therefore guaranteed that no one is querying them in Crash Stats today, and we will not break any existing workflows by deferring adding them to follow-up work as needed/requested. As with all fields Socorro doesn't know about, they will be viewable in the raw crash document only and classified as "protected" until then. Given this, and in discussion with Sven and Alex Franchuk, I have signficantly reduced the scope of this ticket to cover the absolute minimum changes needed for this project: * Ensure that critical path consumers can query existing Crash Report fields that are currently derived from a minidump (Crash Pings won't have a minidump to derive the fields from) * These are fields that I previously identified in [Bug 2055875 Comment 10](https://bugzilla.mozilla.org/show_bug.cgi?id=2055875#c10) that are being added as new crash annotations in Bug 2049749). * I've just confirmed the exact fields with Alex: `CPUArchitecture`, `CPUInfo`, `OS`, `OSVersion`, `LinuxLSBDescription` * Ensure we have the information we need to handle Firefox telemetry data deletion requests. Revised acceptance criteria: * `CPUArchitecture`, `CPUInfo`, `OS`, `OSVersion`, `LinuxLSBDescription` are added to the raw crash schema * Existing processor rules to determine the processed crash equivalents of these fields (`cpu_arch`, `cpu_info`, `os_name`, `os_version` and `os_pretty_version`, respectively) need to be updated to use the derived value from the minidump and fallback to the value provided in the crash annotation. * The processed crash schema description for these fields should be updated to mention that we conditionally use the provided source annotation (or add a comment if possible to that effect). * Copy and index the Glean client ID as a protected field into the processed crash report for handling data deletion requests as mentioned in [Comment 2](https://bugzilla.mozilla.org/show_bug.cgi?id=2052944#c2). * Unit tests added or updated for the schema and processor changes. [1]: The [crash-pings.mozilla.org](https://crash-pings.mozilla.org/) website does surface some crash ping data publicly, but it doesn't provide a comprehensive view of each crash ping and is quite limited in terms of search/filtering functionality relative to Socorro, in addition to being a much newer website that many fewer people know about.
I've spent a good amount of time thinking about this, and to completely do this, it represents about 23 (or more) tickets worth of work, since there are 23 of 58 [`scope: "ping"` crash annotations present in Crash Pings today](https://searchfox.org/firefox-main/rev/ee8e457ae20efb27313476fed0bed50571c1199f/toolkit/crashreporter/CrashAnnotations.yaml#1) but missing from Socorro (with an additional five being added in Bug 2049749 which is in review), and about half of them don't have an explicit data review and approval by a data steward. Those that do limit access/visibility to Firefox telemetry (visible to all MoCo but not the general public in BigQuery etc.)[1]. In our case, making a field public in Socorro means making it public to the world on our website. Because of this, we should really add these fields carefully with a new data review, in many cases on a field-by-field basis. This, among other reasons, is why we typically handle adding a single new field to Socorro as its own separate ticket. Additionally, by definition, these are fields that are not in Socorro. It's therefore guaranteed that no one is querying them in Crash Stats today, and we will not break any existing workflows by deferring adding them to follow-up work as needed/requested. As with all fields Socorro doesn't know about, they will be viewable in the raw crash document only and classified as "protected" until then. Given this, and in discussion with Sven and Alex Franchuk, I have significantly reduced the scope of this ticket to cover the absolute minimum changes needed for this project: * Ensure that critical path consumers can query existing Crash Report fields that are currently derived from a minidump (Crash Pings won't have a minidump to derive the fields from) * These are fields that I previously identified in [Bug 2055875 Comment 10](https://bugzilla.mozilla.org/show_bug.cgi?id=2055875#c10) that are being added as new crash annotations in Bug 2049749). * I've just confirmed the exact fields with Alex: `CPUArchitecture`, `CPUInfo`, `OS`, `OSVersion`, `LinuxLSBDescription` * Ensure we have the information we need to handle Firefox telemetry data deletion requests. Revised acceptance criteria: * `CPUArchitecture`, `CPUInfo`, `OS`, `OSVersion`, `LinuxLSBDescription` are added to the raw crash schema * Existing processor rules to determine the processed crash equivalents of these fields (`cpu_arch`, `cpu_info`, `os_name`, `os_version` and `os_pretty_version`, respectively) need to be updated to use the derived value from the minidump and fallback to the value provided in the crash annotation. * The processed crash schema description for these fields should be updated to mention that we conditionally use the provided source annotation (or add a comment if possible to that effect). * Copy and index the Glean client ID as a protected field into the processed crash report for handling data deletion requests as mentioned in [Comment 2](https://bugzilla.mozilla.org/show_bug.cgi?id=2052944#c2). * Unit tests added or updated for the schema and processor changes. [1]: The [crash-pings.mozilla.org](https://crash-pings.mozilla.org/) website does surface some crash ping data publicly, but it doesn't provide a comprehensive view of each crash ping and is quite limited in terms of search/filtering functionality relative to Socorro, in addition to being a much newer website that many fewer people know about.