I wrote a proposal and shopped it around. One of the ideas behind it was to conform to existing conventions. Glean has this schema for it's payload: https://github.com/mozilla-services/mozilla-pipeline-schemas/blob/main/schemas/glean/glean/glean.1.schema.json Looking at `org_mozilla_firefox.baseline` as an example, the items in the Glean ping are at the top level, there's a "metadata", and some other things added at ingestion. Data engineers don't care where things come from in the way I thought they might. Given that, I think I want to keep the raw crash mostly as is, except that I do want to move all the collector-added bits except for "submitted_timestamp" and "uuid" into a "metadata" section. So we'd end up with something like this (as an example): ``` { "BuildID": "20220902183841", "Comments": "willkg crash test with inline functions 20220902_1544", "CrashTime": "1662147837", "InstallTime": "1662147786", "MozCrashReason": "MOZ_CRASH(Crash via about:crashparent)", "Notes": "Ubuntu 22.04.1 LTSFP(D00-L1000-W00000000-T010) WR? WR+ EGL? EGL- GL Context? GL Context+ ", "ProductID": "{ec8030f7-c20a-464f-9b0e-13a3a9e97384}", "ProductName": "Firefox", "ReleaseChannel": "default", "StartupTime": "1662147817", "SubmittedFrom": "Client", "useragent_locale": "en-US", "metadata": { "collector_notes": [], "dump_checksums": { "upload_file_minidump": "639febfeaca3579d58f098a5fa5e8d6276323cf88a8026060be920691dd612f0" }, "MinidumpSha256Hash": "639febfeaca3579d58f098a5fa5e8d6276323cf88a8026060be920691dd612f0", "payload": "json", "payload_compressed": "0", }, "submitted_timestamp": "2022-09-02T19:44:32.762212+00:00", "uuid": "20aec939-4154-4520-ba3c-024930220902", "version": 2 } ``` Going to work on adding support for that now. 1. update socorro to support new structure; add migration code in crashstorage to convert old structure to new structure 2. update socorro-submitter to support new and old structure 3. update antenna to switch to new structure 4. write up bug for socorro and socorro-submitter to remove handling for old structure to be done in 6 months
Bug 1687987 Comment 10 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
I wrote a proposal and shopped it around. One of the ideas behind it was to conform to existing conventions. Glean has this schema for it's payload: https://github.com/mozilla-services/mozilla-pipeline-schemas/blob/main/schemas/glean/glean/glean.1.schema.json Looking at `org_mozilla_firefox.baseline` as an example, the items in the Glean ping are at the top level, there's a "metadata", and some other things added at ingestion. Data engineers don't care where things come from in the way I thought they might. Given that, I think I want to keep the raw crash mostly as is, except that I do want to move all the collector-added bits except for "submitted_timestamp" and "uuid" into a "metadata" section. So we'd end up with something like this (as an example): ``` { "BuildID": "20220902183841", "Comments": "willkg crash test with inline functions 20220902_1544", "CrashTime": "1662147837", "InstallTime": "1662147786", "MozCrashReason": "MOZ_CRASH(Crash via about:crashparent)", "Notes": "Ubuntu 22.04.1 LTSFP(D00-L1000-W00000000-T010) WR? WR+ EGL? EGL- GL Context? GL Context+ ", "ProductID": "{ec8030f7-c20a-464f-9b0e-13a3a9e97384}", "ProductName": "Firefox", "ReleaseChannel": "default", "StartupTime": "1662147817", "SubmittedFrom": "Client", "useragent_locale": "en-US", "metadata": { "collector_notes": [], "dump_checksums": { "upload_file_minidump": "639febfeaca3579d58f098a5fa5e8d6276323cf88a8026060be920691dd612f0" }, "minidump_sha256_hash": "639febfeaca3579d58f098a5fa5e8d6276323cf88a8026060be920691dd612f0", "payload": "json", "payload_compressed": "0", }, "submitted_timestamp": "2022-09-02T19:44:32.762212+00:00", "uuid": "20aec939-4154-4520-ba3c-024930220902", "version": 2 } ``` Going to work on adding support for that now. 1. update socorro to support new structure; add migration code in crashstorage to convert old structure to new structure 2. update socorro-submitter to support new and old structure 3. update antenna to switch to new structure 4. write up bug for socorro and socorro-submitter to remove handling for old structure to be done in 6 months
I wrote a proposal and shopped it around. One of the ideas behind it was to conform to existing conventions. Glean has this schema for it's payload: https://github.com/mozilla-services/mozilla-pipeline-schemas/blob/main/schemas/glean/glean/glean.1.schema.json Looking at `org_mozilla_firefox.baseline` as an example, the items in the Glean ping are at the top level, there's a "metadata", and some other things added at ingestion. Data engineers don't care where things come from in the way I thought they might. Given that, I think I want to keep the raw crash mostly as is, except that I do want to: 1. move `collector_notes`, `dump_checksums`, `payload`, and `payload_compressed` into a `metadata` section 2. remove `MinidumpSha256Hash` entirely--it's in `dump_checksums` and the processor can pull it out from there 3. add a `version` field indicating the raw crash structure version in case we ever want to change it in a backwards-incompatible way in the future So we'd end up with something like this (as an example): ``` { "BuildID": "20220902183841", "Comments": "willkg crash test with inline functions 20220902_1544", "CrashTime": "1662147837", "InstallTime": "1662147786", "MozCrashReason": "MOZ_CRASH(Crash via about:crashparent)", "Notes": "Ubuntu 22.04.1 LTSFP(D00-L1000-W00000000-T010) WR? WR+ EGL? EGL- GL Context? GL Context+ ", "ProductID": "{ec8030f7-c20a-464f-9b0e-13a3a9e97384}", "ProductName": "Firefox", "ReleaseChannel": "default", "StartupTime": "1662147817", "SubmittedFrom": "Client", "useragent_locale": "en-US", "metadata": { "collector_notes": [], "dump_checksums": { "upload_file_minidump": "639febfeaca3579d58f098a5fa5e8d6276323cf88a8026060be920691dd612f0" }, "payload": "json", "payload_compressed": "0", }, "submitted_timestamp": "2022-09-02T19:44:32.762212+00:00", "uuid": "20aec939-4154-4520-ba3c-024930220902", "version": 2 } ``` Going to work on adding support for that now. 1. update socorro to support new structure; add migration code in crashstorage to convert old structure to new structure 2. update socorro-submitter to support new and old structure 3. update antenna to switch to new structure 4. write up bug for socorro and socorro-submitter to remove handling for old structure to be done in 6 months