schema for processed crash
Categories
(Socorro :: General, task, P2)
Tracking
(Not tracked)
People
(Reporter: willkg, Assigned: willkg)
References
(Blocks 4 open bugs)
Details
Attachments
(8 files, 1 obsolete file)
|
53 bytes,
text/x-github-pull-request
|
Details | Review | |
|
53 bytes,
text/x-github-pull-request
|
Details | Review | |
|
53 bytes,
text/x-github-pull-request
|
Details | Review | |
|
53 bytes,
text/x-github-pull-request
|
Details | Review | |
|
53 bytes,
text/x-github-pull-request
|
Details | Review | |
|
53 bytes,
text/x-github-pull-request
|
Details | Review | |
|
53 bytes,
text/x-github-pull-request
|
Details | Review | |
|
53 bytes,
text/x-github-pull-request
|
Details | Review |
Bug #1753521 covered fixing Elasticsearch indexing to only index data from the processed crash. This simplifies indexing because the processed crash contains data that has been normalized and validated.
Bug #1755095 covered fixing Telemetry crashstorage to build a crash document solely from the processed crash. The processed crash data has been normalized and validated and it reduced the complexity of the Telemetry crashstorage code.
Bug #1762271 covered replacing the json schema reducer so that we can use schemas to encode metadata for other purposes and transforms.
All that work leads us to a point where we can build a processed crash schema.
The processed crash schema will let us centralize a few things:
- a better way to denote which fields to copy/normalize/validate from the raw crash for the
CopyFromRawCrashRule - access control for processed crash fields simplifying a bunch of access-control-like thing littered across the code base
- documentation for the fields including descriptions
- indexing and search instructions (this requires a lot more work to our Elasticsearch code)
This bug covers creating a processed crash schema that specifies structure and types for data as well as permissions that'll be used by the API, report view, and (hopefully) super search fields.
|
Assignee | |
Updated•2 years ago
|
|
|
Assignee | |
Updated•2 years ago
|
|
|
Assignee | |
Comment 1•2 years ago
|
||
I redid the summary and description to refocus this bug.
|
|
Assignee | |
Comment 2•2 years ago
|
||
|
|
Assignee | |
Comment 3•2 years ago
|
||
willkg merged PR #6142: "bug 1764395: processed crash schema" in 83aa2c6.
That covers a huge chunk of the work that needs to be done. The next step is to implement access control for processed crash fields simplifying a bunch of access-control-like thing littered across the code base. That involves:
- annotating the schema with permissions derived from the myriad of places we have permissions now
- writing a reducer for public vs. non-public fields
- fixing the APIs and the report, signature, and super search views to use the schema as the source of truth for permissions
Probably at least a few more days of work.
|
|
Assignee | |
Comment 4•2 years ago
|
||
|
|
Assignee | |
Comment 5•2 years ago
|
||
|
|
Assignee | |
Updated•2 years ago
|
|
|
Assignee | |
Comment 6•2 years ago
|
||
|
|
Assignee | |
Comment 7•2 years ago
|
||
Current status:
I think I'm pretty close. I did a PR (comment #4) a few days ago that I thought finished the work. But while reviewing that, I found an issue and fixing that with the architecture I had wasn't going to work out. So I went back to the drawing board and changed how everything would work.
I'm mostly done the re-re-write now and I'm verifying it. Depending on how that goes, I might be able to land it this weekend. That unblocks being able to update rust-minidump and catching up with the inline function support.
Once we've got the inline function stuff done, I can go back to cleaning up after these changes. I'll probably do that in other bugs.
|
|
Assignee | |
Comment 8•2 years ago
|
||
|
|
Assignee | |
Comment 9•2 years ago
|
||
|
|
Assignee | |
Comment 10•2 years ago
|
||
I need to test that extensively on stage and then let it sit for a bit and see what issues come up.
|
|
Assignee | |
Comment 11•2 years ago
|
||
|
|
Assignee | |
Comment 12•2 years ago
|
||
|
|
Assignee | |
Comment 13•2 years ago
|
||
|
|
Assignee | |
Comment 14•2 years ago
|
||
willkg merged PR #6150: "bug 1764395: remove java_exception_raw" in a8b407c.
This removes the whole java_exception_raw nonsense. Now we can adjust permissions of things using the schema even for fields in deep structures.
|
|
Assignee | |
Comment 15•2 years ago
|
||
|
|
Assignee | |
Comment 16•2 years ago
|
||
|
|
Assignee | |
Comment 17•2 years ago
|
||
|
|
Assignee | |
Comment 18•2 years ago
|
||
|
|
Assignee | |
Comment 19•2 years ago
|
||
|
|
Assignee | |
Comment 20•2 years ago
|
||
willkg merged PR #6153: "bug 1764395: return all data if user has view_pii for ProcessedCrash API" in 83d1f94.
That fixes the ProcessedCrash API to not reduce at all if you have view_pii permissions. That way you can get the processed crash data even if it's not covered in the schema. This comes up with stackwalker output changes.
|
|
Assignee | |
Comment 21•2 years ago
|
||
This has been on stage for a while now and everything looks fine. I re-verified that the publicly-available data via the ProcessedCrash API on stage is fine. There are no new issues in Sentry. Grafana looks fine (modulo that the Elasticsearch crashtorage is now "escrashstorage" and some panels need to be updated). Ergo, this looks good to go.
I deployed this to prod just now in bug #1787851. Everything looks good. There's a bunch of additional things we could/should do now. I'll spin those off into separate bugs. Marking as FIXED.
Description
•