Closed Bug 1764395 Opened 10 months ago Closed 5 months ago

schema for processed crash

Categories

(Socorro :: General, task, P2)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: willkg, Assigned: willkg)

References

(Blocks 4 open bugs)

Details

Attachments

(8 files, 1 obsolete file)

53 bytes, text/x-github-pull-request
Details | Review
53 bytes, text/x-github-pull-request
Details | Review
53 bytes, text/x-github-pull-request
Details | Review
53 bytes, text/x-github-pull-request
Details | Review
53 bytes, text/x-github-pull-request
Details | Review
53 bytes, text/x-github-pull-request
Details | Review
53 bytes, text/x-github-pull-request
Details | Review
53 bytes, text/x-github-pull-request
Details | Review

Bug #1753521 covered fixing Elasticsearch indexing to only index data from the processed crash. This simplifies indexing because the processed crash contains data that has been normalized and validated.

Bug #1755095 covered fixing Telemetry crashstorage to build a crash document solely from the processed crash. The processed crash data has been normalized and validated and it reduced the complexity of the Telemetry crashstorage code.

Bug #1762271 covered replacing the json schema reducer so that we can use schemas to encode metadata for other purposes and transforms.

All that work leads us to a point where we can build a processed crash schema.

The processed crash schema will let us centralize a few things:

  1. a better way to denote which fields to copy/normalize/validate from the raw crash for the CopyFromRawCrashRule
  2. access control for processed crash fields simplifying a bunch of access-control-like thing littered across the code base
  3. documentation for the fields including descriptions
  4. indexing and search instructions (this requires a lot more work to our Elasticsearch code)

This bug covers creating a processed crash schema that specifies structure and types for data as well as permissions that'll be used by the API, report view, and (hopefully) super search fields.

Assignee: nobody → willkg
Status: NEW → ASSIGNED
Blocks: 1755525
Depends on: 1753521, 1755095

I redid the summary and description to refocus this bug.

Depends on: 1783027

willkg merged PR #6142: "bug 1764395: processed crash schema" in 83aa2c6.

That covers a huge chunk of the work that needs to be done. The next step is to implement access control for processed crash fields simplifying a bunch of access-control-like thing littered across the code base. That involves:

  1. annotating the schema with permissions derived from the myriad of places we have permissions now
  2. writing a reducer for public vs. non-public fields
  3. fixing the APIs and the report, signature, and super search views to use the schema as the source of truth for permissions

Probably at least a few more days of work.

Depends on: 1697001
Attachment #9290156 - Attachment is obsolete: true

Current status:

I think I'm pretty close. I did a PR (comment #4) a few days ago that I thought finished the work. But while reviewing that, I found an issue and fixing that with the architecture I had wasn't going to work out. So I went back to the drawing board and changed how everything would work.

I'm mostly done the re-re-write now and I'm verifying it. Depending on how that goes, I might be able to land it this weekend. That unblocks being able to update rust-minidump and catching up with the inline function support.

Once we've got the inline function stuff done, I can go back to cleaning up after these changes. I'll probably do that in other bugs.

I need to test that extensively on stage and then let it sit for a bit and see what issues come up.

willkg merged PR #6150: "bug 1764395: remove java_exception_raw" in a8b407c.

This removes the whole java_exception_raw nonsense. Now we can adjust permissions of things using the schema even for fields in deep structures.

willkg merged PR #6153: "bug 1764395: return all data if user has view_pii for ProcessedCrash API" in 83d1f94.

That fixes the ProcessedCrash API to not reduce at all if you have view_pii permissions. That way you can get the processed crash data even if it's not covered in the schema. This comes up with stackwalker output changes.

This has been on stage for a while now and everything looks fine. I re-verified that the publicly-available data via the ProcessedCrash API on stage is fine. There are no new issues in Sentry. Grafana looks fine (modulo that the Elasticsearch crashtorage is now "escrashstorage" and some panels need to be updated). Ergo, this looks good to go.

I deployed this to prod just now in bug #1787851. Everything looks good. There's a bunch of additional things we could/should do now. I'll spin those off into separate bugs. Marking as FIXED.

Status: ASSIGNED → RESOLVED
Closed: 5 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.