Open Bug 1636210 Opened 1 year ago Updated 28 days ago

[tracker] rewrite Tecken symbolication v4/v5 with symbolic (Eliot)

Categories

(Tecken :: Symbolication, enhancement, P2)

enhancement

Tracking

(Not tracked)

ASSIGNED

People

(Reporter: willkg, Assigned: willkg)

References

(Depends on 8 open bugs, Blocks 5 open bugs)

Details

Attachments

(2 files)

Bug #1621638 covered researching whether it was possible to switch to the Sentry Symbolic crate and what issues need to get figured out to do that. That bug is done.

I want to switch to the Symbolic crate. This bug covers figuring out the details for switching and the actual switching.

I want to create a second symbolication API v5 view. Adding a "symbolic=true" or something like that to the querystring will switch to the new Symbolic-based view. Then we'll write the new v5 view code and have them side-by-side for correctness and performance testing. Then we'll remove the old v5 code.

Next steps:

  1. figure out appropriate disk cache size
  2. figure out how to manage the disk cache
  3. talk to Brian about how this will work on servers

Talked with Brian about this and bug #1644583. We have a new plan for symbolication for Tecken.

In bug #1621683, I threw together a prototype to validate the option. We're going to base a rewrite of the symbolication v4/v5 APIs in Tecken on that prototype.

Symbolication will run as a separate service from the upload/download service. This fixes the problem we potentially have where bursts of symbolication can adversely affect symbol uploads.

It will be in the Tecken repo to make it easier to maintain and deploy. We can use the existing deploy infrastructure.

The urls will continue to be the same. The somethingsomething (ELB?) will route /symbolicate requests to the new service.

That's the rough plan. It might change as needs warrant. Next step is to write up a project plan.

Summary: switch to symbolic for symbolication API v5 → rewrite Tecken symbolication v4/v5 with symbolic
Summary: rewrite Tecken symbolication v4/v5 with symbolic → [tracker] rewrite Tecken symbolication v4/v5 with symbolic
Depends on: 1672271

Status: I've gotten a lot of the new service written. I have to write a diskcache manager and figure out how that'll run. I have some more tests to write and run to compare prod with the new service and suss out any other differences.

However, I hit what I think is a bug in symbolic where it fails lookups for FUNCs with no line records.

https://github.com/getsentry/symbolic/issues/284

I'll follow up on that.

Blocks: 1672271
No longer depends on: 1672271

I hit another bug in symbolic where it fails the lookup if the address doesn't match the line record address.

https://github.com/getsentry/symbolic/issues/285

Depends on: 1673381
Depends on: 1673383

Things to think about with v2:

  1. Restrict max number of frames/stacks/jobs to reduce DOS likelihood. I'm deferring this until I have metrics on sizes first.
  2. Think about adding an in-memory cache for symcache files. Maybe restrict this to system libraries which are small and might have a higher hit rate? I'm deferring this until I know more about how it performs in prod.
Depends on: 1673887
Depends on: 1673910
Depends on: 1674102
Depends on: 1674105
Depends on: 1674107
Summary: [tracker] rewrite Tecken symbolication v4/v5 with symbolic → [tracker] rewrite Tecken symbolication v4/v5 with symbolic (Eliot)
Depends on: 1674402
Depends on: 1674406
Depends on: 1676401
Depends on: 1677570
Depends on: 1681632
Depends on: 1696585
Depends on: 1699546
Depends on: 1699774
Depends on: 1699776
Depends on: 1701653
Depends on: 1701978
Depends on: 1702012
Depends on: 1702048
Depends on: 1704523
You need to log in before you can comment on or make changes to this bug.