1636210 - [tracker] rewrite Tecken symbolication v4/v5 with symbolic (Eliot)

Status: RESOLVED FIXED

(Eliot :: General, enhancement, P2)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

Bug #1621638 covered researching whether it was possible to switch to the Sentry Symbolic crate and what issues need to get figured out to do that. That bug is done.

I want to switch to the Symbolic crate. This bug covers figuring out the details for switching and the actual switching.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

I want to create a second symbolication API v5 view. Adding a "symbolic=true" or something like that to the querystring will switch to the new Symbolic-based view. Then we'll write the new v5 view code and have them side-by-side for correctness and performance testing. Then we'll remove the old v5 code.

Next steps:

1. figure out appropriate disk cache size
2. figure out how to manage the disk cache
3. talk to Brian about how this will work on servers

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Talked with Brian about this and bug #1644583. We have a new plan for symbolication for Tecken.

In bug #1621683, I threw together a prototype to validate the option. We're going to base a rewrite of the symbolication v4/v5 APIs in Tecken on that prototype.

Symbolication will run as a separate service from the upload/download service. This fixes the problem we potentially have where bursts of symbolication can adversely affect symbol uploads.

It will be in the Tecken repo to make it easier to maintain and deploy. We can use the existing deploy infrastructure.

The urls will continue to be the same. The somethingsomething (ELB?) will route /symbolicate requests to the new service.

That's the rough plan. It might change as needs warrant. Next step is to write up a project plan.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Status: I've gotten a lot of the new service written. I have to write a diskcache manager and figure out how that'll run. I have some more tests to write and run to compare prod with the new service and suss out any other differences.

However, I hit what I think is a bug in symbolic where it fails lookups for FUNCs with no line records.

https://github.com/getsentry/symbolic/issues/284

I'll follow up on that.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

I hit another bug in symbolic where it fails the lookup if the address doesn't match the line record address.

https://github.com/getsentry/symbolic/issues/285

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

Things to think about with v2:

1. Restrict max number of frames/stacks/jobs to reduce DOS likelihood. I'm deferring this until I have metrics on sizes first.
2. Think about adding an in-memory cache for symcache files. Maybe restrict this to system libraries which are small and might have a higher hit rate? I'm deferring this until I know more about how it performs in prod.

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 2255: bug 1636210: implement eliot microservice for symbolication

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR #2255: "bug 1636210: implement eliot microservice for symbolication" in b1de831.

Comment 8

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 2263: bug 1636210: improve Eliot docs

Comment 9

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR #2263: "bug 1636210: improve Eliot docs" in 581e3ad.

Comment 10

[Will Kahn-Greene [:willkg] ET needinfo? me]

Google doc for the proposal and project plan: https://docs.google.com/document/d/1U6FBFh9FsEWvUXf7DFzCBV4eCEjaBoxJbs9vtm6Lz6c/edit

Comment 11

[Will Kahn-Greene [:willkg] ET needinfo? me]

Jira issue for setting up Eliot in prod: https://mozilla-hub.atlassian.net/browse/DSRE-98

Comment 12

[Will Kahn-Greene [:willkg] ET needinfo? me]

Eliot is live in production:

https://symbolication.services.mozilla.com/

We still have work to do to migrate users to the new service, but that's covered in other bugs that depend on this one.

Given that, we're done! Marking as FIXED.

Comment 13

[Will Kahn-Greene [:willkg] ET needinfo? me]

Moving to Eliot product.