1739447 - migrate symbolication API use from symbols to symbolication service

Status: RESOLVED FIXED

(Tecken :: Symbolication, task, P2)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

We stood up a new symbolication API (bug #1636210). Now that we've got it stood up, it's time to migrate everyone from using the old one (symbols.mozilla.org/symbolication/v5) to the new one (symbolication.services.mozilla.com/symbolication/v5).

We have some steps here:

1. communicate that the old one is deprecated and users should switch; we might be able to identify some of them and write up issues for their code
2. shunt/proxy the traffic from the old server to the new one
3. stop supporting symbolication API traffic with the old server

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Jira issue for one possible way to do step 2: https://mozilla-hub.atlassian.net/browse/DSRE-220

Another idea I had was to truncate the symbolication API views in the Tecken webapp and have them proxy the request. So then users can continue to use it, but it just turns around and does the request with the symbolication API.

We don't want to maintain that forever.

Maybe it adds an additional "please stop using this--use this other thing instead" field?

Maybe it periodically emits an HTTP 308 permanent redirect?

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Jason suggested using nginx to proxy the request with proxy_pass. We'll set a proxy_read_timeout and also a proxy_set_header.

proxy_read_timeout will kill proxy connections that are taking too long. The client will probably get back an HTTP 504.

proxy_set_header TeckenProxied 1 (or something equivalent) will allow us to track how many requests are getting proxied so we can work that down to 0. I'll need to add support for whatever header we end up with to Eliot.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Jason landed a change that's on stage. I'll run tests on it and then maybe we'll do a prod deploy on Monday.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

The plan is to deploy on Tuesday and push the proxy configuration out.

I sent an email to stability and crash-reporting-wg working groups with this in it:

I'm hereby deprecating the old symbols.mozilla.org symbolication API endpoints in favor of their symbolication.services.mozilla.com counterparts.

Old service -> New service

• https://symbols.mozilla.org/symbolicate/v4 -> https://symbolication.services.mozilla.com/symbolicate/v4
• https://symbols.mozilla.org/symbolicate/v5 -> https://symbolication.services.mozilla.com/symbolicate/v5

Starting Tuesday, we're going to tweak nginx for symbols.mozilla.org to proxy all incoming symbolication API requests [1] to symbolication.services.mozilla.com.

While you're updating code and services to use the new service endpoint, I'd appreciate if you also did the following:

1. Upgrade from the v4 to v5 versions of the API. The v4 version has been deprecated for a while and I'd like to remove it.
2. Add a User-Agent HTTP header to your symbolication API requests. One of the problems I'm having with deprecating things is it's impossible to know what's using it.

The symbolication API documentation has been updated:

https://tecken.readthedocs.io/en/latest/symbolication.html

One thing to notice is that the v5 API returns line numbers and filenames for symbols.

If you have any questions, comments, or concerns, please reach out to me or ask on the #crashreporting channel on Matrix.

Note that this only affects the symbolication API. The uploads and downloads APIs will continue to be at symbols.mozilla.org.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 2448: bug 1739447: emit incr when TeckenProxied header exists and is 1

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR #2448: "bug 1739447: emit incr when TeckenProxied header exists and is 1" in 9ff5c63.

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

This was deployed in bug #1740308 today. Everything looks spiffy. Marking as FIXED.