1764569 - implement sensitive data scrubbing for sentry (tecken)

Status: RESOLVED FIXED

(Tecken :: General, task, P2)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

We updated to sentry-sdk. Now that we're using that, we need to make sure we're not sending PII along with Sentry error reports.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Grabbing this because it's blocking our migration.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 2568: bug 1764569: implement sentry scrubbing

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

I'm in the process of extracting the scrubbing code into a library. Then I'll redo pr 2568 using the library.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 2572: bug 1764569: implement sentry scrubbing

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR #2572: "bug 1764569: implement sentry scrubbing" in 2d4e8fe.

I'll spend some time breaking Tecken in stage to kick up some errors so I can see what's ending up in Sentry and then I can adjust things accordingly.

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

Also, I need to set up graphs for scrub errors in the dashboard.

• tecken webapp: tecken.sentry_scrub_error
• eliot: eliot.sentry_scrub_error with tag service as either cachemanager or webapp

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

I added a dashboard to grafana. It has no data, though, so it's hard to know whether it's set up correctly. But it's probably right.

I pushed this out in bug #1780235. Marking as FIXED.

Comment 8

[Will Kahn-Greene [:willkg] ET needinfo? me]

I updated Tecken to sentry_sdk 1.6.0 which adjusts the Sentry event schema. The Mozilla-hosted Sentry is behind a bit, though, so when Mozilla-hosted Sentry gets the Sentry event, it kicks up an error while ingesting:

transaction_info: Discarded unknown attribute

Triggering an exception in the Tecken webapp worked fine. The relevant bits are scrubbed and the data looks good.

Triggering an exception in the Eliot webapp worked fine. Eliot is using Falcon and it looks like there's no "request" section generated by the FalconIntegration. That makes sense because (if I recall correctly) the Falcon integration is broken in the sentry-sdk. If we ever want better request information (headers, query_string, data, etc), we should probably write our own integration.

The Eliot disk cache manager doesn't have a way to trigger an exception, but given the other things are working and we have integration tests for all three services, I'm confident it's probably working, too.

Comment 9

[Will Kahn-Greene [:willkg] ET needinfo? me]

I'm reopening this to add some additional things to scrub for Tecken webapp and also to fix the Eliot Sentry set up to use the WSGI middleware and update the rules per the Socorro/Tecken Sentry migration plan.

Comment 10

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 2584: bug 1764569: fix minor sentry event scrubbing issues

Comment 11

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR #2584: "bug 1764569: fix minor sentry event scrubbing issues" in cd617f6.

Comment 12

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 2585: bug 1764569: change error capture to be explicit

Comment 13

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR #2585: "bug 1764569: change error capture to be explicit" in e2938a3.

Comment 14

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 2586: bug 1764569: fix transaction name for Eliot Sentry events

Comment 15

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR #2586: "bug 1764569: fix transaction name for Eliot Sentry events" in 86a1305.

Comment 16

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 2587: bug 1764569: minor changes to sentry code

Comment 17

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR #2587: "bug 1764569: minor changes to sentry code" in 960e983.

Comment 18

[Will Kahn-Greene [:willkg] ET needinfo? me]

I re-verified the Sentry events for the Tecken webapp and the Eliot webapp on stage. It looks good--things are scrubbed the way they should be, there's no additional things we need to scrub, and the data shows up in the Sentry interface in a way that we can triage and use.

Comment 19

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 2588: bug 1764569: update fillmore to 0.1.1

Comment 20

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR #2588: "bug 1764569: update fillmore to 0.1.1" in 2d9e642.

Comment 21

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 2594: bug 1764569: update to fillmore 0.1.2; fix fillmore logging

Comment 22

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR #2594: "bug 1764569: update to fillmore 0.1.2; fix fillmore logging" in d455fb3.

Comment 23

[Will Kahn-Greene [:willkg] ET needinfo? me]

I deployed this to prod in bug #1782727. Marking as FIXED.