1571432 - processor memory usage high

Status: RESOLVED FIXED

(Socorro :: Processor, task)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

Over the weekend, we got a series of alerts of high memory usages on the processor nodes. Last week, we pushed out 2019.08.02. Before that deploy, memory usage was stable and after that deploy, memory usage climbs rapidly over time for each node.

Processor node memory usage:

https://app.datadoghq.com/dashboard/kaj-sat-azh/socorro-prod-megamaster-remix?from_ts=1564406362545&live=true&tile_size=m&to_ts=1565011162545&fullscreen_widget=1283211765640647&fullscreen_section=overview

This bug covers figuring out what's going on and fixing it.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Making this a P1 to do now.

The deploy bug was bug #1570963.

John said taht sentry-sdk did a major refactoring. We use that differently between the webapp and processor. They have this in their issue tracker:

https://github.com/getsentry/sentry-python/issues/430

That sure seems related to what we're seeing.

I'm going to downgrade that now and see how that fares.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 4999: bug 1571432: downgrade sentry-sdk to 0.9.5

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR #4999: "bug 1571432: downgrade sentry-sdk to 0.9.5" in 94ffd34.

After that deploys to stage, I'll keep an eye on it. We should see evidence of "yes, that worked" or "no, that didn't" by tomorrow.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

Stage looks fine still.

We haven't seen any more memory alerts in prod, but the problem is still there. What's going on is that since this is during the week, there's a higher churn of processor nodes, so they're not sticking around long enough to run out of memory. That's not true on the weekend.

I'm going to push this out either tonight or tomorrow depending on timing.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

I pushed it out a couple of hours ago. prod looks good so far, but I'll wait until tomorrow to conclude whether downgrading sentry-sdk did it or not.

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

Downgrading sentry-sdk definitely fixed it.

https://app.datadoghq.com/dashboard/kaj-sat-azh/socorro-prod-megamaster-remix?from_ts=1565091154679&live=true&tile_size=m&to_ts=1565177554679&fullscreen_widget=1283211765640647&fullscreen_section=overview

Marking as FIXED.