1381809 - webapp on socorro -stage memoryerrors

Status: RESOLVED WORKSFORME

(Socorro :: Infra, task)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

We're seeing memory errors from one or more of the webapp nodes on socorro -stage:

https://sentry.prod.mozaws.net/operations/socorro-stage/issues/625718/

https://sentry.prod.mozaws.net/operations/socorro-stage/issues/625718/

I checked Datadog to see if we had any monitoring of memory on the webapp nodes, and I don't see any on either -stage or -prod. I'm guessing we don't have alerts, either.

Since -stage is undergoing an Elasticsearch migration, I figured it'd worth writing this up in case the memory error is related (new ES library memory usage different? new leak? different caching? something else?).

This bug covers looking into the memory situation and alleviating that.

Plus we should probably have monitoring of memory on webapp nodes for -stage and -prod. If we don't have monitoring, we'd never know if changes we're making affect memory usage.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

We talked about this earlier today and while we can't be certain, no one thinks this is related to the ES changes. Given that, I think we're just going to sit on this and if it persists and/or happens in -prod, then we'll work on it then.

If no one has touched this in a couple of months, we should close it out.

Comment 2

[Peter Bengtsson [:peterbe]]

Fly-by-comment here; I suspect that we have work to do in the webapp, that is related to MemoryError, to do with failing to store results of an ES query in memcached. Might be because we're now trying to store something much larger or weirder.

Comment 3

[Peter Bengtsson [:peterbe]]

Pretty tempted to resolve this as fixed. 

It's extremely likely that the memory errors come from trying to memcache a SuperSearch query, in the webapp, when SuperSearch was allowed to simply contain too much. A month ago (or so?) we added a piece of code that prevents massive enormous values to go into the ES processed crash fields. That should hopefully prevent this. Also, I don't believe we've seen this error in a loooong time now.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

I made a change that nixed huge keys, but I don't think we're truncating values, yet. We probably should be especially for fields marked as keyword fields.

Regardless of that clarification, I think Peter's right on and that this is probably not an issue any more.

I'm going to mark it WORKSFORME. We can always re-open it later if it comes up again.