967496 - Snippets throughput halved for 20 minutes, timeouts on New Relic

Status: RESOLVED INVALID

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Osmose [:osmose, :mkelly]]

https://rpm.newrelic.com/accounts/263620/incidents/7317727

Snippets appears to intermittently be failing in some way. The event above shows that our throughput was halved for 20 minutes, and the health pings sent by New Relic timed out after 30 seconds.

I've received at least two other events like this over the past two weeks. There's no tracebacks available, and the rest of the stats (error rate, appdex, app server response time) all seem to be fine, only the throughput is affected: 

https://rpm.newrelic.com/accounts/263620/applications/2904874?tw[end]=1391510700&tw[start]=1391507100

Any ideas?

Comment 1

[Osmose [:osmose, :mkelly]]

Just noticed bug 967415, seems to fit the timeframe, although I'm curious why throughput went down but didn't get cut off completely.

https://rpm.newrelic.com/accounts/263620/incidents/7286036 is an example of a similar event from earlier this week, but looking at it, it seems like all of our metrics went bad, unlike the one above, in which only throughput went bad.

Comment 2

[Osmose [:osmose, :mkelly]]

Looking at the reports for the second error more closely, it looks like the DB was down/slow for a bit. Given that these issues are separate, I'm going to call this invalid. I'll reopen if we get more repeated alerts with the same cause. Sorry for the trouble!