933712 - rapid drop in feedback

Status: RESOLVED FIXED

(Input Graveyard :: Submission, defect, P1)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

We have a huge gap in Firefox Desktop input today... can you investigate?

select extract(DAY_HOUR FROM created) as day, count(1) from feedback_response where created > '2013-10-31' and product LIKE 'Firefox' group by day order by day

Gives:
3100    38
3101    37
3102    45
3103    49
3104    32
3105    42
3106    50
3107    65
3108    40
3109    2
3110    3
3111    44
3112    0
3113    0
3114    0
3115    0
3116    0
3117    0
3118    0
3119    0
3120    0
3121    1

Looks like something broke at 8 AM, restored momentarily at 11 AM (although that may be spam) and is still broken.

We see something similar in mobile but firefox OS looks OK.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

This is a fire. Grabbing it, tossing in whiteboard data and working on it right now.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

I have bug #933280 open to fix whatever the problem where statsd data is not getting to graphite. Until that's fixed, I'm kind of flying blind.

However, I'm betting this is related to me landing throttling yesterday so I backed that out just now.

There are hourly dumps of the data, so in a few hours I should be able to see if backing that out "fixed" the problem. If so, I'll go look at the throttling code and try to figure out what's going on.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

I'm looking at the New Relic database insert numbers and they've returned to pre-throttling numbers, so I think we're ok now and the problem was almost certainly due to landing the throttling code yesterday.

Looking at the deploy logs, I think we lost the bulk of 22 hours of data. On a release week, no less. Total suckage. I'm really sorry about that. I think going forward, I'll declare a moratorium on any deployments during release weeks.

I'll reopen the web form throttling bug, but probably won't do anything on that until Monday.

Marking this FIXED mostly because there isn't a a resolution along the lines of "EPICFUCKUPREALLYSORRY".