929647 - web form throttling

Status: RESOLVED FIXED

(Input Graveyard :: Submission, defect, P1)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

We currently throttle incoming responses via the Input API. We should additionally throttle incoming responses via the web form.

This bug covers figuring out all the details:

1. how to implement throttling

2. what levels to throttle at

3. what happens when a user who is entering in feedback triggers the throttle (e.g. at a conference where other people have issued feedback)

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

I'm pretty sure this is a non-trivial bug that will take about a week to implement. It may also involve strings changes depending on how we want to deal with the ui portion. Adding whiteboard data accordingly.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

I just talked to Mike about this since he implemented throttling in SUMO.

SUMO uses django-ratelimit <https://github.com/jsocol/django-ratelimit> which (as you can see) is written by James Socol. That seems to do everything I'd want it to do, plus it's probably easier to fix issues if there are any. I'm doing a pass on the project, issues and pull requests to see how it's faring right now. Then I'll use that.

I don't know what to do about levels to throttle at. I suspect we can crib from SUMO.

I need to make sure we're logging how often someone gets throttled so we can keep track of whether we're hosing more people than we want to be. Just like with CSRF changes we made a while back.

For now, I think I'm just going to ignore the case when someone gets throttled and do nothing. We can figure out later if that's ok. If it's not, then we have some non-trivial work to do to make the feedback form save state and be cleverer about things.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Action plan:

1. use django-ratelimit (waiting on some fixes to land)
2. throttle with same levels as the input api (100/hour)
3. add some statsd code to keep track of how many throttlings we're doing
4. when someone hits the throttle threshold, just redirect to the thank you page

Note: Write up a bug to fix that last bit when we redo the feedback form.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

Just waiting on a django-ratelimit 0.4 release.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

I got tired of waiting.

In a PR: https://github.com/mozilla/fjord/pull/165

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

Landed in master in https://github.com/mozilla/fjord/commit/c426735

Pushed to stage and prod.

However, I don't see anything Input-related on graphite, so I opened bug #933280 to figure out what's going on there.

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

Something is wrong with the throttling code and it caused a massive drop off in responses. I deployed an earlier commit to production. The commit is still in the master branch (I didn't reverse it).

If I had to hazard a guess, it's throttling on the ip address of the load balancer and not the ip address of the client. Reopening this to fix that.

Comment 8

[Will Kahn-Greene [:willkg] ET needinfo? me]

Bug #933712 covers the throttling problem.

Comment 9

[Will Kahn-Greene [:willkg] ET needinfo? me]

Fix in PR https://github.com/mozilla/fjord/pull/166

I'm blocking on bug #933280 because I need to be able to see what's going on on the servers before I try another attempt pushing this.

Comment 10

[Will Kahn-Greene [:willkg] ET needinfo? me]

Fixes landed in master in https://github.com/mozilla/fjord/commit/7a260af

I'm going to push this to stage and prod. statsd/graphite is capturing and showing data again, so I'll keep an eye on it over it for the rest of the day and if things look like they're going south, I'll undo it again.

Comment 11

[Will Kahn-Greene [:willkg] ET needinfo? me]

Pushed to stage and prod just now. I'll monitor this for a day and if after that it looks good, I'll mark it FIXED.

Comment 12

[Will Kahn-Greene [:willkg] ET needinfo? me]

Everything continues to look good. We haven't had any throttling kick in, yet. I don't expect any throttling to happen except in egregious circumstances, so that's expected.

Marking this as FIXED.