1122658 - [spam] Automatically ban IPs based on request rate

Status: RESOLVED FIXED

(developer.mozilla.org Graveyard :: General, enhancement)

Description

[Justin Crawford [:hoosteeno] [:jcrawford]]

All of the below activity (as we elsewhere established) is coming from one IP. As you can see, it is sequential.

https://developer.mozilla.org/en-US/dashboards/revisions?user=LoTD
53 edits in 10 minutes starting 09:27 (in my TZ)

https://developer.mozilla.org/en-US/dashboards/revisions?user=Minat
57 edits in 10 minutes starting 09:04

https://developer.mozilla.org/en-US/dashboards/revisions?user=Penny
53 edits in 11 minutes starting 08:47

https://developer.mozilla.org/en-US/profiles/jensen
67 edits in 8 minutes starting 08:33

https://developer.mozilla.org/en-US/profiles/morello
58 edits in 6 minutes starting 07:56

https://developer.mozilla.org/en-US/profiles/jscape
84 edits in 32 minutes starting 07:06

https://developer.mozilla.org/en-US/profiles/shneeple
<50 edits in 5 minutes starting 06:55

https://developer.mozilla.org/en-US/profiles/fenster
<30 edits in 3 minutes starting 06:46

https://developer.mozilla.org/en-US/dashboards/revisions?user=joemix
<10 edits in 3 minutes starting 06:27

etc. (there are more)

I just opened the edit page and in so doing, made 4 requests against developer.mozilla.org. I saved and made 3 requests. So let's assume that each revision above causes ~7 requests on developer.mozilla.org. Let's see how that applies to the cases we know about. (Please note, I have not broken all of this activity down minute by minute: My estimates are based on averages, not true measures).  

The busiest user id (morello) made 9.67 revisions per minute on average during 6 minutes, or 67.69 requests per minute.

All of the user ids combined made maybe 450 revisions during 3 hours, or 2.5 revisions per minute on average, or 17.5 requests per minute on average (but of course, the IP also made 67.69 requests per minute while it was acting as morello).

Considering the general activity of this IP, throttling IPs at 70 requests/minute (7 revisions a minute) by IP would probably not have caught the IP. 

:cmills, who everyone will agree edits rapidly and prodigiously, made 80 edits on 2014/12/09. Between 2 and 3 a.m. (in my time zone) he made 26 edits; between 3 and 4, 15 edits, between 4 and 5, 17 edits, between 5 and 6, 3 edits, etc. It was a very busy day. But (on average) during his busiest hour he made just over 3 requests to developer.mozilla.org per minute.

:teoli points out that a doc sprint might happen in an office that uses a shared IP to connect to the internet. With a throttle rate of 70 requests per minute, it would take approximately 20 :cmillses working on such a sprint to trigger an IP ban that would halt the sprint until an administrator could be summoned.

We could mitigate that by...
* allowing admins to whitelist IPs so they won't be automatically banned
* making automated bans temporary with a short countdown (15 mins?)
* not allowing more than 10 :cmillses to congregate in one location at one time
* not banning IPs based on throttle rate (i.e. WONTFIXING this bug) and instead perhaps banning userids based on throttle rate (i.e. opening a different bug)

Comment 1

[Justin Crawford [:hoosteeno] [:jcrawford]]

> Considering the general activity of this IP, throttling IPs at 70 requests/minute (7 revisions a minute) > by IP would probably not have caught the IP. 

That should read, "...(10 revisions a minute)..." since we established there are about 7 requests per revision.

Comment 2

[Stephanie Hobson [:shobson]]

On Dec 17 I made 33 edits in 25 minutes. This is not an uncommon side effect of re-organizing CSS. 

If we go this route it would be nice if we could automatically check that none of the users from a problem IP have elevated user permissions (like admins or super users) before blocking. This would also help solve the problem of accidentally blocking a sprint if there was an admin on site :)

Comment 3

[Justin Crawford [:hoosteeno] [:jcrawford]]

(In reply to Stephanie Hobson [:shobson] from comment #2)
> On Dec 17 I made 33 edits in 25 minutes. This is not an uncommon side effect
> of re-organizing CSS. 

Lots of edits: 33*7/25, or (avg.) 9 requests to developer.mozilla.org a minute.

Comment 4

[Luke Crouch [:groovecoder]]

In addition, we need to make sure that any automatic request rate throttle does not affect KumaScript, which can easily make hundreds of requests per minute when rendering a page with many macros.

Comment 5

[Jamon Camisso [:jamon]]

I'll have a go at this by adding white listing to the django-banish module.

settings_local.py seems like the obvious place to pre-populate a list of known good IPs that is then added to memcached. In the event of a memcached restart or failure, this approach will ensure a persistent whitelist for things like kumascript and some of the more zealous editors ;)

I think I have enough info to get started on this task. I'll update with progress and a pull request once I've got something working.

Comment 6

[Jamon Camisso [:jamon]]

I opened a pull request with the app maintainer: https://github.com/yourabi/django-banish/pull/21

The code that I added is visible here in the meantime: https://github.com/jamonation/django-banish/commit/57f71bf34ca4a3d02bf0373ea0e715259993cbdb

Comment 7

[MDN Team (:mdn-dev)]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/705315bc90e1936734941935c083a9ca023530a7
bug 1122658 - extend KumaJob for IPBanJob

https://github.com/mozilla/kuma/commit/e10859a6a6775e7066b321f995e5a1eb246d09ae
bug 1122658 - move get_ip to kuma.core.utils

return proper "60/m" rate string from IPBanJob.empty()

use IPBanJob().invalidate to refresh cache when deleting IPBan

move cache.clear to BannedIPTest.tearDown()

https://github.com/mozilla/kuma/commit/82bd76e72c0566599b4627b49bd7b66da52b1e67
bug 1122658 - custom admin list_display for IPBan

Comment 8

[John Whitlock [:jwhitlock]]

We've addressed performance, rate limiting and IP banning in several ways in the migration to AWS.