1393146 - (bmo-autohammer) Automate blocking IPs that bugzilla flags as exceeding rate limits

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: General, enhancement)

Description

[Dylan Hardison [:dylan] (he/him)]

In many cases, the bugzilla codebase can identify abusive amounts of requests from the same IP. Currently we log this to syslog with a special tag.

It would be nice if *something* could automatially block those.

Comment 6

[Dylan Hardison [:dylan] (he/him)]

Luckily the application also has enough information to help guess who is a legitimate user (hi, glandium) and who is a bot.

Comment 7

[Greg Cox [:gcox]]

(In reply to Kendall Libby [:fubar] from comment #3)
> worth noting that the other day we blocked the paris office because it's
> external IP was hammering us (and there was no rDNS to make it obvious who
> it was). not the first time and won't be the last time that happens. I don't
> know if there's a useful place we can pull a list of known office IPs from,
> but we should have some mechanism for whitelisting them.

bug 1291311 might be of interest to you.

Comment 9

[Peter Radcliffe [:pir]]

(In reply to Greg Cox [:gcox] from comment #7)
> (In reply to Kendall Libby [:fubar] from comment #3)
> > worth noting that the other day we blocked the paris office because it's
> > external IP was hammering us (and there was no rDNS to make it obvious who

Strongly agreed. We've had repeated problems with every blocking method available in blocking offices and causing outages for staff, some small and some (in the case of banhammer) large. 

> bug 1291311 might be of interest to you.

There is also work in bug 1342548 to maintain a consumable list in S3 of IP ranges for white listing in banhammer. Using the same data/process would seem to make sense.

Comment 10

[Dylan Hardison [:dylan] (he/him)]

Since that isn't resolved and I want to get this resolved relatively soon, I'm going to build a set
of IPs used by all regular users over the last year.

Comment 11

[Peter Radcliffe [:pir]]

(In reply to Dylan Hardison [:dylan] (he/him) from comment #10)
> Since that isn't resolved and I want to get this resolved relatively soon,
> I'm going to build a set
> of IPs used by all regular users over the last year.

Office addresses change painfully often. Datacenter addresses are going to change with scl3 to mdc1 migrations.

Hopefully changes will happen less often as we migrate to using our own IP space for offices... but some of those addresses in the last year will now be obsolete as offices have closed (Auckland, for example) and new ones have opened (BER3 quite recently).

If you don't have a way to get updates in the short term (I do understand needing a passable solution quickly) then please, please, consider adding that as soon as is reasonably possible. This is something that bites us, in the MOC, on a regular basis and causes outages.

Comment 12

[Dylan Hardison [:dylan] (he/him)]

(In reply to Peter Radcliffe [:pir] from comment #11)
> (In reply to Dylan Hardison [:dylan] (he/him) from comment #10)
> > Since that isn't resolved and I want to get this resolved relatively soon,
> > I'm going to build a set
> > of IPs used by all regular users over the last year.
> 
> Office addresses change painfully often. Datacenter addresses are going to
> change with scl3 to mdc1 migrations.
> 
> Hopefully changes will happen less often as we migrate to using our own IP
> space for offices... but some of those addresses in the last year will now
> be obsolete as offices have closed (Auckland, for example) and new ones have
> opened (BER3 quite recently).
> 
> If you don't have a way to get updates in the short term (I do understand
> needing a passable solution quickly) then please, please, consider adding
> that as soon as is reasonably possible. This is something that bites us, in
> the MOC, on a regular basis and causes outages.

I know the IP of every logged-in person that's used BMO in the last year, regardless of if they're in an office.
it's a lot of IPs, but if I filter by ones that typically make a lot of requests it is quite manageable.

Comment 13

[Dylan Hardison [:dylan] (he/him)]

Oh, and it wouldn't be a one time list, but something I can re-generate on some sort of automated schedule.

Comment 14

[Dylan Hardison [:dylan] (he/him)]

So when we block an IP, what layer is that?

Do I want to see if this is something mozdef can do by looking our syslog,
or can I write some code that talks to zeus?

I am fairly confident I'll be able to:

1) accurately detect abusive amounts of traffic
2) ignore such when the IP is in a (potentially large, currently 57k) list.

Comment 15

[Kendall Libby [:fubar] (he/him)]

So, neither zeus nor mozdef will be helpful in a couple months. Writing anything that works with them seems like it might be a waste of time. On the other hand, I'm not sure how they might want to handle this in CloudOps; I don't imagine they would want the app changing security groups.

Zeus does appear to let you modify protection classes via API (SOAP), but permissions are NOT fine grained so it's up to webops as to whether or not they would allow us to modify any/all protection classes. It might be possible to have a TrafficScript rule that calls out to a service, but I'd be worried about performance implication.

Comment 16

[Dylan Hardison [:dylan] (he/him)]

Okay, I'll see if I can make httpd bail out early based on... something. I'm pretty sure I'll be able to handle this with ELBs.

Comment 17

[Dylan Hardison [:dylan] (he/him)]

BMO is going to start responding to requests that are "excessive" with HTTP 429 - TOO MANY REQUESTS.

This will only be done for IPs that go over the rate limit repeatedly, and will whitelist (report only)
any IP address that is reasonably determined to not be a scanner.

What I mean to say here is that BMO will not return HTTP 429 lightly.

With that in mind, the load balancer could possibly use this as a signal that a client is being aggressive
and harming "the commons", and take some sort of action?

We can wait on this and see how well the code in 1393888 works, but the cost per request (because of apache) is still quite high.
On bugzilla-dev, I can't get much more than 300ms per request.

Comment 18

[Dylan Hardison [:dylan] (he/him)]

This can't go out until we update the whitelist.