1497600 - Create an iprepd nginx module

Status: RESOLVED FIXED

(Data Platform and Tools Graveyard :: Operations, defect)

Description

[Brian Pitts]

Currently, if an application wants to block requests based on the reputation of the client's IP address, it requires reworking the application to consult with iprepd. This is undesirable for two reasons

1) This can be a decent amount of dev work, particularly if there is no existing iprepd client for the language the application is written in
2) It requires the application's worker processes to spend time processing requests that will be dropped, which means it possible for an attacker to DDoS it despite iprepd

An nginx or openresty module to consult iprepd and block requests would help with both these problems.

1) It would eliminate the dev work and instead require an operator to configure it within nginx. This would be standardized and much quicker/simpler.
2) Nginx would shield the application from spending resources on the blocked requests. Since Nginx is much more efficient than most of our application, this would make it harder to DDoS.

My guess at how, as an operator, I'd expect this to behave are

* It uses remote_addr as the client ip
* I can configure it at the server or location level
* It causes nginx to immediately respond with 403 (maybe configurable?) for clients below a reputation threshold. Requests from clients that are above a threshold or unknown are processed normally.
* It caches responses from iprepd for a sane amount of time (60s? maybe configurable?)
* It times out while connecting to iprepd plus waiting for a complete response within a sane amount of time (2ms? maybe configurable?)
* It fails open
* It log rejections to the error log at a configurable log level

It could be interesting to, instead of immediately responding with a 403, have nginx sleep for some amount of time in order to tarpit attackers.

Comment 1

[Brian Pitts]

On second thought "It uses remote_addr as the client ip" is too simplistic.

Below ELB is Amazon's Elastic Load Balanacer, and GCLB is Google Cloud HTTPS Load Balancer.

On a lot of projects the true client ip address isn't directly accessible because an ELB is between the client and the application. In that case $remote_addr is the ELB's IP address and the client's address is only in the X-Forwarded-For header. The last of the comma-separated entries in that header is set by the ELB and is trustworthy.

On some projects I've started using the realip module (http://nginx.org/en/docs/http/ngx_http_realip_module.html) to have nginx parse X-Forwarded-For and update $remote_addr for me. It can properly handle how an ELB sets the header and it should (with a slighlty different config) handle how a GCLB would set it. Your simplest option would be to require people to use the realip module in order to use the iprepd module.

Alternately, I think you'd need a configurable mode where you either treat $remote_addr as the client address (suitable for projects like our admin panels that do not have a load balancer in front) or you parse it from x-forwarded-for (suitable for our stock nginx config for an app behind an ELB).

If you go the parsing route you'll need to handle it differently for an ELB vs GCLB. With an ELB nginx for the app sees

X-Forwarded-For: <unverified IP(s)>, <immediate client IP>

but with a GCLB I believe it sees

X-Forwarded-For: <unverified IP(s)>, <immediate client IP>, <global forwarding rule external IP>

Comment 2

[AJ Bahnken [:ajvb] (she/her)]

Blocks 1464143

Comment 3

[AJ Bahnken [:ajvb] (she/her)]

https://github.com/ajvb/iprepd-nginx/pull/1

Comment 4

[AJ Bahnken [:ajvb] (she/her)]

Brian, would you mind reviewing https://github.com/ajvb/iprepd-nginx/pull/1 when you have a moment and verifying that it fulfills all current requirements? (sent you an invite to be a collaborator on the repo).

Comment 5

[Brian Pitts]

Of course! I'm pretty busy today, so it might not happen until tomorrow.

Comment 6

[Brian Pitts]

Seems to fit the requirements very well! I did find a few issues, but nothing big.

Comment 7

[AJ Bahnken [:ajvb] (she/her)]

Awesome, thank you for the thorough review brian.

Comment 8

[AJ Bahnken [:ajvb] (she/her)]

v1 completed with https://github.com/mozilla-services/iprepd-nginx/