Closed Bug 973044 (irc-per-ip-limit) Opened 12 years ago Closed 11 years ago

irc connection limit policy and protection

Categories

(Infrastructure & Operations Graveyard :: Infrastructure: IRC, task)

Product:
Infrastructure & Operations Graveyard
Component:
Infrastructure: IRC
Platform:
x86_64
Linux
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: cshields, Assigned: dparsons)

Details

I'm opening this bug and soliciting comments and possible solutions on our current irc configuration. We have a default per-IP limitation of 3 connections, with many exceptions setup for offices, coworking facilities, and conferences. This is problematic to our users as not only do they hit this cap when off on work weeks, conferences, or other events, but there is a long delay in getting an exception put into place. This limitation is setup to mitigate the effects of attacks that come from the same IP. I don't want to lose that protection altogether. While we need to work on relieving the limitation inconveniences, we need to make sure we don't lose the entire network to a DoS. A quick fix would be increasing the limit. However it won't stop there, as we need to resource and address the irc network altogether to ensure we can scale it further to meet the goal of increasing the community of Mozillians. Some conversations have already been happening about this, so I will let those ideas and options come out in this bug.
As a short term fix, why not have an LDAP-accessible page that lets people increase the limit for a given IP.
(In reply to Eric Rescorla (:ekr) from comment #1) > As a short term fix, why not have an LDAP-accessible page that lets people > increase the limit for a given IP. This has been one of the prevailing suggestions, thanks for mentioning it here.. Need to design/write it so its not as quick of a fix as an immediate bump, but a better one in that they would be vetted IPs.
(In reply to Eric Rescorla (:ekr) from comment #1) > As a short term fix, why not have an LDAP-accessible page that lets people > increase the limit for a given IP. +1 to this.
An even quicker fix that could be implemented without writing anything would be to create an internal connection to the IRC server, which can be reached via VPN and VPN-clients would not be subject to per IP limits. So, we could push a route to irc.mozilla.org through the VPN, and give an unlimited connection number to the VPN client subnets. Thus, when connected to MozillaVPN, you'd have no limits and when not connected, you'd have a limit. The downside I see here is that if you are connected to IRC and establish or kill a VPN connection, it would change this route and disrupt the connection? Maybe not though, I haven't tested. We should still consider the LDAP-authenticated self-serve limit bump tool, but in the interim, this could provide people immediate access to IRC via VPN while they wait for the limit to get increased.
Here's another way to do this as well (solution 1): have IRC identities tied to LDAP and allow a very high connection count on these. Easier to administer, too. there could be the nickserv for non-ldap, and SASL authentication for account holders, to facilitate things. Maybe it's easier to implement than the other solutions too (?). Additionally, we may consider providing a ZNC server again in the future (not now), that has good security settings and of course unlimited connections. This would also probably be tied with LDAP however. In the mean time, either using the VPN (solution 2), either using an LDAP-authenticated-page (solution 3) to raise connection limits seems reasonable. I expect the users will prefer solution 1 & 3 to solution 2, tho..
(In reply to Justin Dow [:jabba] from comment #4) > An even quicker fix that could be implemented without writing anything would > be to create an internal connection to the IRC server, which can be reached > via VPN and VPN-clients would not be subject to per IP limits. This would work if the only people affected by this have the Mozilla VPN client. I'm venturing to guess (since I don't have the data) that the people affected by this fall into one of these groups: * volunteer * community member * irc users that connect only from their laptop or mobile device
it sounds like to me that, having one admin page to administrate limits per ip, and have a few managers with access to it, is the most straight-forward and self-service solution right now. In the future however, the IRC auth and tying limit to identity seems the right way to go in my eyes (granted that our IRCd supports this), and having a ZNC server would be a (good) bonus. In any case, if all users have to have this as a self-service or automatic option, it means they'd all have to be vouched in some way (via having a LDAP account, or another kind of account that is vouched by a manager or something)
Can we have an immediate, short term and mid-term plan developed by COB Monday? Note that IRC users are also partners (adding to the list Joes had).
(In reply to Guillaume Destuynder [:kang] (use NEEDINFO!) from comment #5) > Here's another way to do this as well (solution 1): > have IRC identities tied to LDAP and allow a very high connection count on > these. Easier to administer, too. > there could be the nickserv for non-ldap, and SASL authentication for > account holders, to facilitate things. > Maybe it's easier to implement than the other solutions too (?). We could clone the VPN MFA certificate provisioning portal to produce SSL certificates that are accepted and verified against an internal revocation list when they connect to an otherwise-unrestricted IRC port. Both the above and my suggestion imply that users must take extra steps to alter their IRC client configurations to avoid antispam/antiabuse restrictions. It is not apparent whether solutions of that nature are permitted.
I confess I'm a little unclear on the security concern here. Is the idea that someone who has only a single IP address is going to connect to our IRC server and DoS it? That seems like a pretty low-grade threat and a pretty ineffective mitigation given the existence of botnets. How often has this kind of attack actually happened? I would suggest that instead of debating how to slightly loosen this restriction we experimentally remove it entirely and instead monitor the load on the IRC servers. If it get super high, presumably we can institute IP blocking of the offending IPs. I recognize that this has a risk of a short-term DoS while we wait for the block, but right now it represents a persistent self-inflicted DoS.
Guillaume and I had a little chat in SF and came up with some ideas. To summarize : History - There are limits on the IRC server today because there are number of botnets on the internet which have a history of attacking our IRC server. In the last couple of weeks, the limits we have in place on the IRC server have saved us from completely losing the service. IRCOps have had to take action to ban offenders and sometimes legitimate users have been unable to connect immediately because we've been under attack. Requirements - We need a solution that works for everyone - the community, our partners and employees as well as teams outside of offices. Due to this wide spectrum of users, some of the solutions above might not be practical or scalable. Short term (relatively painless) solutions - a) Put a password on an separate IRC server and link that to the current server. This allows people with the password to connect to IRC and not have any limits. b) Setup a web IRC client like qwebirc (http://qwebirc.org). Any one connecting via this would not be restricted by connection limits. This solution does have a captcha on it. From various perspectives, it seems like b) is a more viable solution in the long run. We haven't discussed this with some of our other colleagues, and we will get back with some more options on Tuesday, after a short chat.
(In reply to Shyam Mani [:fox2mike] from comment #11) > Guillaume and I had a little chat in SF and came up with some ideas. > > To summarize : > > History - There are limits on the IRC server today because there are number > of botnets on the internet which have a history of attacking our IRC server. > In the last couple of weeks, the limits we have in place on the IRC server > have saved us from completely losing the service. IRCOps have had to take > action to ban offenders and sometimes legitimate users have been unable to > connect immediately because we've been under attack. If we are being attacked by botnets, how does a per-IP limit help? How big are the botnets and what's the scaling limit here? > Requirements - We need a solution that works for everyone - the community, > our partners and employees as well as teams outside of offices. Due to this > wide spectrum of users, some of the solutions above might not be practical > or scalable. > > Short term (relatively painless) solutions - > > a) Put a password on an separate IRC server and link that to the current > server. This allows people with the password to connect to IRC and not have > any limits. > > b) Setup a web IRC client like qwebirc (http://qwebirc.org). Any one > connecting via this would not be restricted by connection limits. This > solution does have a captcha on it. > > From various perspectives, it seems like b) is a more viable solution in the > long run. > > We haven't discussed this with some of our other colleagues, and we will get > back with some more options on Tuesday, after a short chat.
Shyam, Joe, can we have an update on the next steps?
The next step is that the Systems team needs to decide on which of the following recommendations they wish to implement. If connection limits are the issue, OpSec's recommendations are: 1. Keep the current setup, increase per client connection limit to a higher number (approx 1k). 2. Replace the server software (unrealircd) with the same IRC server used by Freenode (ircd-seven), with high per client IP connection limits (approx 1k). 3. Remove connection limit from IPs associated with LDAP (via SASL) authenticated accounts. 4. Deploy an LDAP authenticated web page that lets a Mozilla employee set a fixed, high number of IRC connection limits from their current IP (approx 1k) for 7 days. 5. Shut down the Mozilla IRC server, and move our channels to Freenode. 6. Deploy a web IRC client (qwebirc) with no connection limits and captcha to prevent abuse. 7. Setup a parallel IRC server with a password shared amongst all Mozilla IRC users. Users that connect to this server will a have high connection limits (approx 1k). We're happy to help test whichever option is implemented.
(In reply to Joe Stevensen [:joes] from comment #14) > 1. Keep the current setup, increase per client connection limit to a higher > number (approx 1k). Won't this just make the current spambot situation worse since the bots will just connect as many clients as they can from the same IP? They already do that very often as it is... > 2. Replace the server software (unrealircd) with the same IRC server used by > Freenode (ircd-seven), with high per client IP connection limits (approx 1k). How does this actually solve any of the problems? > 5. Shut down the Mozilla IRC server, and move our channels to Freenode. I'm assuming this isn't a serious recommendation, considering the amount of sensitive stuff that is sent over Mozilla IRC. > 7. Setup a parallel IRC server with a password shared amongst all Mozilla > IRC users. Users that connect to this server will a have high connection > limits (approx 1k) No need for a separate server. Just set up a new class and allow block.
Thanks Joe, comments below.. (In reply to Joe Stevensen [:joes] from comment #14) > 1. Keep the current setup, increase per client connection limit to a higher > number (approx 1k). are you suggesting 1k connections per client? Not even freenode goes that high (their limit is 150) > 2. Replace the server software (unrealircd) with the same IRC server used by > Freenode (ircd-seven), with high per client IP connection limits (approx 1k). per above, their limit is 150. And, not to belittle the blood, sweat, and tears that the volunteers put into making Freenode what it is, but they have had their own share of attack problems and I'm not sure that we should assume their daemon to be a silver bullet. My irc uptime streak is 4 days in the past 30 there. :( I'm not too thrilled with increasing the limit but would settle for a "30 connections per IP" increase as a quick fix since the rest of the solutions will need time and resourcing. Can we agree to that? Given the limitation of overall connections that each server can handle, 1k unvetted would make a DoS easy.
(In reply to Joe Stevensen [:joes] from comment #14) > 3. Remove connection limit from IPs associated with LDAP (via SASL) > authenticated accounts. Would an infra-hosted multi-user ZNC instance that uses LDAP credentials be an acceptable implementation of this option? See also: https://github.com/jfryman/puppet-znc/blob/master/manifests/user.pp
It seems to me that some critical information is missing: - Do we have data that shows how many clients connections are supported until unrealircd fails? Rationale: It sounds like to me that from one single IP you could connect one client and spam 10000 channels or connect 10000 clients and spam 10000 channels just the same, since we ban by IP (if banning by client name it seems easy to change the name at random before connecting). Is that incorrect?
(In reply to Reed Loden [:reed] from comment #15) > (In reply to Joe Stevensen [:joes] from comment #14) > > 1. Keep the current setup, increase per client connection limit to a higher > > number (approx 1k). > > Won't this just make the current spambot situation worse since the bots will > just connect as many clients as they can from the same IP? They already do > that very often as it is... What is the current spambot situation? I don't have any of this data. I just know that the connection limits are impacting people at work weeks and we've been asked to help solve this problem. If you have data to share, please do. > > 2. Replace the server software (unrealircd) with the same IRC server used by > > Freenode (ircd-seven), with high per client IP connection limits (approx 1k). > > How does this actually solve any of the problems? I've been told that hitting a certain number of connections could crash unrealircd. If that is indeed the case, then I recommend we look at a different daemon. > > > 5. Shut down the Mozilla IRC server, and move our channels to Freenode. > > I'm assuming this isn't a serious recommendation, considering the amount of > sensitive stuff that is sent over Mozilla IRC. This is a serious recommendation. I'm happy to discuss more in detail, but we risk getting off topic. Ping me if you wish to discuss. > > 7. Setup a parallel IRC server with a password shared amongst all Mozilla > > IRC users. Users that connect to this server will a have high connection > > limits (approx 1k) > > No need for a separate server. Just set up a new class and allow block. Great!
(In reply to Corey Shields [:cshields] from comment #16) > Thanks Joe, comments below.. > > (In reply to Joe Stevensen [:joes] from comment #14) > > 1. Keep the current setup, increase per client connection limit to a higher > > number (approx 1k). > > are you suggesting 1k connections per client? Not even freenode goes that > high (their limit is 150) I'm suggesting whatever number will help make life easier for Mozilla IRC users. If 150 is the agreed upon number, great. I have no access to data that tells me what number is bad. > > > 2. Replace the server software (unrealircd) with the same IRC server used by > > Freenode (ircd-seven), with high per client IP connection limits (approx 1k). > > per above, their limit is 150. And, not to belittle the blood, sweat, and > tears that the volunteers put into making Freenode what it is, but they have > had their own share of attack problems and I'm not sure that we should > assume their daemon to be a silver bullet. My irc uptime streak is 4 days > in the past 30 there. :( That uptime number for Freenode is a useful data point. We're all in agreement that we should avoid an unstable IRC network. If Freenode isn't stable, then I'd redact our recommendation. > I'm not too thrilled with increasing the limit but would settle for a "30 > connections per IP" increase as a quick fix since the rest of the solutions > will need time and resourcing. Can we agree to that? Given the limitation > of overall connections that each server can handle, 1k unvetted would make a > DoS easy. Understandable, but what data are you basing your recommendation of 30 connections off of? I'd like to see how many connections per IP are *needed* in the various scenarios that generated this complaint in the first place. Also, at what point (connection wise) does IRC become unstable or goes down?
(In reply to Richard Soderberg [:atoll] from comment #17) > (In reply to Joe Stevensen [:joes] from comment #14) > > 3. Remove connection limit from IPs associated with LDAP (via SASL) > > authenticated accounts. > > Would an infra-hosted multi-user ZNC instance that uses LDAP credentials be > an acceptable implementation of this option? > > See also: https://github.com/jfryman/puppet-znc/blob/master/manifests/user.pp If this solution solves the problem, great. I don't have enough data to tell. I'd like to know how many users affected by this *don't* have LDAP creds.
(In reply to Joe Stevensen [:joes] from comment #21) > I don't have enough data to tell. I'd like to know how many users affected > by this *don't* have LDAP creds. I would as well - I know that Mozillians would be affected, and we have integration options for that datastore - and as far as I know, many (all?) partners have LDAP accounts. So, if we migrate all Mozillians and LDAP users to authenticated IRC connections, how often would an unusually high amount of users would be both (1) unauthenticated due to not being Mozillians, and (2) at an event where we aren't able to set IP limits in a reasonable time?
I think the easiest solution here, for both community members and staff members is for Mozilla to build a couple of beefy servers, and provide the option of running an instance of Quassel in a linux container. These quassel instaces could share a mysql backend, and users could run quassel on their platform of choice. We would retain control over the infrastructure, and could institute reasonable policies around retention of irc logs, since we own the servers we could ramp up the number of connections. We could aim for initially supporting staff, and look for some volunteers to help out with building some automation around setting up instances for users, and then open up support to vouched mozillians.
The recommendation to move away from unrealircd is valid, but not necessarily due to it being crashy with high connections. The problem we have with unrealircd is that we have no upgrade path for it, cannot link additional servers to it, can't move it to a different host. What we have is unsustainable and we cannot recover from the situation we are in. This means we can't move it to a different server, can't scale it, can't upgrade it, and can't test any changes to it (like potential add-on modules or risky configuration). These limitations mean that in order to do some of the suggestions above (linking new servers, testing new configuration, adding LDAP auth module, etc.) are not possible with the setup that we currently have. While we do have many options to tackle this issue, most of them are not possible until we've overhauled our IRC infrastructure. This means a total new architecture, new daemon, new configuration, new servers, etc. This is needed, but it is resource intensive. Once this is built, there will be a global Mozilla IRC outage during the migration as well, so it's not to be taken lightly. This is a huge task. The suggestion to use whatever freenode is using (ircd-seven) is arbitrary, but it is likely the most used IRCd out there and the most likely to be in active development. This is just my opinion. I have no data to back that up. Anyway, so given we need a solution sooner than later to the per-IP connection limit, but do not want to open ourselves up to spambot DoS attacks (not DDoS - the per-IP limit doesn't help there), pretty much the only solutions that we can do quickly are to: - increase the per-IP limit to something reasonable (currently it's 3, larger work weeks are what? 30 people? Average work weeks that are outside of a known Mozilla office - which already have unlimited connections - are probably less than 30 people. This is the problem we are trying to address, so I see where that number is coming from), - adding a shared password without connection limit, - adding a VPN-only endpoint without a connection limit (i.e. adding a route to the IRC server to the VPN configuration and granting the VPN client subnets unlimited connections). This would allow users to at least get on IRC through VPN if needed, until the request to get a specific IP's limit increased can be addressed by IT. Some of the other solutions have been addressed in various forms already. - ZNC server with LDAP auth and no connection limits. This is not trivial. ZNC brings with it a whole new set of problems that we are not equipped to support. Multi-user ZNC with LDAP auth *can* be automated to some extent, but it would take a significant amount of work up front to figure it out. (Hint: I've played around with this quite a bit in the past to no avail. I think it's possible in theory, but have not had success with it. I've wasted significant time on this already. Patches welcome). Also ZNC brings with it its own little quirks, like /query messages when a user is set to /away because their client detaches, in some instances just disappear - the sender thinks the receiver got the message, but they didn't. - Quassel - not client agnostic IIRC? I might be wrong, it's been a long time since I looked at it. Same with the recommendation for qwebirc. This will solve the problem for *some* users, but *most* users want their own client. For instance, we already provide a solution: We provide a shell server for people to run irssi in a screen session and this has no limits. *Some* people make use of it, but not everyone, which is why we have the problem in the first place. So, in a way we already provide this type of solution and people don't make use of it, so adding quassel, qircweb, ZNC, etc. will just help a small subset of users, but not be a global solution (otherwise we can just tell everyone to use people.mozilla.org with irssi/screen and that's that). All of the above is my personal opinion, with hopes of clearing up some confusion. I'm not speaking for IT or my team here. It is just as I see the situation, and I actually know very little about IRC itself, so I'm not further qualified to make a final decision on what we go with, but my actual recommendation would be to just increase the current per-IP limit to something that will unblock most work weeks, but still need a special request for stuff out of the ordinary (like 50 people meeting at some random place). I also think we need to address the issues we have with unrealircd, but that is a separate issue and I brought it up here only to illustrate that some of the solutions suggested are not easily implemented without first tackling a mountain of technical debt.
(In reply to Justin Dow [:jabba] from comment #24) > The problem we have with unrealircd is that we have no upgrade path for it, > cannot link additional servers to it, can't move it to a different host. To clarify, this is only a problem with our *current* unrealircd. A newer version of unrealircd, or literally any other irc daemon as well, would address those specific issues.
Corey, as the service owner for IRC, could you pull together the plan of action & timelines please?
Humble opinion - As Joe alluded to, we lack the historical attack data to really hone in on a true solution. Nonetheless, my preference would be that we do NOT impose additional steps on the end-user. This includes VPN, passwords, etc. Since per-IP limits have nothing to do with DDoS and we're primarily concerned with multiple attacks from the *same* IP, isn't there some magic we can work at the network layer? For instance:
(Sorry (In reply to dtorre from comment #27) > Humble opinion - > > As Joe alluded to, we lack the historical attack data to really hone in on a > true solution. Nonetheless, my preference would be that we do NOT impose > additional steps on the end-user. This includes VPN, passwords, etc. > > Since per-IP limits have nothing to do with DDoS and we're primarily > concerned with multiple attacks from the *same* IP, isn't there some magic > we can work at the network layer? For instance: - Rate limits - Throttling embryonic ("half open") TCP sessions - etc
(In reply to SylvieV from comment #26) > Corey, as the service owner for IRC, could you pull together the plan of > action & timelines please? Sorry - met about this late last night and haven't updated this bug. Our immediate plan (hopefully going out today) is to setup a port and password combination that is unlimited so that people can have concurrent access without an exception. We will work on communication for this feature. We are still working out the long term plan, it will include bringing unrealircd up to date and increasing the footprint of our network to add capacity so that we are not as sucseptable in an attack. (In reply to dtorre from comment #27) > As Joe alluded to, we lack the historical attack data to really hone in on a > true solution. Nonetheless, my preference would be that we do NOT impose > additional steps on the end-user. This includes VPN, passwords, etc. We have some historical data but it does not break down to the IP level to show where this limitation has helped. My preference is to not impose additional steps on the end-user as well but we need to do something right away and the alternative solutions will take some more time and effort. > Since per-IP limits have nothing to do with DDoS and we're primarily > concerned with multiple attacks from the *same* IP, isn't there some magic > we can work at the network layer? For instance: In this case they do when you have a limited number of client connections in an irc server. Let's say (hypothetically) there are 600 available connections, you need 200 systems with a per-ip limit of 3 to fill those connections and prevent anyone else from connecting to irc. As the per-ip limit goes up, the number of resources needed to prevent further users decreases. These are all IRC bots that act like real users, so if you try to impose limits and catch this at the network layer you end up catching real users as well.
Since the question keeps getting asked... most of the botnets that attack us will make as many connections as they can from every IP address they possess, and will keep opening connections from each IP until they hit our limit. So DDoS protection is very relevant here. Every additional connection per IP that we allow decreases the number of IPs they need to have in the botnet to fill our capacity. I believe we've attempted to update unrealircd lately and weren't able to get a stable build working anywhere other than concrete...
As per irc conversation, I've produced a crash-consistent clone of irc1.dmz.phx1, named irc2.dmz.phx1. It's ready for people to work on it - I've prevented unreal/anope from running at boot, and the system needs to be repuppetized under its new hostname.
Corey - checking in on where we are on this. thnx
(In reply to Dave Miller [:justdave] (justdave@bugzilla.org) from comment #30) > Since the question keeps getting asked... most of the botnets that attack us > will make as many connections as they can from every IP address they > possess, and will keep opening connections from each IP until they hit our > limit. So DDoS protection is very relevant here. Every additional > connection per IP that we allow decreases the number of IPs they need to > have in the botnet to fill our capacity. > What's our self-imposed limit vs max capacity?
Component: Infrastructure: Other → Infrastructure: IRC
We've decided that the best way to handle this problem is with a web app that lets the end user increase the limit for their own IP address. Work for this is being tracked in bug 1083407.
Assignee: infra → dparsons
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.