729466 - i.m.o blocking workweek participants in Paris (again)

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Andreas Gal :gal]

Irc is blocking us gain. This is seriously hurting our ability to get work down. Please remove per-ip connection caps for the rest of the work week for every IP in France. I really don't want to file this bug a third time. Thanks.

NICK gal
USER gal 0 * Andreas Gal
NOTICE AUTH *** Looking up your hostname...
NOTICE AUTH *** Couldn't resolve your hostname; using your IP address instead
005: PING
PONG 5E26194C
ERROR Closing Link: gal[80.214.9.4] (Too many connections from your IP)

Comment 1

[Shyam Mani [:fox2mike]]

Andreas,

We understand this is hurting your ability to work. 

Asking us to unblock every IP in France is like saying add everyone in the company to all the mailing lists Mozilla runs, which while doable with some difficulty isn't practical for a lot of reasons.

A lot of teams have had multiple work weeks in multiple locations without a hitch because people have gotten in touch with us before hand and we've worked with them to find out about ISPs and other logistics so they will not run into issues like this.

I'll toss this over to the team to handle. Feel free to add any other IPs you run into on this bug, we'll add them to whitelist ASAP.

Comment 2

[Andreas Gal :gal]

If its easier to simply disable per-ip connection maximums globally, maybe thats what we should do. It seems to be causing a lot of headache for a lot of teams all the time from your description.

Comment 3

[Shyam Mani [:fox2mike]]

(In reply to Andreas Gal :gal from comment #2)
> If its easier to simply disable per-ip connection maximums globally, maybe

There are reasons we don't do that because primarily the IRC world is a pretty bad world to be in. These protections help keep our servers safe/running and reachable and are pretty much standard in any IRCd software worth it's salt. 

While you might hate your seatbelt, it does save lives, which is why it's there on every car.

Comment 4

[Andreas Gal :gal]

A seat belt is a passive safety device that doesn't get in the way of driving your car, and is generally pretty effective. This particular safety device is trivial to circumvent, and it doesn't allow you to drive your car half the time. Hence, I don't think the comparison makes a lot of sense. Do you have any documentation that this practice is widely used? Do we have documented evidence of what happens if we don't use it? imo is the only irc serve I have this problem with.

Comment 5

[Andreas Gal :gal]

This is some random source 30s of searching on google dug up. I would say at least there is no unanimous consent that per-ip blocking is useful, and people seem to think that there are more effective alternatives. I would say we should look into this scientifically and get some actual data.

http://searchirc.com/boards/irc-8963.html

Comment 6

[Shyam Mani [:fox2mike]]

(In reply to Andreas Gal :gal from comment #4)

> you have any documentation that this practice is widely used? Do we have
> documented evidence of what happens if we don't use it? imo is the only irc
> serve I have this problem with.

1) DALNet - http://docs.dal.net/docs/general.html (see section on cloning)
2) EFNet  - http://www.efnet.org/?module=docs&doc=9#IV
3) Freenode - http://freenode.net/policy.shtml (see connect limit)

The reason you don't get affected by these are because 

1) These are massive networks, Freenode for example has 19 servers.
2) They have a lot of staff, watching, cleaning up after or generally being available to police the network.

Neither of this is true for us. 

Even with these protections in place, we've had "join floods" (where a massive number of bots flood the network) and have had to deal with issues arising out of that, to give you one example.

If it's so trivial to circumvent, why haven't you done so already? :) Would have saved you the trouble of filing this bug and having this discussion? 

tl;dr - Everyone implements it, at different levels. I don't think any of the other IRC admins would disagree with me, so I'll cc them for comments, if any.

Comment 7

[Andreas Gal :gal]

Thats a fair point about other irc networks having more scale than us. Can we outsource irc and have freenet (or some other provider) operate our irc network with their many services and admins?

Comment 8

[Shyam Mani [:fox2mike]]

(In reply to Andreas Gal :gal from comment #7)
> Thats a fair point about other irc networks having more scale than us. Can
> we outsource irc and have freenet (or some other provider) operate our irc
> network with their many services and admins?

I don't see the need to outsource anything. You completely missed my point about it being standard practice.

Comment 9

[Andreas Gal :gal]

"The reason you don't get affected by these are because 

1) These are massive networks, Freenode for example has 19 servers.
2) They have a lot of staff, watching, cleaning up after or generally being available to police the network.

Neither of this is true for us."

I would like to make it true for us as well.

Comment 10

[Ashish Vijayaram [:ashish]]

Exceptions added to 80.214.9.* You should be able to connect now.

Comment 11

[Shyam Mani [:fox2mike]]

We're not at that scale. 

Freenode :

20:05:49 -!- There are 254 users and 70870 invisible on 27 servers
20:05:49 -!- 40 IRC Operators online
20:05:49 -!- 9 unknown connection(s)
20:05:49 -!- 36936 channels formed
20:05:49 -!- I have 4258 clients and 1 servers
20:05:49 -!- 4258 6782 Current local users 4258, max 6782
20:05:49 -!- 71124 78459 Current global users 71124, max 78459
20:05:49 -!- Highest connection count: 6783 (6782 clients) (1488809 connections received)

Mozilla : 

04:05:56]            --- | Server: There are 12 users and 1565 invisible on 4 servers
[04:05:56]            --- | Server: 21 operator(s) online
[04:05:56]            --- | Server: 571 channels formed
[04:05:56]            --- | Server: I have 195 clients and 3 servers
[04:05:56]            --- | Server: Current Local Users: 195  Max: 2189
[04:05:56]            --- | Server: Current Global Users: 1577  Max: 2204

70000 vs 1500. 3 servers are overkill for 1500 users, we already have 3, we might even add one more.

Comment 12

[Andreas Gal :gal]

I understand we are not at that scale. Thats why I suggested having someone else operate our irc network on their infra so we get that scale, and all the benefits you described.

Comment 13

[matthew zeier [:mrz]]

(In reply to Andreas Gal :gal from comment #7)
> Can
> we outsource irc and have freenet (or some other provider) operate our irc
> network with their many services and admins?

To that last point, there is nothing preventing you from using some other IRC network.  irc.mozilla.org will remain hosted on Mozilla infra because of the security/privacy issues.

(In reply to Andreas Gal :gal from comment #9)
> Neither of this is true for us."
> 
> I would like to make it true for us as well.

I need more head count.

Comment 14

[Ben Kero [:bkero]]

The reason we overcame on Freenode is that I personally know Freenode staff and got them to make an exception for our shell servers.  They aggressively do per-IP connection limits.  Whatever other network you take the channel to is going to have the same limits, be it Freenode, OFTC, DALNet, EFNet, or Undernet.

A temporary workaround would be to tell folks to connect through people.mozilla.org, we've whitelisted this host to establish any number of connections.