803391 - Making login.persona.org pingable

Status: RESOLVED WONTFIX

(Cloud Services :: Operations: Miscellaneous, task)

Description

[François Marier [:francois]]

login.persona.org is currently not pingable: https://github.com/mozilla/browserid/issues/2550

I was wondering whether there's a reason for this, because it is somewhat annoying when doing performance tests for example.

If that's just part of the default firewall rules and it's not a big deal to stop filtering those, then it would be nice to allow these icmp packets through.

Comment 1

[Ravi Pina [:ravi]]

A ping of the IP (which could very well change in the future, become an anycast, a VIP, etc) doesn't provide interesting metrics, I don't think.

A test that can measure the response time of a query and check for valid data (e.g. HTTP GET) would provide much better and useful data.

Comment 2

[François Marier [:francois]]

You're right, that's a fairly limited way of getting performance metrics. Our performance tracking work will involve a lot more than pinging the host (see https://id.etherpad.mozilla.org/make-us-fast for the details), so I guess this bug was more general.

Basically: if we don't have a good reasons for blocking pings, can we allow it?

I'm perfectly happy to close this bug if there is in fact a good reason for defaulting to dropping icmp packets.

Comment 3

[Kevin Locke]

Ping is a convenient way to test connectivity.  It's not the only way, nor the best way (depending on the symptom and user doing the testing), just another tool.  Your support crew can weigh in on how much they want it to help them walk users through issues.

However, what motivated me to mention it (and then file a bug for it) is that the server responds to ICMP Echo with ICMP Port Unreachable.  This is an odd choice since ICMP doesn't have a notion of ports, and it doesn't avoid any of the security concerns that might motivate silently discarding ICMP Echo (at least, none that I am aware of).  So I am guessing that this behavior is not by design.  Take it for what it's worth.

Comment 4

[casey ransom [:casey]]

I agree. Ping is a fundamental sanity check that people use to know 'is this host alive' and by blocking it, it only causes confusion and makes troubleshooting difficult and I see no gains.  Unfortunately, netops isn't the one blocking ICMP here as it's the service operations group running the load balancer.  I'll redirect this request to them to hopefully make the change for login.persona.org as well as the rest of the frontend IPs.

Comment 5

[:Atoll]

Opsec, can you signoff on this change? I wasn't personally aware we blocked ping, but I'd still like to see an "OK" before we proceed.

Comment 6

[Michal Purzynski [:michal`] (use NEEDINFO)]

Sure, i ACK it.
I see no problem with allowing ICMP echo request + reply, we do it already company wide, and as :casey said, denying it brings a lot more trouble than it's worth (and no security gains either).

Comment 7

[Michal Purzynski [:michal`] (use NEEDINFO)]

(In reply to Kevin Locke from comment #3)

> However, what motivated me to mention it (and then file a bug for it) is
> that the server responds to ICMP Echo with ICMP Port Unreachable.

That's a standard behavior on RedHat based distributions and appliances, as there's probably a catch all rule to do just that (and no rule for allowing it).

Comment 8

[Kevin Locke]

(In reply to Michal Purzynski [:michal`] from comment #7)
> That's a standard behavior on RedHat based distributions and appliances, as
> there's probably a catch all rule to do just that (and no rule for allowing
> it).

I'll be happy to file bug reports with the distributions as well.  However, I couldn't confirm the behavior on CentOS 6.3 (using the LiveCD).

Comment 9

[casey ransom [:casey]]

Ping... no pun intended... on this.

Aside from general connectivity, dropping all ICMP also breaks path MTU detection which can also be troublesome. 

Could this get scheduled for the next change window?

Comment 10

[Gene Wood [:gene]]

casey, can you describe the change that you'd like to see? I'm not sure if this is a NetOps change to a network device or a change to the zeus load balancers.

Also if it isn't that urgent and can wait 4 weeks we'll be out of the datacenters entirely and not using zeus (AWS with elastic load balancers instead) at which point I suspect the issue becomes moot.

Comment 11

[casey ransom [:casey]]

(In reply to Gene Wood [:gene] from comment #10)
> casey, can you describe the change that you'd like to see? I'm not sure if
> this is a NetOps change to a network device or a change to the zeus load
> balancers.

Entirely an iptables zeus change.

> Also if it isn't that urgent and can wait 4 weeks we'll be out of the
> datacenters entirely and not using zeus (AWS with elastic load balancers
> instead) at which point I suspect the issue becomes moot.

Yep, it becomes a non issue.  If the phx1 zeus boxes are being retired as well, then there's no point in keeping this open.

Comment 12

[Gene Wood [:gene]]

Ok, ya we'll be out of phx1 and scl2 by 4/30 at the latest, but 4/1 is our scl2 planned exit date and 2 weeks later for phx1. I'll close this based on that.

Also, apologies on this. I hate being that guy who waits so long that a problem becomes inapplicable because of some fundamental change like this. I hate it when other people do it to me and I try not to do it to others. So ya, sorry about that.

Comment 13

[Kevin Locke]

Not a problem.  I'm much more interested in getting it fixed than I am in whether it's fixed by changing the configuration or the datacenter.  Thanks for the update.