Unable to ssh to people.mozilla.com

RESOLVED FIXED

Status

--
major
RESOLVED FIXED
8 years ago
4 years ago

People

(Reporter: fabrice, Assigned: jlaz)

Tracking

Details

(Reporter)

Description

8 years ago
people.mozilla.com is not accessible from ssh.

ping is ok:
PING people.mozilla.com (63.245.208.169) 56(84) bytes of data.
64 bytes from people.mozilla.com (63.245.208.169): icmp_req=1 ttl=62 time=5.09 ms
64 bytes from people.mozilla.com (63.245.208.169): icmp_req=2 ttl=62 time=2.22 ms
64 bytes from people.mozilla.com (63.245.208.169): icmp_req=3 ttl=62 time=2.39 ms
ssh -v people gives this output:
{
[dholbert@fx:~]$ ssh -v people
OpenSSH_5.8p1 Debian-1ubuntu3, OpenSSL 0.9.8o 01 Jun 2010
debug1: Reading configuration data /home/dholbert/.ssh/config
debug1: Applying options for people
debug1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to people.mozilla.org [63.245.208.169] port 22.
debug1: Connection established.
debug1: identity file /home/dholbert/.ssh/id_rsa type 1
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
debug1: identity file /home/dholbert/.ssh/id_rsa-cert type -1
}
and then hangs.

If I ssh -v to e.g. office.mozilla.org, then I get the same output, followed by these lines (and then a lot more, and then a successful connection).
{
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.2
debug1: match: OpenSSH_5.2 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-1ubuntu3
}

So it looks like the SSH server on people is accepting TCP connections, but failing before it sends back the remote protocol version, or something.
Also, looks like a bunch of people just got booted off of IRC, because they were connected via people.  (that was at least the case for dolske)

This needs fixing ASAP.
I've got this problem too. Earlier SSH connections were just hanging, now I get a quick "connection refused".

Oddly, when I tried earlier I was able to get quick replies from people via FTP and WWW. So, something unique to SSH or shell logins?

FWIW, the 3 of us are in the MV office, lest it be a local connectivity issue.
Severity: critical → blocker
Hmm, seems fixed now? I just sshed in.
WFM too now.
Severity: blocker → critical
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → WORKSFORME
(er sorry for stomping on dolske's severity-tweak -- darned bugzilla-reload-not-updating-flags fail)
Severity: critical → blocker
We restarted people just a few minutes ago since we could not bring up a console at all.  We're trying to determine what happened to cause the system to be non-responsive to SSH.  

Things seem fine now after the restart.  Let us know if you notice anything else that seems out of place.

Updated

8 years ago
Assignee: server-ops → jlazaro
Severity: blocker → major
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
(In reply to comment #7)
> We restarted people just a few minutes ago

(Ah - so maybe dolske's "connection refused" from comment 3 were during the restart)
From what I've been told, it sounds like oomkiller fired off and killed some important processes since it was running out of resources.
I think this was reopened by mistake.
Status: REOPENED → RESOLVED
Last Resolved: 8 years ago8 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.