651780 - hg.mozilla.org & svn.mozilla.org are rejecting SSH connections (can't push / commit)

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Daniel Holbert [:dholbert]]

[dholbert@fx:~/builds/mozilla-central/mozilla]$ hg out
comparing with ssh://hg.mozilla.org/mozilla-central/
remote: ssh_exchange_identification: Connection closed by remote host
abort: no suitable response from remote hg!
[dholbert@fx:~]$ ssh hg.mozilla.org
ssh_exchange_identification: Connection closed by remote host


Looks like hg.mozilla.org is rejecting SSH connections.  This is very bad; it means no one can push to any repos hosted there.

Comment 1

[Daniel Holbert [:dholbert]]

(I've confirmed on several computers that were working earlier today, and dbaron confirmed that this affects him too)

Comment 2

[Daniel Holbert [:dholbert]]

The last push to mozilla-central was at 17:02 PM (54 minutes ago), so it was apparently working as recently as then, and something became busted in the last hour.

Comment 3

[Pascal Chevrel:pascalc]

probably relatedn svn is also rejecting connections with the same error message

Comment 4

[Phong Tran [:phong]]

It should have recovered now.

Comment 5

[Daniel Holbert [:dholbert]]

hasn't yet...  still getting the same results as comment 0

Comment 6

[Pascal Chevrel:pascalc]

same for me on svn, not recovered yet

Comment 7

[Daniel Holbert [:dholbert]]

http://hg.mozilla.org now seems to be rejecting HTTP connections, too (but maybe that's just because something's being rebooted as part of fixing this bug?)

Comment 8

[Phong Tran [:phong]]

now it should be up again.  had to reboot the second host as well.

Comment 10

[Richard Newman [:rnewman]]

Err... I just duped my report to this, but this is marked as FIXED but I'm still being rejected:

  remote: Permission denied (publickey,gssapi-with-mic).
  abort: no suitable response from remote hg!

Any thoughts?

Comment 11

[:Gavin Sharp [email: [e-mail]]]

Still broken for me too.

Comment 12

[Chris Cooper [:coop] (he/him)]

This is also blocking the 3.5.19 release process.

Comment 13

[Daniel Holbert [:dholbert]]

WORKSFORME now (from outside the mozilla office if that matters):
> [dholbert@azure:~]$ ssh hg.mozilla.org
> No interactive shells allowed here!
> Connection to hg.mozilla.org closed.
> [dholbert@azure:~/builds/mozilla-central/mozilla]$ hg out
> comparing with ssh://hg.mozilla.org/mozilla-central/
> searching for changes
> no changes found

gavin / coop / rnewman, still busted for you?

Comment 14

[Chris Cooper [:coop] (he/him)]

WFM. Update step for 3.5.19 release is running now.

Comment 15

[Daniel Holbert [:dholbert]]

Re-resolving as fixed per comment 13 & comment 14.  Assuming there was just a delay on comment 8 taking effect for some reason.

Comment 16

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Adding others in case this also impacts FF3.6.17 builds also in progress.

Comment 17

[Richard Newman [:rnewman]]

Working now, thanks!

Comment 18

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

DUH - midair'd and accidently REOPENED. Phong had this FIXED, so re-FIXING.

Comment 19

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Phong: can you give details on what went wrong? There is no explanation here of what went wrong, and what can be done to avoid this in future.

Comment 20

[Phong Tran [:phong]]

The main svn/hg server is on a hp dl360 G4.  It has a known issues with the current RHEL 5.6 kernel that cause it panic.  LDAP is running on that server which causes everything to go down.  We are putting a fix in place.