499288 - ldap authed services (mail, mpt-vpn, intranet wiki+forum, maybe more) are down

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[[e-mail] (:bhearsum)]

Having trouble accessing some services right now. The intranet wiki and forum are down, I cannot ssh to mpt-vpn nor connect to it with Viscosity.

Catlee is having problems with Mail, too.

Filing as blocker because I need mpt-vpn to push out 3.5rc2 today.

Comment 1

[[e-mail] (:bhearsum)]

Looks like hg.m.o ssh access is in bad shape, too:

bitters-2:bouncerbuddy bhearsum$ ssh hg.mozilla.org
ssh_exchange_identification: Connection closed by remote host

Comment 2

[Pascal Chevrel:pascalc (PTO until April 26)]

other services unavailable:
smtp.mozilla.org
https://stage.mozilla-europe.org
https://www-trunk.stage.mozilla.com/

Comment 3

[Johnathan Nightingale [:johnath]]

likewise people, mail (but not bugzilla, happily.)

Comment 4

[Robert Kaiser]

cb-jumphost-01 also closes the connection when trying to ssh into it.

Comment 5

[Mike Beltzner [:beltzner, not reading bugmail]]

Looks LDAP related. I can get to:

 https://wiki.mozilla.org

but not

 https://intranet.mozilla.org

or anything else that requires LDAP authentication.

Comment 6

[Johnathan Nightingale [:johnath]]

bhearsum prodded nagios to manually page dmoore, since we're not entirely confident it would happen on its own, and we can't get to phonebook to look up his cell directly.  If we don't hear back shortly, I'll be the annoying guy and wake mrz up, since I have his cell number in my own address book.

Comment 7

[Axel Hecht [:Pike]]

we have some local test code that uses ldap auth, gandalf tested:

ldap.SERVER_DOWN: {'info': 'error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure', 'desc': "Can't contact LDAP server"}
handshake failure

So yeah, it's ldap itself that's not answering our pleas.

Comment 8

[Johnathan Nightingale [:johnath]]

mrz is now on the case

Comment 9

[Johnathan Nightingale [:johnath]]

08:57 < mrz> i restarted ldap, working for anyone else?
08:58 < bhearsum> yep
08:58 < johnath> mrz: working here now - I can reach people again
08:58 < johnath> and mail's working too

Comment 10

[Aravind Gottipati [:aravind]]

Should be all fixed now.  Sorry about the problems guys, I hope to get down to the bottom of this thing today.