1380373 - https://nagios1.*.mdc1.mozilla.com/* is non-functional

Status: RESOLVED FIXED

(Infrastructure & Operations :: MOC: Problems, task)

Description

[Amy Rich [:arr] [:arich]]

I'm unable to reach https://nagios1.private.releng.mdc1.mozilla.com/releng-mdc1/ from the VPN. I'm assuming it got denied in the default-deny.

Comment 1

[Amy Rich [:arr] [:arich]]

Hm, actually, it does ask me to auth, but eventually gives me an ISE:

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator at root@localhost to inform them of the time this error occurred, and the actions you performed just before this error.

More information about this error may be available in the server error log.

Comment 2

[Keegan Ferrando [:fauweh]]

There appears to be an issue getting proper responses back from the LDAP VIP, perhaps firewall related. I'll gather some debug info.

Comment 3

[Ryan C [:ryanc] (UTC-4)]

(In reply to Keegan Ferrando [:fauweh] from comment #2)
> There appears to be an issue getting proper responses back from the LDAP
> VIP, perhaps firewall related. I'll gather some debug info.

Yeah, the response from the vip doesn't look consistent,

[[e-mail]1 rchilds]# date && ldapsearch -h 'ldap-slave.vips.private.mdc1.mozilla.com/dc=mozilla?mail?sub?(|(&(objectClass=inetOrgPerson)(|(o:dn:=org)(o:dn:=com)(o:dn:=net)))(uid=mntos))' -p 389 -D 'uid=bindweb3,ou=logins,dc=mozilla' -w 'X' | tail -n 2
Thu Jul 13 03:51:47 UTC 2017
# numResponses: 4438
# numEntries: 4437
[[e-mail]1 rchilds]# date && ldapsearch -h 'ldap-slave.vips.private.mdc1.mozilla.com/dc=mozilla?mail?sub?(|(&(objectClass=inetOrgPerson)(|(o:dn:=org)(o:dn:=com)(o:dn:=net)))(uid=mntos))' -p 389 -D 'uid=bindweb3,ou=logins,dc=mozilla' -w 'X' | tail -n 2
Thu Jul 13 03:51:50 UTC 2017
...


VS


[[e-mail]3 ~]$ date && ldapsearch -h 'ldap-slave.vips.private.scl3.mozilla.com/dc=mozilla?mail?sub?(|(&(objectClass=inetOrgPerson)(|(o:dn:=org)(o:dn:=com)(o:dn:=net)))(uid=mntos))' -p 389 -D 'uid=bindweb3,ou=logins,dc=mozilla' -w 'X' | tail -n 2
Thu Jul 13 03:50:36 UTC 2017
# numResponses: 4438
# numEntries: 4437
[[e-mail]3 ~]$ date && ldapsearch -h 'ldap-slave.vips.private.scl3.mozilla.com/dc=mozilla?mail?sub?(|(&(objectClass=inetOrgPerson)(|(o:dn:=org)(o:dn:=com)(o:dn:=net)))(uid=mntos))' -p 389 -D 'uid=bindweb3,ou=logins,dc=mozilla' -w 'X' | tail -n 2
Thu Jul 13 03:50:38 UTC 2017
# numResponses: 4438
# numEntries: 4437


Then from the ldap server, sometimes it gets the request, other times nothing even appears in the logs,

[[e-mail]1 ~]$ sudo tail -F /var/log/ldap/ldap.log
Jul 13 03:51:42 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16198 op=2 UNBIND
Jul 13 03:51:42 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16198 fd=29 closed
Jul 13 03:51:47 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16199 fd=29 ACCEPT from IP=10.48.75.40:42246 (IP=0.0.0.0:389)
Jul 13 03:51:47 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16199 op=0 BIND dn="uid=bindweb3,ou=logins,dc=mozilla" method=128
Jul 13 03:51:47 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16199 op=0 BIND dn="uid=bindweb3,ou=logins,dc=mozilla" mech=SIMPLE ssf=0
Jul 13 03:51:47 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16199 op=0 RESULT tag=97 err=0 text=
Jul 13 03:51:47 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16199 op=1 SRCH base="dc=mozilla" scope=2 deref=0 filter="(objectClass=*)"
Jul 13 03:51:49 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16199 op=1 SEARCH RESULT tag=101 err=0 nentries=4437 text=
Jul 13 03:51:49 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16199 op=2 UNBIND
Jul 13 03:51:49 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16199 fd=29 closed

...

Jul 13 03:53:54 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16201 fd=39 ACCEPT from IP=10.48.75.40:42472 (IP=0.0.0.0:389)
Jul 13 03:53:54 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16201 op=0 BIND dn="uid=bindweb3,ou=logins,dc=mozilla" method=128
Jul 13 03:53:54 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16201 op=0 BIND dn="uid=bindweb3,ou=logins,dc=mozilla" mech=SIMPLE ssf=0
Jul 13 03:53:54 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16201 op=0 RESULT tag=97 err=0 text=
Jul 13 03:53:54 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16201 op=1 SRCH base="dc=mozilla" scope=2 deref=0 filter="(objectClass=*)"
Jul 13 03:53:55 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16201 op=1 SEARCH RESULT tag=101 err=0 nentries=4437 text=
Jul 13 03:53:55 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16201 op=2 UNBIND
Jul 13 03:53:55 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16201 fd=39 closed

Comment 4

[Ryan C [:ryanc] (UTC-4)]

Since this was busted, I thought it'd make sense to just setup auth0 on these, which happened for all mdc1 instances in bug 1373186#c7.

Amy, can you test?

Comment 5

[Amy Rich [:arr] [:arich]]

I can confirm that I can log into the mdc1 nagios instance now.

Comment 6

[Ryan C [:ryanc] (UTC-4)]

Great, calling this good then.