RESOLVED FIXED

Status

Infrastructure & Operations
MOC: Problems
RESOLVED FIXED
3 months ago
3 months ago

People

(Reporter: arr, Assigned: ryanc)

Tracking

Details

(Reporter)

Description

3 months ago
I'm unable to reach https://nagios1.private.releng.mdc1.mozilla.com/releng-mdc1/ from the VPN. I'm assuming it got denied in the default-deny.
(Reporter)

Comment 1

3 months ago
Hm, actually, it does ask me to auth, but eventually gives me an ISE:

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator at root@localhost to inform them of the time this error occurred, and the actions you performed just before this error.

More information about this error may be available in the server error log.
Assignee: nobody → rchilds
There appears to be an issue getting proper responses back from the LDAP VIP, perhaps firewall related. I'll gather some debug info.
(Assignee)

Updated

3 months ago
Status: NEW → ASSIGNED
Summary: https://nagios1.private.releng.mdc1.mozilla.com/releng-mdc1/ is non-functional → https://nagios1.*.mdc1.mozilla.com/* is non-functional
(Assignee)

Comment 3

3 months ago
(In reply to Keegan Ferrando [:fauweh] from comment #2)
> There appears to be an issue getting proper responses back from the LDAP
> VIP, perhaps firewall related. I'll gather some debug info.

Yeah, the response from the vip doesn't look consistent,

[rchilds@nagios1.private.mdc1 rchilds]# date && ldapsearch -h 'ldap-slave.vips.private.mdc1.mozilla.com/dc=mozilla?mail?sub?(|(&(objectClass=inetOrgPerson)(|(o:dn:=org)(o:dn:=com)(o:dn:=net)))(uid=mntos))' -p 389 -D 'uid=bindweb3,ou=logins,dc=mozilla' -w 'X' | tail -n 2
Thu Jul 13 03:51:47 UTC 2017
# numResponses: 4438
# numEntries: 4437
[rchilds@nagios1.private.mdc1 rchilds]# date && ldapsearch -h 'ldap-slave.vips.private.mdc1.mozilla.com/dc=mozilla?mail?sub?(|(&(objectClass=inetOrgPerson)(|(o:dn:=org)(o:dn:=com)(o:dn:=net)))(uid=mntos))' -p 389 -D 'uid=bindweb3,ou=logins,dc=mozilla' -w 'X' | tail -n 2
Thu Jul 13 03:51:50 UTC 2017
...


VS


[rchilds@nagios3.private.scl3 ~]$ date && ldapsearch -h 'ldap-slave.vips.private.scl3.mozilla.com/dc=mozilla?mail?sub?(|(&(objectClass=inetOrgPerson)(|(o:dn:=org)(o:dn:=com)(o:dn:=net)))(uid=mntos))' -p 389 -D 'uid=bindweb3,ou=logins,dc=mozilla' -w 'X' | tail -n 2
Thu Jul 13 03:50:36 UTC 2017
# numResponses: 4438
# numEntries: 4437
[rchilds@nagios3.private.scl3 ~]$ date && ldapsearch -h 'ldap-slave.vips.private.scl3.mozilla.com/dc=mozilla?mail?sub?(|(&(objectClass=inetOrgPerson)(|(o:dn:=org)(o:dn:=com)(o:dn:=net)))(uid=mntos))' -p 389 -D 'uid=bindweb3,ou=logins,dc=mozilla' -w 'X' | tail -n 2
Thu Jul 13 03:50:38 UTC 2017
# numResponses: 4438
# numEntries: 4437


Then from the ldap server, sometimes it gets the request, other times nothing even appears in the logs,

[rchilds@slave1.ldap.mdc1 ~]$ sudo tail -F /var/log/ldap/ldap.log
Jul 13 03:51:42 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16198 op=2 UNBIND
Jul 13 03:51:42 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16198 fd=29 closed
Jul 13 03:51:47 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16199 fd=29 ACCEPT from IP=10.48.75.40:42246 (IP=0.0.0.0:389)
Jul 13 03:51:47 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16199 op=0 BIND dn="uid=bindweb3,ou=logins,dc=mozilla" method=128
Jul 13 03:51:47 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16199 op=0 BIND dn="uid=bindweb3,ou=logins,dc=mozilla" mech=SIMPLE ssf=0
Jul 13 03:51:47 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16199 op=0 RESULT tag=97 err=0 text=
Jul 13 03:51:47 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16199 op=1 SRCH base="dc=mozilla" scope=2 deref=0 filter="(objectClass=*)"
Jul 13 03:51:49 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16199 op=1 SEARCH RESULT tag=101 err=0 nentries=4437 text=
Jul 13 03:51:49 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16199 op=2 UNBIND
Jul 13 03:51:49 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16199 fd=29 closed

...

Jul 13 03:53:54 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16201 fd=39 ACCEPT from IP=10.48.75.40:42472 (IP=0.0.0.0:389)
Jul 13 03:53:54 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16201 op=0 BIND dn="uid=bindweb3,ou=logins,dc=mozilla" method=128
Jul 13 03:53:54 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16201 op=0 BIND dn="uid=bindweb3,ou=logins,dc=mozilla" mech=SIMPLE ssf=0
Jul 13 03:53:54 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16201 op=0 RESULT tag=97 err=0 text=
Jul 13 03:53:54 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16201 op=1 SRCH base="dc=mozilla" scope=2 deref=0 filter="(objectClass=*)"
Jul 13 03:53:55 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16201 op=1 SEARCH RESULT tag=101 err=0 nentries=4437 text=
Jul 13 03:53:55 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16201 op=2 UNBIND
Jul 13 03:53:55 slave1.ldap.mdc1.mozilla.com slapd[1319]: conn=16201 fd=39 closed
Component: MOC: Service Requests → MOC: Problems
Depends on: 1379780
(Assignee)

Comment 4

3 months ago
Since this was busted, I thought it'd make sense to just setup auth0 on these, which happened for all mdc1 instances in bug 1373186#c7.

Amy, can you test?
Flags: needinfo?(arich)
(Reporter)

Comment 5

3 months ago
I can confirm that I can log into the mdc1 nagios instance now.
Flags: needinfo?(arich)
(Assignee)

Comment 6

3 months ago
Great, calling this good then.
Status: ASSIGNED → RESOLVED
Last Resolved: 3 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.