par1.tier2 (interoute) outage pbx1.voip.par1.mozilla.com:Asterisk Trunk - SIP/Interoute-SIP is CRITICAL: SIP/Interoute-SIP UNREACHABLE

RESOLVED FIXED

Status

Infrastructure & Operations
Telecom
RESOLVED FIXED
2 years ago
6 months ago

People

(Reporter: jedi, Assigned: justdave)

Tracking

Details

(Reporter)

Description

2 years ago
<@nagios-scl3> Fri 20:31:35 PDT [5072] pbx1.voip.par1.mozilla.com:Asterisk Trunk - SIP/Interoute-SIP is CRITICAL: SIP/Interoute-SIP UNREACHABLE
                     (http://m.mozilla.org/Asterisk+Trunk+-+SIP/Interoute-SIP)


[mdevney@pbx1.voip.par1 ~]$ sudo asterisk -rx "sip show peers like Interoute-SIP"
Name/username             Host                                    Dyn Forcerport ACL Port     Status
Interoute-SIP             84.233.180.73                                N             5060     UNREACHABLE
1 sip peers [Monitored: 0 online, 1 offline Unmonitored: 0 online, 0 offline]
(Reporter)

Comment 1

2 years ago
00:07 < justdave_cloud> Yeah, mtr to admin1.par1.mozilla.com dies in interoute's net
00:08 < justdave_cloud> Follow playbook for Interoute being down
(Reporter)

Comment 2

2 years ago
- No maintenances on whistleping.
- CAN ping fw1.par1.mozilla.net
- CAN ping fw1.tier1.par1.mozilla.net
- can NOT ping ping fw1.tier2.par1.mozilla.net
(Reporter)

Comment 3

2 years ago
<@nagios-scl3> Fri 21:39:58 PDT [5101] pbx1.voip.par1.mozilla.com:Asterisk Trunk - SIP/Interoute-SIP is OK: SIP/Interoute-SIP OK (11 ms)


00:39 < justdave> hey, well how about that
00:39 < justdave> Interoute's SIP endpoint is routable over level3
00:39 < justdave> SIP is back online :)


Leaving this bug open for the wider outage, though.
(Reporter)

Updated

2 years ago
Summary: pbx1.voip.par1.mozilla.com:Asterisk Trunk - SIP/Interoute-SIP is CRITICAL: SIP/Interoute-SIP UNREACHABLE → par1.tier2 (interoute) outage pbx1.voip.par1.mozilla.com:Asterisk Trunk - SIP/Interoute-SIP is CRITICAL: SIP/Interoute-SIP UNREACHABLE
(Reporter)

Comment 4

2 years ago
00:40 < justdave> it's going to fail again when Interoute comes back up, we'll have to kick it again to switch it back to Interoute
00:42 < justdave> this is what you have to do to make it switch back:
00:42 < justdave> clear security flow session destination-prefix 84.233.180.73/32
00:42 < justdave> I think it needs someone with netops access though
(Reporter)

Comment 5

2 years ago
Interoute ticket # 1-1867981604
(Reporter)

Comment 6

2 years ago
 <@nagios-scl3> Sat 00:12:58 PDT [5005] fw1.tier2.par1.mozilla.net (195.81.232.10) is UP :PING OK - Packet loss = 0%, RTA = 147.61 ms
par1 tier2 (interoute) had another small outage today...

14:31 <@nagios-scl3> Tue 14:31:16 PDT [5540] admin1.par1.mozilla.com (89.202.203.49) is DOWN :PING CRITICAL - Packet loss = 100%
14:32 <@nagios-scl3> Tue 14:32:07 PDT [5541] fw1.tier2.par1.mozilla.net (195.81.232.10) is DOWN :PING CRITICAL - Packet loss = 100%


14:41 <@nagios-scl3> Tue 14:41:57 PDT [5552] admin1.par1.mozilla.com (89.202.203.49) is UP :PING OK - Packet loss = 0%, RTA = 147.82 ms
14:41 <@nagios-scl3> Tue 14:41:58 PDT [5553] fw1.tier2.par1.mozilla.net (195.81.232.10) is UP :PING OK - Packet loss = 0%, RTA = 147.24 ms
This was fixed a long time ago. (and has been re-broken and re-fixed several times since)
Assignee: telecom → justdave
Status: NEW → RESOLVED
Last Resolved: 6 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.