SFO1 twtc connectivity issue for 1 min

RESOLVED FIXED

Status

Infrastructure & Operations
NetOps
RESOLVED FIXED
5 years ago
5 years ago

People

(Reporter: ravi, Assigned: Aaron)

Tracking

Details

(Whiteboard: [TWTC# CI000892949])

Attachments

(1 attachment)

447.12 KB, application/x-zip-compressed
Details
(Reporter)

Description

5 years ago
+++ This bug was initially created as a clone of Bug #868117 +++

Connectivity across the sfo1 twtc circuit was degraded briefly from ~2126-2127 UTC with 70ms time and ~20% loss.
(Reporter)

Comment 1

5 years ago
Circuit looks clean in pao1

root@border1.pao1> show interfaces ge-1/1/2 extensive | grep error
  Link-level type: Flexible-Ethernet, MTU: 1518, Speed: 1000mbps, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled,
  Input errors:
    Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0,
    Resource errors: 0
  Output errors:
    Carrier transitions: 3, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0
    CRC/Align errors                         0                0
    FIFO errors                              0                0
    Output packet error count                                 0
(Assignee)

Comment 2

5 years ago
Old case w/ twtc was auto-closed.  Opened a new case with them to at least get their help tacking this down.  For reference, previous cases with them are:

CI000889906 
CI000861369
Whiteboard: [TWTC# CI000892949]
(Reporter)

Comment 3

5 years ago
sfo side is also clean

ravi@switch1.sfo1.mozilla.net> show interfaces ge-1/1/3 extensive | grep error
  Link-level type: Ethernet, MTU: 1514, Speed: 1000mbps, Duplex: Full-Duplex, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled,
  Input errors:
    Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0,
    Resource errors: 0
  Output errors:
    Carrier transitions: 5, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0
    CRC/Align errors                         0                0
    FIFO errors                              0                0
(Assignee)

Comment 4

5 years ago
Created attachment 747556 [details]
TW monitoring snapshot

TW provided the attached snapshots of their monitoring on the circuit.  I'm asking for further information:

What time zone are the times listed on the graph?  Also, these show errors and throughput.  You mentioned some part of the system does active checks (pings) to ensure packet delivery.  Are there logs of these active probes?  Part of our issue was a momentary jump in the latency of the circuit.  Do you have any monitoring that would give an indication of latency (or lack thereof?)
(Assignee)

Updated

5 years ago
Assignee: network-operations → ahill
(Assignee)

Comment 5

5 years ago
They've referred us to our account manager =/

Hello,


Sorry for the delay. For more in-depth circuit monitoring, please get with your account executive who I believe is Brian Hou/ 303-566-1000. He will be able to set you folks up with enhanced monitoring which will enable you to see what we see.

Thanks,

Feridoun “Lee” Salehi
Customer Assurance Technician
5700 S. Quebec St.
Greenwood Village, CO 80111
(Reporter)

Comment 6

5 years ago
We'll close this out, but in the meantime I've mailed our account manager and we will look to increase our already extensive monitoring we have in place to help pinpoint the span that is having issues.
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.