869667 - SFO1 twtc connectivity issue for 1 min

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: NetOps, task)

Description

[Ravi Pina [:ravi]]

+++ This bug was initially created as a clone of Bug #868117 +++

Connectivity across the sfo1 twtc circuit was degraded briefly from ~2126-2127 UTC with 70ms time and ~20% loss.

Comment 1

[Ravi Pina [:ravi]]

Circuit looks clean in pao1

[e-mail]1> show interfaces ge-1/1/2 extensive | grep error
  Link-level type: Flexible-Ethernet, MTU: 1518, Speed: 1000mbps, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled,
  Input errors:
    Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0,
    Resource errors: 0
  Output errors:
    Carrier transitions: 3, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0
    CRC/Align errors                         0                0
    FIFO errors                              0                0
    Output packet error count                                 0

Comment 2

[Aaron]

Old case w/ twtc was auto-closed.  Opened a new case with them to at least get their help tacking this down.  For reference, previous cases with them are:

CI000889906 
CI000861369

Comment 3

[Ravi Pina [:ravi]]

sfo side is also clean

[e-mail]> show interfaces ge-1/1/3 extensive | grep error
  Link-level type: Ethernet, MTU: 1514, Speed: 1000mbps, Duplex: Full-Duplex, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled,
  Input errors:
    Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0,
    Resource errors: 0
  Output errors:
    Carrier transitions: 5, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0
    CRC/Align errors                         0                0
    FIFO errors                              0                0

Comment 4

[Aaron]

Attachment: TW monitoring snapshot

TW provided the attached snapshots of their monitoring on the circuit.  I'm asking for further information:

What time zone are the times listed on the graph?  Also, these show errors and throughput.  You mentioned some part of the system does active checks (pings) to ensure packet delivery.  Are there logs of these active probes?  Part of our issue was a momentary jump in the latency of the circuit.  Do you have any monitoring that would give an indication of latency (or lack thereof?)

Comment 5

[Aaron]

They've referred us to our account manager =/

Hello,


Sorry for the delay. For more in-depth circuit monitoring, please get with your account executive who I believe is Brian Hou/ 303-566-1000. He will be able to set you folks up with enhanced monitoring which will enable you to see what we see.

Thanks,

Feridoun “Lee” Salehi
Customer Assurance Technician
5700 S. Quebec St.
Greenwood Village, CO 80111

Comment 6

[Ravi Pina [:ravi]]

We'll close this out, but in the meantime I've mailed our account manager and we will look to increase our already extensive monitoring we have in place to help pinpoint the span that is having issues.