1138360 - London office internet outage.

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: NetOps: Office Carrier, task)

Description

[Ryan Watson [:w0ts0n]]

Internet has been reported as down. Only a single line in the office. Currently trying to find route cause.

Comment 1

[Peter Radcliffe [:pir]]

Level3 report BGP down.

Comment 2

[Peter Radcliffe [:pir]]

Level3 ticket number 8956372

Comment 3

[Peter Radcliffe [:pir]]

Level 3 Communications

This is to confirm that Level 3 ticket 8956372 has been created regarding your service.

Organization: Mozilla Corporation
Billing Account Name: MOZILLA CORPORATION
Customer Ticket ID: 1138360
Service Identifier: FRO2004822096
Ticket Created On: FRO2004822096
Product: Dedicated Internet Access
Bandwidth: FE
Symptom: BGP Bouncing or Down
Ticket Opened: 2015-03-02 11:15 GMT
A Location Name: Mozilla
A Location Address: 101 SAINT MARTIN'S LANE, LONDON, UNKNOWN United Kingdom
Ticket Opened: 2015-03-02 11:15 GMT
Z Location Name: ENGLAND, LONDON - GLOBAL CROSSING POP
Z Location Address: EAST INDIA DOCK ROAD, LONDON BOROUGH OF TOWER HAMLETS, EN UNITED KINGDOM
Ticket Opened: 2015-03-02 11:15 GMT

Notes History:

03/02/2015 11:15:55 GMT - Customer Testing Triage = PRODUCT IP
IP Address: 208.48.238.169/30
Date and Time Trouble Began: 2015-03-02
Customer Equipment Validated: No
Intrusive Testing Allowed? : Yes
Problem Detail:

- DOWN HARD
- No Planned Maint noted
- OK to test
Symtom: BGP Bouncing or Down
BGP Peer IP Address :208.48.238.169/30
Does Circuit Go Down when BGP Drops? :Yes
AS#: 00.00.00.00
Does BGP Circuit have MD5 Password? :No

Comment 4

[Peter Radcliffe [:pir]]

No updates from Level3 as yet.

Comment 5

[Peter Radcliffe [:pir]]

Escalated with Level3 as it was still waiting to be picked up by an available tech.

Comment 6

[James Barnell [:jbarnell]]

I've escalated with Level3.  The ticket, at 7:32 was not assigned which is a service level breach.  They will update within an hour.  I've also authorized testing and onsite if needed.

Comment 7

[Peter Radcliffe [:pir]]

Update from Level3:



Level 3 Communications

This is to confirm that an update has been recorded on Level 3 ticket 8956372.

TICKET UPDATE
Update:

The customer James called in and I gave him the update:

1. I am going thru the patch and so farI am seeing BGP down.


Next step:

1. I am going to open a ticket with the local carrier for further investigation.

Thank you

Comment 8

[Peter Radcliffe [:pir]]

Level 3 Communications

This is to confirm that an update has been recorded on Level 3 ticket 8956372.

TICKET UPDATE
Update: e-mail

I have opened a ticket with our team in Europe to have them open a ticket with the local carrier for further investigation.

I will update you in about 1hr.

Comment 9

[Peter Radcliffe [:pir]]

Alerts recovering, tentatively link looks to be back up.

Comment 10

[Peter Radcliffe [:pir]]

"Tentatively" was too positive. Connection down again.

Comment 11

[Peter Radcliffe [:pir]]

"At this time, service has been restored to your office.  We do not anticipate any lingering problems for start of business tomorrow."

I'm in the office right now, wifi, net, connecting to vidyo, etc, all working as usual.