6/6/2019 – v2 NYJ004 Network Failure

Affected Services

None. Connectivity to devices registered to NYJ would’ve been affected. However services were already shut down for maintenance.

Event Summary

On June 6th, 2019, at 23:24 EDT, there was a fiber outage reported at the NYJ004 Data Center. This impacted one of the redundant backbone spans providing upstream connectivity to NYJ004. Investigation into the cause of the fiber outage, and restoration of this span, are still underway by the fiber provider. A few hours later at 01:34 EDT a second event took place at a partner site on the remaining redundant backbone span at the NYJ004 location. This resulted in losing upstream connectivity and isolating the NYJ004 Data Center. At 04:39 EDT INAP Senior Engineers were able to work with field technicians at the partner site to restore connectivity to NYJ004. The root cause of this event is also still under investigation. In summary, these series of events were independent of each other; they are on separate carrier networks, on diverse fiber paths and are serviced by different fiber providers. The INAP RFO will be updated when both partners RFO’s are received.

Event Timeline

June 6th, 2019

• 23:24 Major Incident Fiber outage on Backbone 2. IMPACT Loss of Redundancy

June 7th, 2019

• 00:16 Fiber provider acknowledges Major Incident and investigating fiber cut.
• 01:34 Secondary incident occurred, which impacted a diverse carrier of Backbone 1. IMPACT START NYJ004 Isolated
• 01:45 NOC Management paged.
• 02:00 Major Incident declared, senior engineers paged, and customers notified.
• 02:30 INAP Engineers troubleshooting connectivity on Backbone 1.
• 03:30 Continuing restoration process of Backbone 1.
• 04:39 Fiber issue restored on Backbone 1 at an INAP partner Data Center. IMPACT END Connectivity restored
• 05:00 Backbone 2 Fiber outage ongoing. The Fiber provider located the fiber cut and is dispatching additional crews.
• 08:53 Backbone 2. All fiber crews are at the cut location and are beginning repair work. 
• 11:27 Backbone 2. Fiber provider reports there are 1200 feet of damaged cable to repair.
• 14:28 Backbone 2. The Fiber provider has completed laying a new span of fiber and is working on securing it for splicing.

Root Cause Analysis

Two separate and isolated events occurred which severed both redundant backbone spans that provide connectivity to the NYJ004 Data Center.

Future Preventative Action

1. INAP is doing a site audit and root cause investigation to the outage that occurred at the partner site.
2. INAP has requested the fiber carrier send a formal RFO on the current fiber cut impacting the redundant link.
3. INAP is deploying updates to the NYJ004 infrastructure to provide local egress connectivity.
4. An updated RFO will be sent upon review of the two partner RFO