[ispnet-announce] Manhole fire and reduced redundancy

[Bob Tinkelman]

Timeline summary
----------------

06/07/18 14:40 EDT - Fire in manhole outside 85 Tenth Ave

Damaged circuits included redundant pop-to-pop circuit
(85 Tenth, NY, to 165 Halsey, Newark) and circuit to
one of three major upstreams (NTT).

All operations continued over fall-back paths with
reduced redundancy.

06/08/18 10:24 EDT - Service restored: POP-to-POP circuit

Temporary repair.

06/09/18 02:40 EDT - Service restored: ISPnet-to-NTT

As this service restoral followed a notice from the carrier
that they had a splice crew working in the affected manhole,
we believe this is a permanent fix (not a temporary work-
around) but we have not been able to verify this yet with
the carrier.


Current status
--------------

The network is operational at full redundancy.

During the course of the outage, no customers lost service.

A few customers reported performance problems, mostly related
to tunnels operating over fall-back paths.  These problems
were addressed as they were reported.  We believe there are
no currently open issues.


Next Steps
----------

We will work with the carriers for both circuits to
schedule maintenance windows for fiber splcing, if
required, in cases of temporary repairs.




Customers experiencing any service related issues should
contact ISPnet Support.
--
Bob Tinkelman          <bob at tink.com>
ISPnet, Inc.    http://www.ispnet.net
+1 (718) 464-4747  office
+1 (800) 806-NETS  toll free