[ispnet-announce] Manhole fire and reduced redundancy

[Bob Tinkelman]

Work on this maintenance has begun.

We'll send another email when work is complete.

- Bob




> Timeline summary
> ----------------

> 06/07/18 14:40 EDT - Fire in manhole outside 85 Tenth Ave

> Damaged circuits included redundant pop-to-pop circuit
> (85 Tenth, NY, to 165 Halsey, Newark) and circuit to
> one of three major upstreams (NTT).

> All operations continued over fall-back paths with
> reduced redundancy.

> 06/08/18 10:24 EDT - Service restored: POP-to-POP circuit

> Temporary repair.

> 06/09/18 02:40 EDT - Service restored: ISPnet-to-NTT

> As this service restoral followed a notice from the carrier
> that they had a splice crew working in the affected manhole,
> we believe this is a permanent fix (not a temporary work-
> around) but we have not been able to verify this yet with
> the carrier.


> Current status
> --------------

> The network is operational at full redundancy.

> During the course of the outage, no customers lost service.

> A few customers reported performance problems, mostly related
> to tunnels operating over fall-back paths.  These problems
> were addressed as they were reported.  We believe there are
> no currently open issues.


> Next Steps
> ----------

> We will work with the carriers for both circuits to
> schedule maintenance windows for fiber splcing, if
> required, in cases of temporary repairs.




> Customers experiencing any service related issues should
> contact ISPnet Support.
> --
> Bob Tinkelman          <bob at tink.com>
> ISPnet, Inc.    http://www.ispnet.net
> +1 (718) 464-4747  office
> +1 (800) 806-NETS  toll free