[ispnet-announce] Manhole fire and reduced redundancy

[Bob Tinkelman]

This email is a status update to our earlier posting,
which is quoted below for reference.

At 10:24 this morning, full redundant connectivity was
restored to the ISPnet backbone, so we are again in
the normal state where no customer service will be
disrupted by the failure of a single ISPnet backbone
circuit.

We have been back to this fully-redundant state for
the past two hours and everything appears stable.

The fiber vendor for the affected service informed us
that the fix they installed is temporary.  They
"routed around" the problem manhole.  It will probably
be 24 to 48 hours before all the carriers with damaged
fiber will get the needed access.  At some point there
will be a cut-over when our carrier can resplice our
circuit along its original path.  This will involve a
very brief period where we are operating with
reduced redundancy.

The only major ISPnet circuit not yet restored is the
connection between ISPnet at 85 Tenth Ave and NTT at
111 8th Ave.

Some ISPnet customers who use tunnels that normally
connected to ISPnet via NTT reported problems with
their fall-back paths.  We were able to address these
problems by rerouting the affedted traffic through
Hurricane Electric.

Any customers still experiencing any problems should
contact ISPnet Support.

--
Bob Tinkelman          <bob at tink.com>
ISPnet, Inc.    http://www.ispnet.net
+1 (718) 464-4747  office
+1 (800) 806-NETS  toll free





> This is an advisory notice only.

> Summary
> -------
> ISPnet is operating at reduced redunancy due to a number of
> carrier and vendor issues.

> Details
> -------
> At around 14:40 EDT today, several circuits went down as a result
> of a manhole fire at the corner of 9th Ave and 15th St (just
> outside 85 Tenth Ave, ISPnet's Manhattan POP).

> One of these circuits was one leg of the triangle connecting
> ISPnet's three major NYC-area POPs.  Traffic failed-over,
> automatically, to the other two legs of the triangle.

> Another circuit was our connection to NTT at 111 8th Ave.  ISPnet
> has three major upstreams in the NY area, and all traffic that
> normally utilizes the NTT path failed over to use one of the
> other two.

> ISPnet connects with over 100 other networks via NYC-area peering
> points (NYIIX and DE-CIX-NY).  A number of these peers are
> probably affected by the manhole fire.  In addition, the NYIIX
> has been having platform-related problems at 111 8th Ave.  As a
> result, the number of active peering sessions has been reduced by
> about 10%.  Traffic that would normally utilize these peers has
> failed over other peers or to one of our upstreams.

> Overall, our fall-backs are working as designed, with more than
> sufficient available capacity.

> Our network is, however operating with a lower level of
> redundancy.  Under normal circustances there are no single points
> of failure (other than for single-homed, single-connected
> customers).  At this point, a failure on one of the remaining
> legs of our backbone triangle would cause loss of service for
> some customers.

> The latest reports regarding the fire indicated that the NY Fire
> Department had not yet cleared the scene to allow access to
> repair crews of the various carriers.  No ETR is yet available
> for any of the affected circuits.

> --
> Bob Tinkelman          <bob at tink.com>
> ISPnet, Inc.    http://www.ispnet.net
> +1 (718) 464-4747  office
> +1 (800) 806-NETS  toll free