[ispnet-announce] Manhole fire and reduced redundancy

[Bob Tinkelman]

This is an advisory notice only.

Summary
-------
ISPnet is operating at reduced redunancy due to a number of
carrier and vendor issues.

Details
-------
At around 14:40 EDT today, several circuits went down as a result
of a manhole fire at the corner of 9th Ave and 15th St (just
outside 85 Tenth Ave, ISPnet's Manhattan POP).

One of these circuits was one leg of the triangle connecting
ISPnet's three major NYC-area POPs.  Traffic failed-over,
automatically, to the other two legs of the triangle.

Another circuit was our connection to NTT at 111 8th Ave.  ISPnet
has three major upstreams in the NY area, and all traffic that
normally utilizes the NTT path failed over to use one of the
other two.

ISPnet connects with over 100 other networks via NYC-area peering
points (NYIIX and DE-CIX-NY).  A number of these peers are
probably affected by the manhole fire.  In addition, the NYIIX
has been having platform-related problems at 111 8th Ave.  As a
result, the number of active peering sessions has been reduced by
about 10%.  Traffic that would normally utilize these peers has
failed over other peers or to one of our upstreams.

Overall, our fall-backs are working as designed, with more than
sufficient available capacity.

Our network is, however operating with a lower level of
redundancy.  Under normal circustances there are no single points
of failure (other than for single-homed, single-connected
customers).  At this point, a failure on one of the remaining
legs of our backbone triangle would cause loss of service for
some customers.

The latest reports regarding the fire indicated that the NY Fire
Department had not yet cleared the scene to allow access to
repair crews of the various carriers.  No ETR is yet available
for any of the affected circuits.

--
Bob Tinkelman          <bob at tink.com>
ISPnet, Inc.    http://www.ispnet.net
+1 (718) 464-4747  office
+1 (800) 806-NETS  toll free