Power issues at 7 Teleport Drive, Staten Island

[Bob Tinkelman]

This email is a recap regarding power issues that occured today
at ISPnet's POP at Telehouse's facility at 7 Teleport Drive,
Staten Island.

The issue caused service interruptions to some customers.
The affected customers were those with equipment located at 7
Teleport Drive (colocated with Telehosue or with ISPnet) and who
have only a single cross-connect to ISPnet.  Dual-connected
customers were not affected.

There were two outages.

The first occured at 5:35 AM and lasted for about a half hour.
The trigger was a "power hit" that Telehouse suffered on their
ConEd electrical service.  Their power units failed-over,
apparently in a way that triggered the circuit breakers to
both electrical circuits to one of ISPnet's cabients to trip.
Service remained down (in that cabinet) until Telehouse reset the
circuit breakers.

The second out occured at approximately 18:45.  ISPnet was
on-site and working to isolate the cause of the problem.  While
we were doing what should have been a non-service-affecting test,
we lost power to one of our switch stacks.

This failure pointed to a problem with one of our Automated
Transfer Switches.  We believe the damage occured duing this
morning's outage.

We were able to recofigure things so that everything is up and we
are not vulnerable to an outage for the same reason as this
morning's.

We will need to work with Telehouse and the transfer switch
manufacturer to come up with a new configuration that will allow
us full protection but not expose us to the same problems.

We will also need to replace the transfer switch in which we
diagnosed the problem.

We will do this work in announced maintenance windows.

In the mean time, we are confident that we will not have problems
so long as Telehouse's power distribution units are able to
provide continuous power.

We greatly regret the inconvenience caused to some of our
customers today.  We have tried to react as quickly as practical
to diagnose the cause of the failures and to remedy the
situation.

--
Bob Tinkelman          <bob at tink.com>
ISPnet, Inc.    http://www.ispnet.net
+1 (718) 464-4747  office
+1 (800) 806-NETS  toll free