ISPnet service interruption

[Bob Tinkelman]

Between 2:30 and 3:00 PM Eastern, today, we experienced a
problem centered on one of our routers at 25 Broadway:
     gw1.nycmnycz.ispnetinc.net

The customer-visible symptoms were degraded routing and, for
some customers, complete loss of connectivity.

The cause of the problem has been identified.  It was a
combination of two factors:

    o   A failure on a bgp-speaking customer router,
        causing it to bgp-announce to ISPnet an
        incorrect set of routes, and

     o  A failure of the portion of the ISPnet router
        configuration (for this particular customer)
        that was designed to protect against exactly
        this situation.
        
We haven't completely analyzed all the details.  At this
point it's possible that part of the issue relates to the
version of the cisco IOS we loaded onto our routers last
weekend.

However, we have updated our configurations to provide
additional safeguards against any recurrances of the problem.


You have my sincere apologies for any problems caused, and my
assurance that we will do what is needed to prevent this or
similar service interruptions in the future.

--
Bob Tinkelman          <bob at tink.com>
ISPnet, Inc.  http://www.ispnetinc.net

+1 (718) 464-4747  office
+1 (800) 806-NETS  toll free
+1 (718) 217-9407  fax