ISPnet routing problems, 20-Nov-2012

ISPnet Support support at ISPNETINC.NET
Wed Nov 21 09:28:47 EST 2012


The purpose of this email to document a problem which occured last night.

Starting at approximately 7:30 PM EST and continuing until 10:00 PM, there
was routing instability on ISPnet routers at two of our nodes:

    nycmnycz - 25 Broadway, New York, NY
    nwrknjmd - 165 Halsey St, Newark, NJ

During the above period, certain parts of the Internet were not reachable
from ISPnet.  Most, but not all of the affected sites were direct customers
of Cogent.

ISPnet has several local peering links with Cogent in the New York area,
some on routers at the two above POPs.

At 7:30 PM last night, Cogent made an error while updating the configuration
of one of their routers that peer with ISPnet.  The result was that one of
our bgp sessions with them went down.  At that point, our traffic to Cogent
should have failed over to a secondary link.  However, there was a
configuration error on the ISPnet routers involved which became exposed
only in this particular failure scenario.

At 10:00 PM last night, we shut down all local peering with Cogent,
reestablishing functional routing to the entire net, including Cogent sites,
although with slightly longer route paths.

At 8:00 AM today, we worked with Cogent to get our primary local peering
path them them working again.

We will leave our secondary path to Cogent disabled until we can schedule a
time to test a corrected configuration, most likely as part of another
regularly scheduled late-night weekend maintenance --- which will, of course,
be announced on this list.

Some ISPnet customers were affected by last night's problems more than others.
I'd like to extend my apologies to all customers and assure you that we are
taking steps to avoid any reoccurance.

--
Bob Tinkelman          <bob at tink.com>
ISPnet, Inc.    http://www.ispnet.net
+1 (718) 464-4747  office
+1 (800) 806-NETS  toll free



More information about the ispnet-announce mailing list