Routing issuess in New York

[Bob Tinkelman]

In this follow-up to our earlier post, quoted below for
reference, we can provide a more detailed description of the
problems of earlier this morning.

ISPnet has multiple upstream links in NY, one of which, NTT, had
a scheduled maintenance window last night, starting midnight,
local time.

Like most ISPs, NTT announces maintenance windows in which they
do routing maintenance work, on the average of 6 to 12 times per
year (at least in the case of NTT).

At some time during an announced window, NTT will shut down the
bgp sessions between their routers and the ISPnet routers,
causing the traffic that would normally use the NTT-ISPnet
connection to flow over one of the alternate paths.  Then, later,
NTT will re-enable the sessions, and traffic flows revert to
their normal paths.

That's what happened this time, but with an important exception.

As part of NTT's maintenance, they "moved" the ISPnet connection
from one of their routers at 111 8th Ave to a different router,
but due to an oversight, failed to remove one portion of the old
configuration from the old router.

The result was that traffic destined for ISPnet and arriving at
NTT's NY POP would either get forwarded to ISPnet correctly or it
would get dropped depending on how it entered the NTT POP.

Most of the reports we received involved traffic from Verizon to
ISPnet through NTT, but at least some reports involved networks
other than Verizon.


Once we and NTT had identified the problem, we updated the ISPnet
routers so _incoming_ ISPnet traffic would not use the ISPnet-NTT
NY link but would use other paths into ISPnet instead.

This was around 7 AM, and, as far as we know, all problems
cleared around that time.  We emailed or spoke with all the
customers who reported problems and all who have responded have
confirmed no problems since at least 7AM.


While we are not bgp-announcing most customer nets to NTT, at the
moment, we are announcing a number of test networks and have been
using them to confirm that NTT's fix is working.


It's been aboutn 4 hours since our "work-around" and our plans
are to re-enable use of the NTT-ISPnet link some time over the
next hour.


We weighed delaying doing this until a scheduled night-time
maintenance window, but decided against that.

This was based on an assessment of the tradeoffs:

   o  While we are confident that NTT has fixed their
      problem, there is always a risk of an error when
      we make a routing change.

   o  NTT is one of our major upstream links, and we
      are operating with a lower level of redundancy
      that we'd like while the link to them is disabled.

Clearly we will do our best to monitor for any problems as
we do this work.

If you notice anything, please contact us immediately, preferably
by phone.

You should receive one more email in this stream after we've
reenabled the NTT link and have confirmed the stability of the
network.

--
Bob Tinkelman          <bob at tink.com>
ISPnet, Inc.    http://www.ispnet.net
+1 (718) 464-4747  office
+1 (800) 806-NETS  toll free










Original posting:

> ISPnet is currently working on a problem that is affecting
> routing between ISPnet customers in New York and certain
> parts of the Internet.

> At this time we don't have a complete diagnosis or an ETR,
> but will continue working on the issue and post updates to
> this list as the situation changes.

> Briefly, this is what we have so far:

> There is an issue, in NY, on a link between Verizon and NTT
> (one of ISPnet's upstreams in NY) that is affecting traffic
> from Verizon customers to ISPnet ip addresses.

> We are working with NTT and they have reached out to Verizon.

> NTT did have an announced maintenance window this morning.
> During that window, our bgp sessions to NTT went down (as
> expected) and (again, as expected) connectivity to the
> world was unaffected, with traffic rerouted over other paths.

> The ISPnet-NTT bgp sessions came back up around 2 AM (Eastern)
> and problems appeared to have started occuring soon after.


> Again, we'll continue to give this problem our full attention
> and will report again as soon as we can.
> --
> Bob Tinkelman          <bob at tink.com>
> ISPnet, Inc.    http://www.ispnet.net
> +1 (718) 464-4747  office
> +1 (800) 806-NETS  toll free