Problems routing to ISPnet customers via NTT

[Bob Tinkelman]

This is a status update on what turns out to have been two
different problems that have affected ISPnet customers over the
past two days.

Diagnosis was complicated as not all customers were affected by
both problems and each problem affected communications with only
certain parts of the Internet.



PROBLEM 1 - Monday evening

This problem started some time Monday afternoon or evening.

It affected primarily customers with equipment colocated at
Telehouse's facility at 7 Teleport Drive, Staten Island.

The symptoms were high levels of packet loss to sites whose
normal path to ISPnet involves Cogent Communications.

Cogent is one of ISPnet's upstreams connecting to us at our POP
at 165 Halsey St, Newark, NJ.

Once we'd localized the problem to a particular path between our
Staten Island POP and the Cogent Newark router, we disabled that
path, letting traffic flow over various alternate paths.

This problem is very similar to one he had previously with the
Cogent POP in Newark which we (and Cogent) thought we'd solved.

While this problem is no longer affecting customers, we will need
to troubleshoot further (possibly jointly with Cogent).  We don't
expect that work to impact customer traffic, but will perform it
during an announced maintenance window, just in case.



PROBLEM 2 - Tuesday morning

This problem started some time after 3AM EST Tuesday morning.
(Today.)

It affected all ISPnet customers.

The symptoms were intermittent losses of connectivity to sites
normally reachable from ISPnet through NTT.

NTT is one of ISPnet's upstreams connecting to us via a link
between our POP at 85 Tenth Ave and their POP at 111 Eighth Ave,
both in Manhattan.

We disabled our NTT link (in order to insulate our customers from
the problem) around 4AM and then worked with NTT to diagnose
the problem and repair it.

The trigger for the problem was a config change to the NTT router
in NYC to which ISPnet connects.

The symptoms, inside NTT, were stable routing to ISPnet in the
general NY area but unstable (flapping) routing to ISPnet from
other parts of NTT.  This first showed up in their Ashburn VA
POP, but occured elsewhere (worldwide) within NTT.

After NTT and ISPnet engineers diagnosed the problem, we
implemented a fix, tested it, and then turned up our NTT link in
NYC, some time before 7:30 AM.

Things have been running smoothly for the past two hours and we
believe this problem has been fixed, permanently.




APOLOGIES

We apologize to all our customers for the effects of the two
recent (unrelated) problems which occured in such close
proximity.

Fortunately, we were able to limit the effects of each by
rerouting traffic through alternate paths. 





As always, we urge customers to contact us immediately if they
encounter any problems related to our service, using email and/or
phone, as appropriate to each problem.


--
Bob Tinkelman   <bob at tink.com>
Support         <support at ispnet.net>
ISPnet, Inc.    http://www.ispnet.net
+1 (718) 464-4747  office
+1 (800) 806-NETS  toll free