[ispnet-announce] ISPnet peering at the Telehouse NYIIX

[Bob Tinkelman]

The problem described in our earlier post has been resolved.

Summary:

On Sunday, we became aware of a problem that involved
intermittent periods of high packet loss on connections
to ISPnet through the Telehouse NYIIX peering switch.

On Monday, Telehouse announced the existence of a problem
and, Monday afternoon, ISPnet shut down all our NYIIX
peering sessions, rerouting the traffic over other links.

Late Tuesday, Telehouse announced they had installed a
fix that they had received from their switch vendor.

Wednesday morning, we restored peering with one NYIIX
peer and then, around noon, with all our NYIIX peers,
and our connection to the NYIIX has been running, with
no problem since.

If you notice any reoccurances of the original issues,
please contact us immediately.

--
Bob Tinkelman          <bob at tink.com>
ISPnet, Inc.    http://www.ispnet.net
+1 (718) 464-4747  office
+1 (800) 806-NETS  toll free



> This is a informational announcement to ISPnet custoemrs, not
> requiring any action on your part.

> On Sunday and Monday we received problem reports from several
> customers who were observing high packet loss on connections to
> ISPnet sites from certain other networks, with the most common
> complaints involving Cablevision/Optimum/Lightpath. The problems
> occured for brief periods that seemed to occur randomly, roughly
> on the order of once every 3 to 5 hours.

> We localized the problem to the point where ISPnet connects with
> Cablevision at Telehouse's NYIIX peering switch in New York.

> Telehouse reported that they were working on a similarly desribed
> problem affecting all traffic through their switch fabric.

> Yesterday afternoon, ISPnet shut down all our peering bgp
> sessions that utilize the NYIIX and plan to bring these back up
> only after Telehouse has found and fixed their problem.

> Under normal circumstances, between 30 and 50% of ISPnet's
> external traffic uti.izes direct-peering with other networks,
> mostly over the NYIIX and the DE-CIX-NY peering exchanges, the
> latter of which is _not_ involved in this problem.

> For now, the portion of our traffic that would normally have
> utilized the NYIIX will take another path, either via peering at
> the DE-CIX-NY or via one of ISPnet's upstream links.

> We would not expect this change to cause any problems or, in
> fact, to be visible in any way to a normal user.  [Someone
> using "traceroute" to investigate network flow will see a
> different path than normal.]

> The portion of traffic affected will vary on a customer by
> customer basis.

> This email is mainly precautionary.  Something in ISPnet routing
> has changed.  You should be aware of that and should (as usual)
> report any routing-related issues to us.

> --
> Bob Tinkelman          <bob at tink.com>
> ISPnet, Inc.    http://www.ispnet.net
> +1 (718) 464-4747  office
> +1 (800) 806-NETS  toll free