[ispnet-announce] ISPnet service outage, morning of 10/14/16

[Bob Tinkelman]

ISPnet suffered a problem on one of its backbone switches
this morning (a bit after 10:00 EDT).  The problem impacted
multiple customers with symptoms ranging from high rates of
packet loss to total loss of connectivity.  For most affected
customers, the problems started between 10:10 and 10:20 and
cleared between 10:40 and 10:50.

Post-mortum analysis has not produced a complete picture of
this morning's events, but we have learned enough to be able 
to adjust our standard procedures to avoid re-occurances.

For those curious about the technical details, I can say it
involved the failure of the Ethernet spanning tree algorithm
to prevent the formation of a loop on a pair of parallel GigE
customer circuits, possibly due to one of the carriers involved
incorrectly blocking transmission of BPDU Ethernet packets.

We apologize to all the customers whose service was affected,
and would like to assure everyone that we are doing everything
we can to avoid any repetition of this problem.

--
Bob Tinkelman          <bob at tink.com>
ISPnet, Inc.    http://www.ispnet.net
+1 (718) 464-4747  office
+1 (800) 806-NETS  toll free