Wed Dec 12 06:06:55 PST 2001 — UUNet T3 down, backup link functioning. Our 45Mbps T3 to UUNet is currently down, and UUNet is working with PacBell now to isolate the trouble and repair it. Meanwhile, our 15Mbps link to Cable & Wireless is functioning well, thought it is pretty full. Internet performance this AM may be a bit sluggish until PacBell repairs the link to UUNet.
Update, Wed Dec 12 07:03:23, Pacific Bell has isolated the trouble as a loop-back in the Santa Rosa central office, and has dispatched a technician to that location to check the circuit. No ETA is available from Pacific Bell for the technician’s arrival. Currently, Internet connectivity is available, but packet loss is running 10%-20%, and performance is sluggish.
Update, Wed Dec 12 08:05:29, UUNet reports that they’ve escalated to tier three support at Pacific Bell, and that Pacific Bell’s projected ETA at the Santa Rosa central office for the technician was 7:30AM PST. As of 7:50AM, they had not heard from the repair technician with any new information. Currently, Internet connectivity is available. Packet loss is running around 13% now, and latency is about half a second longer than it should be. -Dane
Update: For a few of network statistics the UUNet outage, see:
Note that we’ve got 15Mbps to Cable and Wireless, which is working fine, but our primary 45Mbps link to UUNet is down. As you can see from the graph, we’ve got less capacity than we need at this point, so we’ve got functioning Internet connectivity, but it’s performance is likely to become more degraded as the morning goes on.
For a comparison of utilization of the links, see the following graph. Note the green and dark blue is UUNet, which drops to zero at around 5:30AM, and the red and light blue is Cable and Wireless, which immediately jumps to near capacity.
Sonic.net’s stats pages provide extensive network health information, and are available at www.sonic.net/stats/. -Dane
Update, Dec 12 09:17:00: We are on the phone with a Pac Bell tester and UUNet support. They suspect the problem is between the Santa Rosa CO and Sonic.net. Further updates will follow as information becomes available. -Matt, Dane and Steve
Update, Dec 12 10:05:16: There is now a Pac Bell tech working on the circuit in the Santa Rosa CO. We are on the phone with him and UUNet narrowing this down. -Matt
Update, Wed Dec 12 11:05:01: There are apparently two problems with the circuit. Pac Bell is working on the DSX3 in the Santa Rosa CO and dispatching a technician to Sonic.net. -Matt, Dane and Kelsey
Update, Wed Dec 12 12:11:30: Pac Bell isolated to problem to a bad DSX card and has replaced the card. This seems to have resolved the problem and traffic has shifted back to the UUNet circuit. Packet loss is gone and latency has returned to normal. We will be keeping a close eye on the T3 but expect that the problem is resolved. -Matt, Dane, Scott, Kelsey, Steve, Russ and Eli