Thu Mar 10 18:11:34 PST 2005 — Update on Network Event. Upon further investigation we were able to determine that the problem was not caused by a failure inside our network but that our L2 transport provider between Santa Rosa and San Jose was flooding both ends of our circuit with more than 100kpps of broadcast traffic saturating the Juniper RE CPUs at both ends. The CPU congestion on these two routers lead to some of our internal and external BGP and IGP sessions flapping leading to general network instability. Once the source of the problem was isolated and the link shut down the network healed itself. -Kelsey, Nathan, John, Zeke and Jared.

Thu Mar 10 16:49:14 PST 2005 — Backbone Network Degradation. A little after 4 PM today our router in San Jose failed. This affected our outside connectivity until we could reroute traffic onto backup circuits in San Francisco a short time ago. We are working now to diagnose the router. -Nathan, Kelsey and John

Tue Mar 8 08:55:37 PST 2005 — Loss from Santa Rosa to San Jose. This morning at around 7:55am our link from Santa Rosa to San Jose started to show packet drops. We shut the circuit down at 8:15am, leveraging our redundant gigabit ethernet ring between Santa Rosa, San Francisco and San Jose to provide healthy service despite the failure. Users may have experienced degraded service during this period. -Nathan

Fri Mar 4 22:17:43 PST 2005 — GlobalPops to begin restricting mail server access. Our national dialup service provider is going to begin blocking our customers’ access to any other email forwarding servers besides mail.sonic.net. For most customers, this means no change. For any customer who sends email through another server, for instance, the one at their workplace, they will have to switch to mail.sonic.net instead. Please contact Sonic.net Tech Support if you have any questions. -John and Russ

Tue Mar 1 14:58:21 PST 2005 — Enhanced Network Capacity and Resiliency. On Friday, 2/25 we successfully installed and subsequently tested a new GigabitEthernet link between our 200 Paul Ave. POP in San Francisco and our POP at Equinix in San Jose. This new link has been added in an effort to increase capacity and network resiliency, ensuring continued high-speed delivery of traffic in the event of a single link failure within our network. -Network Operations

Mon Feb 21 17:11:22 PST 2005 — Multicast and IPv6 outage. The router which provides both multicast and IPv6 service is currently down. We’re working on restoring service, but don’t have an ETR at this time. -Nathan and John

Update Mon Feb 21 22:37:58 PST 2005 — Multicast and IPv6 service is restored. -Nathan

Fri Feb 18 17:22:42 PST 2005 — DSL outage in eastern Santa Rosa. ASI/SBC has acknowledged that there is a widespread problem with DSL services in the Eastern Santa Rosa coverage area. They have deployed repair technicians, and service should be restored shortly. -Kavan

Fri Feb 18 01:36:27 PST 2005 — Latency and loss to San Jose, resolved. One of our links from Santa Rosa to San Jose has started showing latency and packet loss. Fortunately, our redundant ring architecture has allowed us to move traffic off the impacted link until the vendor can repair it. -Nathan

Mon Feb 14 07:01:24 PST 2005 — Another SAN related mail server event. Gremlins continue to plague the SAN network. This morning an interface on a different NetApp locked-up and stopped passing traffic. This may indicate that the failures are actually being caused by a common shared gige switch as two back to back interface lock ups would be very unusual otherwise. The event occurred between approximately 5:30 and 6:50AM, during this period most people would have experienced timeouts or other errors while trying to retrieve their mail from the POP3 or webmail servers. -Kelsey

Sun Feb 13 08:21:50 PST 2005 — POP3 Server Outage. Due to a failure in one of the four clustered NetApp filers network interfaces our POP3 servers went unavailable early this morning. We’ll investigate why the failure occurred; why the redundant network interface wasn’t able to take over the failed one. All services have been restored. -Kelsey