Tue Jan 15 16:15:16 PST 2008 — Customer router upgrades. This Thursday at 12:01 AM we will be performing minor maintenance on two of our customer routers in Santa Rosa. One router will be undergoing a minor software upgrade and reboot, while the other is having new hardware installed. There will be no customer downtime aside from the 5 minute reboot of the first router. -Jared, Matt and Tim

Wed Jan 9 11:20:42 PST 2008 — SpamAssassin has been upgraded to the latest stable release along with some of the underlying utilities that detect and filter spam. We expect users to see better mail filtering and performance from these upgrades. -William and the SOC

Fri Jan 4 19:53:00 PST 2008 — Widespread network outage. At approximately 5pm today we logged a massive amount of inbound traffic headed toward one of the colocation customers in our Santa Rosa datacenter. This distributed denial of service attack (DDoS) consisted of well over a gigabit of traffic aimed at this customer, sourced by thousands of zombie computers likely part of a massive botnet. This attack caused two of our gigabit transit links to flap wildly, which caused routing instability inside and outside of our network. This flapping was curtailed by a controlled shutdown and bring-up of these transit links. During this attack, most traffic continued to flow normally, but connectivity to some sites was significantly degraded or unavailable.

Further complicating matters was the rather confusing loss of a Santa Rosa datacenter router. In the middle of the DDoS, one of the two core routers that services our Santa Rosa datacenter suffered a hard drive failure. In addition to contributing a bit of red herring to the mess, this router seems to have spewed some incorrect routing information during the confusion, further complicating our restoration. At this time the router is still down pending hardware replacement. We’ve got on-site spares for this unit, and will be swapping them in around midnight tonight during a maintenance window. There are no customers directly connected to this router, and it’s set up with a redundant neighbor that can take over its duties as necessary. No customers are affected by this router being off-line.

As if that wasn’t enough, one of our network engineers made an unfortunate typo in the heat of battle, the end result of which was a nearly network-wide loss of routing protocol packets. This occurred at around 6:20pm, after internet-wide connectivity was almost fully restored. Emergency roll-back procedures were set into motion, and rapid service restoration required usage of our out-of-band management system to remotely console the affected devices and deactivate the change. Even with these procedures, fully restoring network connectivity took around 25 minutes.

We’ll be discussing this outage at length internally to put policies and procedures in place to prevent any possibility of recurrence, as well as investigating why the routing instability caused such an impact to our network core. Our apologies for the downtime!

-Nathan, Jared, Matt, and the Sonic.net NOC

Fri Jan 4 17:48:24 PST 2008 — We have just experienced a very large distributed denial of service attack which impacted network performance. Our NOC is has isolated the attack and is wrapping defensive measures. Performance is improving, and should be completely normal shortly. -Dane and team

Thu Jan 3 14:53:56 PST 2008 — Sonic.net launches Satellite Broadband service. Sonic.net now offers high-speed Internet access via satellite to areas where broadband service was previously unavailable. Like DSL, this is a broadband (512kbps to 1.5Mbps) speed always-on connection without the need for dialing up or using a phone line while you are online.

Pricing and details can be found at www.sonic.net/sales/sb/. If you know folks who would be interested in broadband access in an area not served by DSL, please spread the word. Thanks! -Dane

Wed Dec 26 16:12:00 PST 2007 — Upstream Network Problems. At approximately 3:45 this afternoon we observed network problems in one of our upstream network provider’s networks. We are working with our provider to isolate and resolve this problem. -Jared and Nathan

Mon Dec 24 13:17:44 PST 2007 — Sonic.net will be closing at 3pm today, Monday December 24th. We will re-open on Wednesday at 6am.

Enterprise dedicated (T1, T3, Business T, Fiber DIA) and colocation clients may call Sonic.net’s 24 hour network operation center in case of emergency. Happy holidays! -Dane

Mon Dec 17 16:05:59 PST 2007 — Router Maintenance. Tonight at 12:01 AM we will be performing minor hardware maintenance on one of our customer T1 aggregation routers in Santa Rosa. No customer downtime is expected. -Jared

Update Tue Dec 18 00:11:54 PST 2007 — Router Maintenance Complete. The hardware maintenance was completed with no problems. There was no customer downtime. -Jared

Wed Dec 12 15:05:48 PST 2007 — Router Reload. Our Business-T and FRATM aggregation router crashed this afternoon, during routine configuration. The router is back up and running normally, and we are investigating the cause of the crash. Northern California Business-T and FRATM customers experienced approx. 5 minutes of downtime. -Jared

Update Thu Dec 13 16:17:44 PST 2007 — Router Reload. Our Business-T and FRATM aggregation router crashed again today. We have isolated the cause of these crashes and have ensured that they will not occur again. -Jared and Nathan

Fri Dec 7 11:24:36 PST 2007 — Minor Internet Routing Problem. Earlier today, another ISP accidentally announced one of our networks into the global Internet routing table. This caused all Sonic.net customers in 209.204.154.0/24 to have no connectivity to the Internet at large. We contacted the ISP in question and had them remove the erroneous route. Internet connectivity for the IPs in question has been restored. -Jared and Jasper