Sat Jan 19 12:17:17 PST 2008 — Massive DoS Attack. One of our customers was the target of a massive denial of service attack. This attack caused minor internal routing issues as we tracked it down and blocked it at our edge. Customers may have seen spotty Internet connectivity in the interim. At this time the DoS attack has been blocked and all is well. -Jared and Nathan

Fri Jan 18 01:09:33 PST 2008 — Santa Rosa Transport Failure. Earlier today one of our two redundant GigaMAN transport links out of our Santa Rosa datacenter flapped. We took the link out of routing and began troubleshooting the problem with AT&T. Customers saw no disruption as all our Santa Rosa traffic took advantage of our redundant GigaMAN link to go around our Bay Area ring. Then, this evening, our colocation provider in San Jose began emergency maintenance on their cabling plant. This emergency maintenance took down our redundant Santa Rosa GigaMAN link, effectively severing our Santa Rosa datacenter from the rest of our network and the Internet. We are currently working with both AT&T and our colocation provider to restore service as quickly as possible. This outage affects all of our colocation, web hosting, and WBA customers, as well as T1 customers in the Santa Rosa area. All access to Sonic.net email services are affected as well. All of our dial-up, DSL, Business-T and T1 customers in the San Francisco area have complete access to the Internet, thanks to our redundant DNS and authentication server clusters in our San Francisco POP. We will continue to provide updates as the situation develops. -Jared and Nathan

Update Fri Jan 18 01:46:19 PST 2008 — Our San Jose colocation provider has completed their emergency maintenance, thus restoring one of our GigaMAN connections. All services and customers are back online at this time and we do not foresee any further problems. Sonic.net’s network is built to be extremely redundant and protected from most failures, but even the most protected network can be laid low by the right combinations of the right failures at the wrong time. -Jared and Nathan

Fri Jan 18 00:31:09 PST 2008 — The previously mentioned customer SQL upgrades that were to take place tonight are being rescheduled. -Don

Thu Jan 17 17:08:30 PST 2008 — Customer SQL server maintenance. Tonight at 12:01AM I will be performing upgrades on custsql.sonic.net which will result in a required reboot of the system. Customers using custsql may notice a period of about 10 minutes where their databases may be unresponsive. -Don and the SOC

Tue Jan 15 16:15:16 PST 2008 — Customer router upgrades. This Thursday at 12:01 AM we will be performing minor maintenance on two of our customer routers in Santa Rosa. One router will be undergoing a minor software upgrade and reboot, while the other is having new hardware installed. There will be no customer downtime aside from the 5 minute reboot of the first router. -Jared, Matt and Tim

Wed Jan 9 11:20:42 PST 2008 — SpamAssassin has been upgraded to the latest stable release along with some of the underlying utilities that detect and filter spam. We expect users to see better mail filtering and performance from these upgrades. -William and the SOC

Fri Jan 4 19:53:00 PST 2008 — Widespread network outage. At approximately 5pm today we logged a massive amount of inbound traffic headed toward one of the colocation customers in our Santa Rosa datacenter. This distributed denial of service attack (DDoS) consisted of well over a gigabit of traffic aimed at this customer, sourced by thousands of zombie computers likely part of a massive botnet. This attack caused two of our gigabit transit links to flap wildly, which caused routing instability inside and outside of our network. This flapping was curtailed by a controlled shutdown and bring-up of these transit links. During this attack, most traffic continued to flow normally, but connectivity to some sites was significantly degraded or unavailable.

Further complicating matters was the rather confusing loss of a Santa Rosa datacenter router. In the middle of the DDoS, one of the two core routers that services our Santa Rosa datacenter suffered a hard drive failure. In addition to contributing a bit of red herring to the mess, this router seems to have spewed some incorrect routing information during the confusion, further complicating our restoration. At this time the router is still down pending hardware replacement. We’ve got on-site spares for this unit, and will be swapping them in around midnight tonight during a maintenance window. There are no customers directly connected to this router, and it’s set up with a redundant neighbor that can take over its duties as necessary. No customers are affected by this router being off-line.

As if that wasn’t enough, one of our network engineers made an unfortunate typo in the heat of battle, the end result of which was a nearly network-wide loss of routing protocol packets. This occurred at around 6:20pm, after internet-wide connectivity was almost fully restored. Emergency roll-back procedures were set into motion, and rapid service restoration required usage of our out-of-band management system to remotely console the affected devices and deactivate the change. Even with these procedures, fully restoring network connectivity took around 25 minutes.

We’ll be discussing this outage at length internally to put policies and procedures in place to prevent any possibility of recurrence, as well as investigating why the routing instability caused such an impact to our network core. Our apologies for the downtime!

-Nathan, Jared, Matt, and the Sonic.net NOC

Fri Jan 4 17:48:24 PST 2008 — We have just experienced a very large distributed denial of service attack which impacted network performance. Our NOC is has isolated the attack and is wrapping defensive measures. Performance is improving, and should be completely normal shortly. -Dane and team

Thu Jan 3 14:53:56 PST 2008 — Sonic.net launches Satellite Broadband service. Sonic.net now offers high-speed Internet access via satellite to areas where broadband service was previously unavailable. Like DSL, this is a broadband (512kbps to 1.5Mbps) speed always-on connection without the need for dialing up or using a phone line while you are online.

Pricing and details can be found at www.sonic.net/sales/sb/. If you know folks who would be interested in broadband access in an area not served by DSL, please spread the word. Thanks! -Dane

Wed Dec 26 16:12:00 PST 2007 — Upstream Network Problems. At approximately 3:45 this afternoon we observed network problems in one of our upstream network provider’s networks. We are working with our provider to isolate and resolve this problem. -Jared and Nathan