Month: January 2008

Thu Jan 31 14:29:24 PST 2008 — Upstream Network Problems. At 2:00pm this afternoon, one of our upstream network providers at our Los Angeles POP experienced a widespread outage. Customers in the LA area would have noticed spotty Internet connectivity. We have since rerouted traffic via a redundant link and the problem appears to have been resolved. We will continue to monitor the affected link and will return it to service when we are assured the problem has been resolved. -Dusty, Nathan and the NOC

Tue Jan 29 15:54:31 PST 2008 — Sonic.net now offers WAP enabled Webmail for your favorite mobile devices (Phone, PDA, etc.); this provides a quick and easy way to access your email when you are on the go. To try it visit m.sonic.net from any WAP2/XHTML compatible mobile device. -Augie

Mon Jan 28 21:16:52 PST 2008 — Hardware changes are going to be performed in a maintenance window from 12:01 AM – 2:00 AM (Tuesday morning, the 29th) on ssl.sonic.net. I will also be performing some changes to the replication system on custsql.sonic.net, which may cause some periods of inactivity. I will also take this opportunity to fix a simple yet important problem on the system which will result in a reboot. Customer down time will be only as required and as minimal as possible. –Don Update: Downtime for custsql was approximately 40 minutes. All other upgrades/changes went as expected. –Don

Sat Jan 26 18:24:57 PST 2008 — San Diego Packet Loss. This afternoon San Diego DSL customers began noticing 10-15% packet loss on their DSL lines. Upon inspection we found that one of the uplinks on our SMS that serves that area was taking errors. We re-routed traffic to the other uplink, thus resolving the packet loss problem. We will be performing maintenance on that SMS soon to resolve the packet loss problem permanently. -Jared and Nathan

Sat Jan 19 12:17:17 PST 2008 — Massive DoS Attack. One of our customers was the target of a massive denial of service attack. This attack caused minor internal routing issues as we tracked it down and blocked it at our edge. Customers may have seen spotty Internet connectivity in the interim. At this time the DoS attack has been blocked and all is well. -Jared and Nathan

Fri Jan 18 01:09:33 PST 2008 — Santa Rosa Transport Failure. Earlier today one of our two redundant GigaMAN transport links out of our Santa Rosa datacenter flapped. We took the link out of routing and began troubleshooting the problem with AT&T. Customers saw no disruption as all our Santa Rosa traffic took advantage of our redundant GigaMAN link to go around our Bay Area ring. Then, this evening, our colocation provider in San Jose began emergency maintenance on their cabling plant. This emergency maintenance took down our redundant Santa Rosa GigaMAN link, effectively severing our Santa Rosa datacenter from the rest of our network and the Internet. We are currently working with both AT&T and our colocation provider to restore service as quickly as possible. This outage affects all of our colocation, web hosting, and WBA customers, as well as T1 customers in the Santa Rosa area. All access to Sonic.net email services are affected as well. All of our dial-up, DSL, Business-T and T1 customers in the San Francisco area have complete access to the Internet, thanks to our redundant DNS and authentication server clusters in our San Francisco POP. We will continue to provide updates as the situation develops. -Jared and Nathan

Update Fri Jan 18 01:46:19 PST 2008 — Our San Jose colocation provider has completed their emergency maintenance, thus restoring one of our GigaMAN connections. All services and customers are back online at this time and we do not foresee any further problems. Sonic.net’s network is built to be extremely redundant and protected from most failures, but even the most protected network can be laid low by the right combinations of the right failures at the wrong time. -Jared and Nathan

Fri Jan 18 00:31:09 PST 2008 — The previously mentioned customer SQL upgrades that were to take place tonight are being rescheduled. -Don

Thu Jan 17 17:08:30 PST 2008 — Customer SQL server maintenance. Tonight at 12:01AM I will be performing upgrades on custsql.sonic.net which will result in a required reboot of the system. Customers using custsql may notice a period of about 10 minutes where their databases may be unresponsive. -Don and the SOC

Tue Jan 15 16:15:16 PST 2008 — Customer router upgrades. This Thursday at 12:01 AM we will be performing minor maintenance on two of our customer routers in Santa Rosa. One router will be undergoing a minor software upgrade and reboot, while the other is having new hardware installed. There will be no customer downtime aside from the 5 minute reboot of the first router. -Jared, Matt and Tim

Wed Jan 9 11:20:42 PST 2008 — SpamAssassin has been upgraded to the latest stable release along with some of the underlying utilities that detect and filter spam. We expect users to see better mail filtering and performance from these upgrades. -William and the SOC