Wed Dec 26 16:12:00 PST 2007 — Upstream Network Problems. At approximately 3:45 this afternoon we observed network problems in one of our upstream network provider’s networks. We are working with our provider to isolate and resolve this problem. -Jared and Nathan

Mon Dec 24 13:17:44 PST 2007 — Sonic.net will be closing at 3pm today, Monday December 24th. We will re-open on Wednesday at 6am.

Enterprise dedicated (T1, T3, Business T, Fiber DIA) and colocation clients may call Sonic.net’s 24 hour network operation center in case of emergency. Happy holidays! -Dane

Mon Dec 17 16:05:59 PST 2007 — Router Maintenance. Tonight at 12:01 AM we will be performing minor hardware maintenance on one of our customer T1 aggregation routers in Santa Rosa. No customer downtime is expected. -Jared

Update Tue Dec 18 00:11:54 PST 2007 — Router Maintenance Complete. The hardware maintenance was completed with no problems. There was no customer downtime. -Jared

Wed Dec 12 15:05:48 PST 2007 — Router Reload. Our Business-T and FRATM aggregation router crashed this afternoon, during routine configuration. The router is back up and running normally, and we are investigating the cause of the crash. Northern California Business-T and FRATM customers experienced approx. 5 minutes of downtime. -Jared

Update Thu Dec 13 16:17:44 PST 2007 — Router Reload. Our Business-T and FRATM aggregation router crashed again today. We have isolated the cause of these crashes and have ensured that they will not occur again. -Jared and Nathan

Fri Dec 7 11:24:36 PST 2007 — Minor Internet Routing Problem. Earlier today, another ISP accidentally announced one of our networks into the global Internet routing table. This caused all Sonic.net customers in 209.204.154.0/24 to have no connectivity to the Internet at large. We contacted the ISP in question and had them remove the erroneous route. Internet connectivity for the IPs in question has been restored. -Jared and Jasper

Thu Dec 6 15:56:30 PST 2007 — Network instability. At approximately 3:36pm one of our core routers in San Francisco restarted all of its BGP sessions without cause. The sessions took a few minutes to restore. Customers may have noticed intermittent reachability to some destinations during this period. We’ll be investigating the cause to try and prevent this from happening again! -Nathan and the NOC

Sun Nov 18 20:54:04 PST 2007 — Incoming mail server outage. Tonight our incoming mail server cluster stopped serving mail for POP3, IMAP, and webmail customers. Our operations staff is currently working on this problem. No other services should be affected. — John F

Update: Sun Nov 18 21:13:04 PST 2007 — One of our four clustered filers that handles customer email experienced a catastrophic disk failure. Catastrophic because it took both the A and B FC-AL loops causing all disks on it’s loop to go unavailable. As a result, all POP and IMAP services went off-line. A hard power reset brought all of the other disks back on-line and the offending disk has been removed. All services have now been restored and no data has been lost. -Kelsey and Nathan

Thu Nov 8 16:17:16 PST 2007 — Dial Upgrades. This Friday, November 9th at 10:00 AM we will be migrating a large portion of our dial-up numbers to new hardware. The migration will not cause any active calls to be dropped. The new hardware provides many improvements over our prior gear, including V.92 support, as well as increased reliability. -Dusty and Nathan

Sat Oct 27 03:17:56 PDT 2007 — Upgrades to Email storage complete. We have finished our scheduled maintenance on the Email storage system and Email is flowing again. Total downtime was around one hour. We also took the opportunity to run some software updates on many of our other systems. –Augie, Kelsey, Christopher, and Don.

Thu Oct 25 11:32:30 PDT 2007 — Upgrades to Email storage. This Friday at 12:00 AM we will be adding several new shelves to the NetApp Filers that store customer Email. This upgrade will provide greater capacity for future growth and improved performance for all of Sonic.net’s Email customers. Downtime is expected to be one hour while we install the shelves and flush mail queues. -Augie, Kelsey, Don, and Christopher.