Thu Dec 6 15:56:30 PST 2007 — Network instability. At approximately 3:36pm one of our core routers in San Francisco restarted all of its BGP sessions without cause. The sessions took a few minutes to restore. Customers may have noticed intermittent reachability to some destinations during this period. We’ll be investigating the cause to try and prevent this from happening again! -Nathan and the NOC

Sun Nov 18 20:54:04 PST 2007 — Incoming mail server outage. Tonight our incoming mail server cluster stopped serving mail for POP3, IMAP, and webmail customers. Our operations staff is currently working on this problem. No other services should be affected. — John F

Update: Sun Nov 18 21:13:04 PST 2007 — One of our four clustered filers that handles customer email experienced a catastrophic disk failure. Catastrophic because it took both the A and B FC-AL loops causing all disks on it’s loop to go unavailable. As a result, all POP and IMAP services went off-line. A hard power reset brought all of the other disks back on-line and the offending disk has been removed. All services have now been restored and no data has been lost. -Kelsey and Nathan

Thu Nov 8 16:17:16 PST 2007 — Dial Upgrades. This Friday, November 9th at 10:00 AM we will be migrating a large portion of our dial-up numbers to new hardware. The migration will not cause any active calls to be dropped. The new hardware provides many improvements over our prior gear, including V.92 support, as well as increased reliability. -Dusty and Nathan

Sat Oct 27 03:17:56 PDT 2007 — Upgrades to Email storage complete. We have finished our scheduled maintenance on the Email storage system and Email is flowing again. Total downtime was around one hour. We also took the opportunity to run some software updates on many of our other systems. –Augie, Kelsey, Christopher, and Don.

Thu Oct 25 11:32:30 PDT 2007 — Upgrades to Email storage. This Friday at 12:00 AM we will be adding several new shelves to the NetApp Filers that store customer Email. This upgrade will provide greater capacity for future growth and improved performance for all of Sonic.net’s Email customers. Downtime is expected to be one hour while we install the shelves and flush mail queues. -Augie, Kelsey, Don, and Christopher.

Wed Oct 24 12:37:07 PDT 2007 — Core Router Maintenance. This Friday at 12:01 AM we will be performing a software upgrade on one of our core routers in our San Francisco POP. Some customers may notice a brief interruption in Internet connectivity as traffic is re-routed around this router. -Jared

Update Fri Oct 26 01:08:13 PDT 2007 — Router Maintenance Complete. We have completed the software upgrade on our core router in San Francisco. The entire operation went as planned and there was no customer traffic interruption. -Jared

Tue Oct 23 14:58:05 PDT 2007 — Upstream Network Issue. At 2PM today one of our upstream network providers suffered a problem in their network. We have routed traffic away from this provider as we work with them to resolve their issue. Customers may have seen 10-15 minutes of spotty or slow connectivity to sites on the Internet. -Jared and Nathan

Tue Oct 23 01:59:18 PDT 2007 — LA POP Migration complete. We have completed the migration of all DSL and Business-T customers in the Los Angeles LATA to our new LA POP. Downtime was 45 minutes at the worst, though most customers would have seen 20 minutes or less of downtime. Welcome to LA! -Jared, Nathan, Dusty, Matt and Tim

Fri Oct 19 16:23:05 PDT 2007 — LA POP Migration. The Network Operations team has spent the last couple of weeks deploying a new Point of Presence in Los Angeles. This huge new expansion, which includes redundant border routers, switches, and DSL termination hardware is now on-line and ready to accept customers. We will be swinging all of our LA DSL subscribers as well as our Business-T customers in the Los Angeles area to this new POP at 12:01am on Tuesday, October 23rd. We expect approximately one hour worth of downtime for affected customers as circuits are migrated. This deployment brings with it decreased latency to the Internet for our LA customers as well as a more redundant, scalable network deployment. -Nathan, Dusty, Jared and Tim

Wed Sep 26 11:52:46 PDT 2007 — Salinas DSL Maintenance. This Friday at 12:01 AM we will be performing maintenance on the backend systems that aggregate our Salinas DSL circuits. This maintenance will change the backend aggregation of this link to match all our other DSL circuits. The maintenance will result in approximately 15 minutes of downtime for all of our Salinas DSL customers. -Jared and Dusty Update Fri Sep 28 09:31:01 PDT 2007 — Due to unforseen complications, this maintenance has been postponed until 12:01 AM Tuesday, October 2. -Jared

Update Tue Oct 2 00:25:28 PDT 2007 — Salinas DSL Maintenance completed. Maintenance on our backend systems that aggregate Salinas DSL circuits was completed successfully. Total downtime for customers was 5 minutes. -Dusty and Jared