Wed May 21 10:54:20 PDT 2008 — Technical support closing at 10pm Friday. Tech support will close an hour early on Friday May 23 for our employee movie night. Support will open at the regular time Saturday morning. -Eli and the Sonic.net staff

Tue May 20 09:38:02 PDT 2008 — Router Software Upgrade. At 12:01 AM Wednesday we will be performing a software upgrade on our Los Angeles Business-T/FRATM aggregation router. This software upgrade will extend our available feature set and should resolve the problem that has required us to perform emergency reloads of this router. Customers connected to this router are expected to be offline for 5 minutes as the router reloads. -Jared

Fri May 16 09:57:17 PDT 2008 — Emergency Router Maintenance. In 5 minutes we will be performing an emergency router reload on one of our ATM customer aggregation routers in Los Angeles. All connected Business-T and FRATM customers will experience approximately 5 minutes of downtime during the reload. -Dusty, Jared, Tim and Matt

Mon May 12 12:09:22 PDT 2008 — Office Telephone System Upgrade. On Tuesday morning, May 13 we will be replacing the internal phone system that serves our headquarters here in Santa Rosa. Our operating hours should be minimally affected- We plan to close at 10 pm Monday night, and open Support at 6 am on Tuesday morning. However, office voice mail will be unavailable overnight while we replace the equipment. The NOC hotline will not be affected.

We are anticipating a smooth roll-out, but if you have any difficulties when trying to contact our offices Tuesday morning please be patient – we’ll be working hard to complete the deployment as quickly as we can. We are excited about the new system, as it will allow us to provide new features and better customer service to you! – Eli, and the entire team at Sonic.net.

Tue May 6 09:48:59 PDT 2008 — San Jose POP Equipment Move. Tonight at midnight we will be moving the equipment at our San Jose POP to a new cage. The new cage will allow us to continue to expand and grow our network to match. During the maintenance traffic will be routed around our San Jose POP, so there should be no customer impact. However, our redundant ring around the bay will be compromised for approx 15-20 mins as we move gear that the ring passes through. We will be taking every effort to minimize this potential vulnerability. The entire move is expected to take 1-2 hours. -Jared and Tim

Mon May 5 17:24:12 PDT 2008 — Emergency Router Maintenance. In 5 minutes we will be performing an emergency router reload on one of our ATM customer aggregation routers. All connected Business-T and FRATM customers will experience approximately 5 minutes of downtime during the reload. -Jared, Dusty, Tim and Matt

Update Sat Apr 19 00:30:38 PDT 2008 — ATM Switch Maintenance Complete. We have completed the replacement of the backup management card in our ATM switch and all went as expected. There should have been no customer impact, and traffic levels look normal on the switch. -Jared and Matt

Fri Apr 18 11:29:19 PDT 2008 — ATM Switch Maintenance. Tonight at midnight, we will be replacing the management card in our ATM switch that started throwing errors on 4/2/08. This card is in a backup role on the switch and replacing it should not cause any customer impact. However, in the event of a failure, DSL service to customers on this ATM switch may experience up to 5 minutes of downtime. Potentially affected customers are DSL customers in the greater Bay Area, Chico, Stockton, and San Diego areas. -Jared and Matt

Thu Apr 17 12:21:00 PDT 2008 — News Server Outage. news.sonic.net is currently offline due to back to back disk failures in the redundant load balanced pair of servers that handle news readers services. At this time it is unclear how long it will take to restore news reader services.

Failure One: Last night nnrp1 (the news reader slave server) was taken off-line due to a disk failure. Users may have seen poor news performance if their streams were on this cluster until it was taken off-line. The failed disk was replaced this morning with a spare and work was started to rebuild it and bring it back into service. Failure Two: The news server systems have removable face plates that cover all of the drive bays. In order to confirm the correct configuration of the drives in nnrp1, the faceplate for nnrp0 was removed to expose the hot swap disk carriers. At this point, nnrp0 (the now single, and master news reader server) unexpectedly powered off. Upon reboot, one of it’s news related disks has also hard failed.

We will continue efforts to restore news services as quickly as possible. In any case, it will take at least the remainder of the day to restore services and is likely that article numbering will be lost. -Kelsey

Update:: Thu Apr 17 15:31:16 PDT 2008 — We have scavenged parts from nnrp1 to rebuild nnrp0 and are currently restoring a backup that we hope will allow us to retain article numbering across all but our local sonic.* groups. Unfortunately, it is going to take several more hours before the data restoration is complete. If this fails, we will bring the server back up with all groups starting at zero. -Kelsey

Update:: Thu Apr 17 21:21:35 PDT 2008 — news.sonic.net is back online. We were able to restore nnrp0 with article numbering more or less intact but did lose numbering of our local groups as expected. At this time our spoolers and extra header feeds and busy pushing their backlogged header feeds in. Time will tell how big of a gap remains once it has caught up. Some users may find that they have to unsubscribe/subscribe from our local sonic.* news groups in order to see new news or force their news readers to do a full refresh of the group’s headers. -Kelsey

Sat Apr 12 09:08:51 PDT 2008 — Los Angeles DHCP Failure. One of our DHCP servers in our Los Angeles PoP suffered a failure that prevented it from providing DHCP leases at approx. 2:30 AM today. This problem would have interrupted connectivity for dynamic IP DSL subscribers in the Los Angeles area until the problem was resolved 6 hours later at 8:40 AM. We apologize for the outage, and will be investigating the root cause of the problem our DHCP server experienced. -Jared and Tim