Category: Network

Tonight, April 21 at 12:01 AM, we will be performing a maintenance reload of one of our DSL aggregation routers. This router serves DSL to customers in the Bay Area. Expected impact is less than 5 minutes of service interruption for DSL customers served by this router.

-Jared

Update: The SMS reboot did not go as smoothly as planned. Due to a configuration error, on reboot, the router loaded an archived configuration that was out of date. The proper configuration has been restored and all affected customers should now be back online. We apologize for this additional downtime.

At approximately 3PM today a DoS attack was aimed at one of our Business-T customers. The attack saturated the uplinks of one of our customer aggregation routers, causing 5-10 minutes of service interruption to customers served by that router. The DoS attack has been blocked at our network edge.

-Jared

This upcoming Friday, April 10 at 12:01 AM, we will be performing a maintenance reload on our ATM customer aggregation routers. This will result in 5-10 minutes of downtime for Business-T and FRATM customers.

-Jared

Update: The maintenance reloads were completed without issue. Total downtime for affected customers was under 10 minutes.

This Wednesday, April 8 at 12:01 AM, we will be doing brief cable maintenance on the DS3 that backhauls some of our T1 capacity in our San Francisco POP. This will result in approximately 15 minutes of downtime for T1 customers on the affected DS3.

-Jared and Nathan

Edit: Fixed incorrect month.

Update: This maintenance is complete. Actual downtime for affected customers was approximately 221 seconds.  -Nathan and Clay

This Thursday, March 26 at 12:01 AM, we will be inserting a new card into our core ATM switch in Los Angeles. This switch handles all DSL, Business-T and FRATM connections in the Los Angeles area. No impact is expected, though the hardware insertion may require a reboot of the chassis. If this is necessary, all DSL, Business-T, and FRATM customers in the Los Angeles area will suffer 5-10 minutes of downtime.

-Jared

Update: The card insertion has been completed without incident. There was no impact to any customers.

At approximately 4:30 PM today, one of the DSL aggregation routers that serve DSL customers in the Bay Area unexpectedly reloaded. This caused about 5 minutes of downtime for the customers served by this router.We are investigating the cause of this reload.

-Jared

This Wednesday, March 18 at 11:00 PM, we will be inserting new interface cards into two of our T1/T3 customer aggregation routers in San Francisco. This work is not expected to cause an interruption of service, but a router reload may be required to complete this operation. If a reload is required, customers served by the affected router will experience 5-10 minutes of service interruption.

-Jared and Matt

Update: This maintenance has been completed without incident. There was no customer impact.

We are currently tracking an issue with our OC-3 aggregation circuit that serves the Fresno area.  The circuit went down for 5 minutes about 10 minutes ago.  It is presently up, but not passing any traffic.  We’ve been in touch with AT&T with regards to this issue and will update when we have more information.  At this time we suspect a problem with AT&T’s ATM switch upstream of us.  -Dusty and Nathan

Update: AT&T has isolated the issue to one of the core ATM switches in Fresno, which has ceased proper functioning. AT&T continues to work on repair of the affected ATM switch. Due to the core nature of this ATM switch, it has been confirmed that it is affecting most DSL in the Fresno area, not just Sonic.net.

Update: As of approximately 6:15 PM today, AT&T has repaired the failed switch, and traffic appears to be flowing normally across our Fresno OC-3 at this time. AT&T does not expect any further impact to the DSL network in Fresno.

This coming Tuesday, Feb 17, beginning at 12:01 AM, we will be performing a retrofit of our network build supporting our Sonic Telecom CLEC installation at the main Santa Rosa central office. The maintenance window is expected to be up to an hour, but customer impact should be minor, no more than 2 10-minute outages during that time frame. This maintenance will only affect customers who recieve Sonic Telecom services in the downtown Santa Rosa area.

-Jared and Clay

Update: The retrofit has been completed. All operations went according to plan, and customers should have seen at most 10 minutes of service interruption. All customers served from this CO appear to be fully functional again.

One of the DHCP servers that serves our Bay Area DSL subscribers had a disk begin failing, which caused the server to not properly handle all DHCP requests sent to it. This caused spotty DHCP connectivity for some customers until we were able to diagnose the silently failing disk. We have removed the failing server from service, and DHCP service should be fully restored to the affected customers.

-Jared and Nathan