At approximately 4:30 PM today, one of the DSL aggregation routers that serve DSL customers in the Bay Area unexpectedly reloaded. This caused about 5 minutes of downtime for the customers served by this router.We are investigating the cause of this reload.
-Jared
This Wednesday, March 18 at 11:00 PM, we will be inserting new interface cards into two of our T1/T3 customer aggregation routers in San Francisco. This work is not expected to cause an interruption of service, but a router reload may be required to complete this operation. If a reload is required, customers served by the affected router will experience 5-10 minutes of service interruption.
-Jared and Matt
Update: This maintenance has been completed without incident. There was no customer impact.
At 12:00PM this afternoon we will be performing an emergency router reload on one of our ATM customer aggregation routers in Los Angeles. All connected Business-T and FRATM customers will experience approximately 5 minutes of downtime during the reload.
-Jared, Tim and Dusty
We are currently tracking an issue with our OC-3 aggregation circuit that serves the Fresno area. The circuit went down for 5 minutes about 10 minutes ago. It is presently up, but not passing any traffic. We’ve been in touch with AT&T with regards to this issue and will update when we have more information. At this time we suspect a problem with AT&T’s ATM switch upstream of us. -Dusty and Nathan
Update: AT&T has isolated the issue to one of the core ATM switches in Fresno, which has ceased proper functioning. AT&T continues to work on repair of the affected ATM switch. Due to the core nature of this ATM switch, it has been confirmed that it is affecting most DSL in the Fresno area, not just Sonic.net.
Update: As of approximately 6:15 PM today, AT&T has repaired the failed switch, and traffic appears to be flowing normally across our Fresno OC-3 at this time. AT&T does not expect any further impact to the DSL network in Fresno.
This Thursday, March 5 at 12:01 AM, we will be performing a software upgrade on one of our T1 customer aggregation routers in San Francisco. The software upgrade will require a reboot of the router, resulting in approximately 5 minutes of downtime for customers served by this router.
-Jared
Update: The software upgrade was completed without incident. Total downtime was 5 minutes for all customers served by this router.
This Tuesday, March 3 at 12:01 AM, we will be performing software upgrades on two of our T1 and WBA customer aggregation routers in Santa Rosa. The software upgrade will require a reboot of the routers, resulting in approximately 5 minutes of downtime for customers served by these routers. These routers serve T1 customers in the Santa Rosa area, and a small subset of our WBA customers.
-Jared
The router software upgrades have been completed. Of the two routers being upgraded, one went smoothly, but the other had some issues properly loading the new software. This router was offline for approximately 20 minutes across 3 intervals while we worked to get the new software loaded. At this time, both routers are fully upgraded and appear to be operating normally. We apologize for the extra downtime incurred during this operation.
This morning around 6:30 AM one of the servers in our web cluster locked up and became unavailable. Whilst en route to investigate the problem two more servers in the cluster locked up and became unavailable. Customers may have noticed decreased performance with the cluster but should not have suffered any actual down time. Unfortunately due to human error, during the restoration of service we also had one of our MySQL servers go down because someone unplugged the wrong server. 🙁
It went something like this 🙂 . We apologize for any inconvenience and are trying to determine the cause to prevent future failures of this nature.
-William and the SOC
At 12:01 this morning, I will be performing the second and final phase of maintenance items on our databases that serve the Sonic.net Webmail interfaces During this time, there will be a slight interruption in the ability to connect to these interfaces. In no way does this effect the flow of mail, inbound/outbound, or IMAP/POP services. –Don
Update: Maintenance completed. All systems restored.
This coming Tuesday, Feb 17, beginning at 12:01 AM, we will be performing a retrofit of our network build supporting our Sonic Telecom CLEC installation at the main Santa Rosa central office. The maintenance window is expected to be up to an hour, but customer impact should be minor, no more than 2 10-minute outages during that time frame. This maintenance will only affect customers who recieve Sonic Telecom services in the downtown Santa Rosa area.
-Jared and Clay
Update: The retrofit has been completed. All operations went according to plan, and customers should have seen at most 10 minutes of service interruption. All customers served from this CO appear to be fully functional again.
At 3:45AM this afternoon we will be performing an emergency router reload on one of our ATM customer aggregation routers. All connected Business-T and FRATM customers will experience approximately 5 minutes of downtime during the reload. -Jared and Matt