Month: March 2009

Emergency Router Maintenance

At 12:00PM this afternoon we will be performing an emergency router reload on one of our ATM customer aggregation routers. All connected Business-T and FRATM customers will experience approximately 5 minutes of downtime during the reload.

-Tim and Dusty

Fresno ATM Backhaul Issue

At 1:08 AM today, our ATM backhaul circuit to the Fresno area went down unexpectedly. We immediately began diagnosing and troubleshooting the circuit with AT&T, but before we could isolate the problem, the circuit came back up, approximately 10 minutes later.

We are currently monitoring the circuit with AT&T, and apologize for any inconvenience this brief outage caused.

-Jared

Los Angeles ATM Maintenance

This Thursday, March 26 at 12:01 AM, we will be inserting a new card into our core ATM switch in Los Angeles. This switch handles all DSL, Business-T and FRATM connections in the Los Angeles area. No impact is expected, though the hardware insertion may require a reboot of the chassis. If this is necessary, all DSL, Business-T, and FRATM customers in the Los Angeles area will suffer 5-10 minutes of downtime.

-Jared

Update: The card insertion has been completed without incident. There was no impact to any customers.

PHP url_include removal

Monday March 30th we will be disabling the url_include ability in our default PHP setup in order to improve the security of our web cluster. This ‘feature ‘ of PHP is frequently misused by web developers and is the by far the most common vector used by hackers to gain access to exploit customer websites. Web hosting customers that require this functionality have several options to either work around or re-enable it presented in further detail in a FAQ at http://www.sonic.net/support/faq/advanced/url_include.shtml

If you think you may be using this feature we urge you to review your php code before March 30th and make any necessary changes to ensure that you will not be affected.

Update: We have completed the changes to php on our web cluster. Please note that at this time these changes only affect customers using the default php configuration.

Unexpected DSL Aggregation Router Reload

At approximately 4:30 PM today, one of the DSL aggregation routers that serve DSL customers in the Bay Area unexpectedly reloaded. This caused about 5 minutes of downtime for the customers served by this router.We are investigating the cause of this reload.

-Jared

Customer Router Hardware Maintenance

This Wednesday, March 18 at 11:00 PM, we will be inserting new interface cards into two of our T1/T3 customer aggregation routers in San Francisco. This work is not expected to cause an interruption of service, but a router reload may be required to complete this operation. If a reload is required, customers served by the affected router will experience 5-10 minutes of service interruption.

-Jared and Matt

Update: This maintenance has been completed without incident. There was no customer impact.

Emergency Router Maintenance

At 12:00PM this afternoon we will be performing an emergency router reload on one of our ATM customer aggregation routers in Los Angeles. All connected Business-T and FRATM customers will experience approximately 5 minutes of downtime during the reload.
-Jared, Tim and Dusty

Fresno ATM Issues

We are currently tracking an issue with our OC-3 aggregation circuit that serves the Fresno area.  The circuit went down for 5 minutes about 10 minutes ago.  It is presently up, but not passing any traffic.  We’ve been in touch with AT&T with regards to this issue and will update when we have more information.  At this time we suspect a problem with AT&T’s ATM switch upstream of us.  -Dusty and Nathan

Update: AT&T has isolated the issue to one of the core ATM switches in Fresno, which has ceased proper functioning. AT&T continues to work on repair of the affected ATM switch. Due to the core nature of this ATM switch, it has been confirmed that it is affecting most DSL in the Fresno area, not just Sonic.net.

Update: As of approximately 6:15 PM today, AT&T has repaired the failed switch, and traffic appears to be flowing normally across our Fresno OC-3 at this time. AT&T does not expect any further impact to the DSL network in Fresno.

Router Software Upgrade

This Thursday, March 5 at 12:01 AM, we will be performing a software upgrade on one of our T1 customer aggregation routers in San Francisco. The software upgrade will require a reboot of the router, resulting in approximately 5 minutes of downtime for customers served by this router.
-Jared

Update: The software upgrade was completed without incident. Total downtime was 5 minutes for all customers served by this router.