At 9:40AM this morning we will be performing an emergency router reload on one of our ATM customer aggregation routers. All connected Business-T and FRATM customers will experience approximately 5 minutes of downtime during the reload.
-Tim and the NOC
One of the remote terminals serving a large number of customers in Santa Rosa is currently off line resulting in a sync no surf situation. AT&T reports that they have a technician dispatched to repair but we do not have an estimated time of repair.
This evening, beginning at 12:01AM, we will be performing maintenance on equipment serving Fusion and FlexLink customers in Windsor. Affected customers may experience up to 30 minutes of downtime while the work is performed.
-Nathan, Jared, and Tim
This Friday, starting at 9AM, one of the three UPSes that serves our Santa Rosa datacenter will be undergoing its annual preventative maintenance. The maintenance is fully scripted with our vendor and no interruption of services are expected. During the maintenance window our standby genset will be kept running should our PG&E service fail. -Kelsey and Russ
Update: Fri Feb 18 19:49:15 PST 2011, the schedule maintenance has been completed with some complication. One of of the bypass contactors jammed preventing the UPS from going back online. While the contactor eventually actuated and the UPS was brought online, it’s replacement will be scheduled as soon as possible. -Kelsey and Russ.
Tonight shortly before 7:30pm we lost connectivity on one of the backhaul circuits serving ADSL1 customers in the Bay Area and North Coast. This has resulted in many customers experiencing a sync-no-surf outage, where the DSL modem indicates it has a connection to the local DSLAM, but no traffic passes through. Out network engineers have been working on the problem since immediately after it happened, but we have no ETR yet. Fusion, wireless, and dialup service, as well as ADSL1 service outside the Bay Area and North Coast are unaffected. — John F and the NOC staff
Update: Our network engineers have been working counterparts at AT&T and have narrowed the problem down to what appears to be a faulty switch on their end. Still no ETR. — John F and the NOC staff
Update: As of 11:03 PM tonight, the ATM backhaul circuit was restored to service by ATT. All affected customers should be back online at this time. If you are still experiencing a problem with your DSL connection, please reboot your DSL modem.
-Jared
One of our core internal MySQL servers has experienced a failure off the SSD based storage system that it was recently migrated to. We’re restoring from backups now, but it is too early to estimate when things will be back up and running. This does not directly impact any of or access or hosting services but does disable access to our member tools, signup forms and many other internal systems.
Update: We’ve successfully migrated back to the old spinning disk RAID and while performance isn’t as good, it is a stable config that has operated for several years without issue. At this time all affected services are back up and running. Back to the drawing board for our next generation SQL server platform. -Kelsey
One of our Redback DSL aggregation routers failed this morning at approximately 11:39 AM. We are in the process of migrating all customers off that Redback to a hot spare device. We expect that all affected DSL subscribers will be restored to service in 5 minutes.
-Jared
This Friday, Jan 14th at 12:01 AM we will be performing non-intrusive power maintenance on our equipment in the downtown Santa Rosa CO. We do not anticipate any customer impact during this work.
-Nathan and Tim
Update: This maintenance has been pushed to Friday, Jan 14th at 12:01 AM
At 8:03PM this evening one of the DSL aggregation routers that serves legacy DSL customers in LATA 1 rebooted unexpectedly. Service for affected customers was restored by 8:05PM. We are investigating the cause of this reboot and will monitor closely for any further issues.
-Tim and Jared
As of 3:30pm PST, DSL customers served out of many San Francisco wire centers are currently without internet due to an AT&T equipment failure. Our estimated time of repair is within several hours. We are working with AT&T to fix the problem, and we hope to update our MOTD with a more exact repair time shortly.
Fusion customers in San Francisco are not affected by this outage.
Update: At 4:20pm, most of our customers appear to be surfing again, and AT&T expects the remaining customers to be fixed shortly.