Update (2:00AM): This maintenance is now complete.
Update: This maintenance has been delayed until THURSDAY (6/18/2015) night.
Beginning tonight at midnight we will be performing maintenance on a core router in the Palo Alto area. No downtime is expected from this operation but users may notice some routing changes.
– Robbie
2:37AM PDT – This maintenance is now complete.
Tonight starting at 11:59PM PDT we will be performing software upgrades on routers in the Los Angeles area. No customer impact is expected as traffic will be migrated from these routers prior to the software upgrade.
-Tim J.
Email connectivity for IMAP, POP3, and Webmail is currently down. We believe this to be caused by a power event effecting an isolated set of servers in our datacenter. We will post more information when the systems come back online.
— Joe and the SOC
Update: Our IMAP/POP3 cluster is having a thundering herd problem and we’re working hard to encourage it to come up smoothly. -Kelsey and William
Update: All services have been restored and stabilized. It appears that the root of the problem lies in our central radius authentication servers – while it may be as simple as needing to upgrade the existing cluster of 6 servers with faster CPUs, we suspect we may have hit an internal performance limitation in the radius server software that prevented them from being able answer requests fast enough. We strive for complete redundancy and should not have had any lasting stability problems after a few servers lost power. We’ll investigate this failure over the next few days and resolve as needed to ensure this problem doesn’t occur again. -Kelsey and William
A small subset of Fusion customers are experiencing an outage in San Francisco. Â ETR is 10pm to replace the appropriate piece of equipment.
-Brandon
Service has been restored
-Brandon
This maintenance is now complete.
Tomorrow night, 6/11/15, starting at 11:59PM PDT we will be performing software upgrades of equipment in Orange County CA. No customer impact is expected from these upgrades.
-Tim J.
This maintenace is now complete.
Tomorrow night, 6/11/15, beginning at 11:59PM PDT we will be performing software upgrades of routers serving fiber DIA customers. This will impact customers in the Santa Rosa Airport Business park and other Fiber DIA customers. The expected outage is 15-20 minutes as the router reloads.
-Tim J.
This maintenance is now complete.
Tonight beginning at 11:59PM PDT we will be performing a software upgrade of routers in our Santa Rosa datacenter as well as in Palo Alto. Traffic will be drained from these devices before the upgrade begins, so no customer impact is expected.
-Tim J.
This maintenance is now complete.
Tonight starting at 11:59PM PDT we will be performing a software upgrade of OLTs and ONTs serving Brentwood CA customers. This upgrade is expected to last 30-45 minutes with small outage during.
-Tim J.
3:00AM PDT – This maintenance is now complete.
Tonight, 6/9/2015, starting at 11:59PM PDT we will be performing software upgrades on routers in Santa Rosa, and San Jose. Traffic will be migrated from these routers prior to upgrading and no customer impact is expected.
-Tim J.
Update(12:50PM): Services to Sebastopol were restored earlier today and services in Fort Bragg have just been restored.
Update(1:30AM): This maintenance is now complete. However, a small portion of customers are currently out of service in Fort Bragg and Sebastopol due to some hardware issues uncovered during the software upgrades tonight. We will be dispatching a technician asap to replace the failed hardware.
Beginning tonight at midnight I will be performing maintenance on equipment serving Fusion VDSL customers in the following cities:
Cloverdale, Cotati, Fort Brag, Guerneville, Healdsburg, Larkspur, Mill Valley, Mendocino, Napa, Occidental, Petaluma, Rohnert Park, Sebastopol, San Rafael, Sonoma, Sausalito, Saint Helena, Tiburon, Ukiah, Vallejo, Windsor, Yountville
Expected downtime is around 15 minutes.
– Robbie