Mail Connectivity Issues

Email connectivity for IMAP, POP3, and Webmail is currently down. We believe this to be caused by a power event effecting an isolated set of servers in our datacenter. We will post more information when the systems come back online.

— Joe and the SOC

Update:  Our IMAP/POP3 cluster is having a thundering herd problem and we’re working hard to encourage it to come up smoothly.  -Kelsey and William

Update: All services have been restored and stabilized.  It appears that the root of the problem lies in our central radius authentication servers – while it may be as simple as needing to upgrade the existing cluster of 6 servers with faster CPUs, we suspect we may have hit an internal performance limitation in the radius server software that prevented them from being able answer requests fast enough.  We strive for complete redundancy and should not have had any lasting stability problems after a few servers lost power.  We’ll investigate this failure over the next few days and resolve as needed to ensure this problem doesn’t occur again.  -Kelsey and William

Intrusive Network Maintenance Fiber DIA – 6/11/15

This maintenace is now complete.

Tomorrow night, 6/11/15, beginning at 11:59PM PDT we will be performing software upgrades of routers serving fiber DIA customers. This will impact customers in the Santa Rosa Airport Business park and other Fiber DIA customers. The expected outage is 15-20 minutes as the router reloads.

-Tim J.

Intrusive Maintenance – Brentwood CA

This maintenance is now complete.

Tonight starting at 11:59PM PDT we will be performing a software upgrade of OLTs and ONTs serving Brentwood CA customers. This upgrade is expected to last 30-45 minutes with small outage during.

-Tim J.

Fusion VDSL Intrusive Maintenance – North of the Bay Area

Update(12:50PM): Services to Sebastopol were restored earlier today and services in Fort Bragg have just been restored.

Update(1:30AM): This maintenance is now complete. However, a small portion of customers are currently out of service in Fort Bragg and Sebastopol due to some hardware issues uncovered during the software upgrades tonight. We will be dispatching a technician asap to replace the failed hardware.

Beginning tonight at midnight I will be performing maintenance on equipment serving Fusion VDSL customers in the following cities:

Cloverdale, Cotati, Fort Brag, Guerneville, Healdsburg, Larkspur, Mill Valley, Mendocino, Napa, Occidental, Petaluma, Rohnert Park, Sebastopol, San Rafael, Sonoma, Sausalito, Saint Helena, Tiburon, Ukiah, Vallejo, Windsor, Yountville

Expected downtime is around 15 minutes.

– Robbie

Intrusive Fusion Maintenance

Beginning at midnight tonight I will be preforming intrusive maintenance for fusion customers in the Emeryville and Piedmont areas.  Expected downtime is ~20 minutes.

-Brandon

 

Maintenance is now complete

-Brandon