Author: Grant Keller

UPDATE: Maintenance complete

Tonight at 10pm we will be performing software updates on several backend systems, including IMAP/POP3 servers. We don’t anticipate any downtime.

-SOC

System maintenance started at 9:10 am on the public shell server resulted in more downtime than expected. The problem that caused the downtime was found and resolved as of 11 am.

– SOC

UPDATE: Maintenance complete

Tonight beginning at 11pm we will be upgrading multiple systems, including our customer shell server cluster.  Any service interruptions should be very brief, and we expect to complete the maintenance within 2 hours.

– SOC

UPDATE:Maintenance complete.

Tonight starting at 11pm we will be performing system maintenance. The following services may experience brief interruptions:

• IMAP/POP3
• IPv6 Tunnels

Maintenance should be complete within the hour.

– SOC

UPDATE: 10:59pm Mail access restored.

During maintenance this evening starting at roughly 10:40 pm, our IMAP/POP3 cluster suffered a service impacting failure. We are working to restore mail access as soon as possible.

– SOC

At 3:42 PM this afternoon one of the UPSes that supplies power to our Santa Rosa data center had it’s load disconnected during routine maintenance.  Power was restored to approximately 3 minutes later at 3:45 PM.  The affected UPS handles both our own internal loads as well as our colocated customers.  The overwhelming majority of our systems are supplied with redundant power fed from two independent power systems and no public services experienced anything more than a transient failure.  More information will be forthcoming once our internal investigation of the procedural failure has completed.  – Kelsey

A configuration error is resulting in a small number of users encountering login errors when attempting to check mail. We are currently fixing the issue for any users experiencing the error.

-SOC

We are currently in the process of upgrading and improving our mail delivery infrastructure. During this process a configuration error was found that resulted in a small percentage of  users experiencing intermittent authentication errors when logging in to check mail. We are currently investigating.

UPDATE: Maintenance complete

Tonight beginning at 10:30pm we will begin routine maintenance of multiple public-facing systems, including webmail and VPN. Downtime for each system should be a few minutes, and the maintenance window is one hour.

– SOC

Update: Power has been restored to our Santa Rosa location.

We are currently experiencing a power outage at our Santa Rosa main office. The backup generator system is functioning correctly, so there is no interruption of service at this time. PG and E has an ETR of 1:15pm.

• SOC and Facilities