Month: May 2014

Server Upgrades

Update: These upgrades are complete.

 

Tonight at approximately 11:59pm, we will be performing maintenance upgrade on several customer facing systems. Expect brief outages to the following services:

  • Webmail
  • Mail and IMAP/POP3
  • Membertools
  • IPv6 tunnels
  • Customer Websites

All service interruptions should be brief.

 

-Grant and Dexter

System Operations

Brief Mail Service Interruption

One of the interfaces in the load balancer that handles, mail.sonic.net, pop.sonic.net and imap.sonic.net had an interface lock up and stop passing traffic.  During the few minutes this was an issue, these services would have been intermittently available.  Traffic has been manually rerouted through the standby load balancer.  Under nearly every other failure possible the standby load balancer would automatically take over but, unfortunately, its ability to deal with this lockup is not one of them.  -Kelsey

Fusion/FlexLink Maintenance

Update(1:30AM): This maintenance is now complete.

This evening, beginning at 11:59PM, we will be performing intrusive maintenance on equipment serving Fusion and FlexLink customers in Santa Rosa, Windsor, Cotati, and Rohnert Park. Expected downtime is less than 15 minutes. A small portion of customers may need to reboot their DSL modems after these code upgrades take place.

– Robbie

Fusion/FlexLink Intrusive Maintenance

Update(4:30AM): This maintenance is now complete.

This evening, beginning at 11:59PM, we will be performing intrusive maintenance on equipment serving Fusion and FlexLink customers in Santa Rosa. Expected downtime is less than 15 minutes.

– Robbie

SSL certificate issues

At 9:50 am this morning we received reports of several of our SSL certificates being revoked. These certificates have been updated, and any security warnings for the mail.sonic.net, pop.sonic.net,  and imap.sonic.net should be resolved.

 

Grant and SOC

Legacy DSL Outage – Bay Area

Update: These customers are now be back online.

A small portion of legacy DSL customers are currently offline in the bay area due to a failed optical port in our network. We are currently in the process of migrating those customers off of the failed port to restore service asap.

– Robbie

Fusion/FlexLink Intrusive Maintenance

Update (12:40AM): This maintenance is now complete.

This evening, beginning at 11:59PM, we will be performing intrusive maintenance on equipment serving Fusion and FlexLink customers in Larkspur. Expected downtime is less than 15 minutes.

– Robbie

Fusion/FlexLink Outage – LA

Update(9:45AM): The failed router has been replaced and all affected customers are now back online.

Update (7:00AM): We are working on getting a entire chassis replacement to this location at this time. A technician is en route but we dont quite have a firm ETR yet.

A small portion of customers in the LA/Sunset Blvd area are currently offline due to trouble with a router in the area. We are currently investigating the cause of the problem and will update this motd once we have a ETR.

– Robbie

Fusion/FlexLink Intrusive Maintenance

Update (2:33AM); This maintenance is now complete.

This evening, beginning at 11:59PM, we will be performing intrusive maintenance on equipment serving Fusion and FlexLink customers in Ukiah. Expected downtime is less than 15 minutes.

– Robbie

Legacy DSL and FRATM Maintenace in Salinas

Tonight, starting at 11:59PM, ATT will be performing intrusive maintenance on an aggregation circuit serving legacy DSL and FRATM circuits in the Salinas area. Expected downtime is less than two hours.

 

-Tomoc