Category: Uncategorized

Fusion/Flexlink Outage

Update: The fibers were repaired and service restored around 2:30 AM this morning.

Update: AT&T has confirmed a fiber cut between Woodland and Davis, and have dispatched a crew to fix the affected span. No ETR at this time.

Today, at 12:25pm, vendor circuits serving the Woodland central office went down. We have opened tickets with our vendor and will work with them to get this issue resolved as soon as possible.

-Tomoc

Custom Databases Unreachable

At approximately 6pm today, our main customer database server suffered an error with its RAID controller. Operations is working to restore full functionality to the server as soon as possible.

Update: 8:35PM, we were able to restore services by 7:30 and are monitoring for further issues with the system.

— Joe and Sysops

Legacy DSL and FRATM Maintenace in Salinas

Tonight, between the hours of 12:00AM and 5:00AM, ATT will be performing intrusive maintenance on an aggregation circuit serving legacy DSL and FRATM circuits in the Salinas area. Expected downtime is less than one hour.

Update: Due to unforeseen circumstances, AT&T has pushed this maintenance to the same time window tonight.

Update: AT&T has postponed this maintenance again. New date TBD

-Tomoc

Legacy DSL Maintenance

Tonight, starting at 11:59pm, we will be performing maintenance on aggregation equipment serving a small subset of legacy DSL customers San Diego and the bay area. Expected downtime is less than 10 minutes.

Update: Maintenance complete.

-Tomoc

Server Upgrades

Update: These upgrades are complete.

 

Tonight at approximately 11:59pm, we will be performing maintenance upgrade on several customer facing systems. Expect brief outages to the following services:

  • Webmail
  • Mail and IMAP/POP3
  • Membertools
  • IPv6 tunnels
  • Customer Websites

All service interruptions should be brief.

 

-Grant and Dexter

System Operations

SSL certificate issues

At 9:50 am this morning we received reports of several of our SSL certificates being revoked. These certificates have been updated, and any security warnings for the mail.sonic.net, pop.sonic.net,  and imap.sonic.net should be resolved.

 

Grant and SOC

Legacy DSL and FRATM Maintenace in Salinas

Tonight, starting at 11:59PM, ATT will be performing intrusive maintenance on an aggregation circuit serving legacy DSL and FRATM circuits in the Salinas area. Expected downtime is less than two hours.

 

-Tomoc

Intrusive Fiber DIA Maintenance

Tonight, April 28, starting at 11:59PM, we will be performing maintenance on aggregation gear servicing a subset of Fiber DIA subscribers. Expected downtime is less than 30 minutes.

Update: No customer traffic was affected during this maintenance window. Due to unforeseen circumstances, this maintenance is being postposed until the night of April 30, starting at 11:59PM.

Update: Maintenance complete.

– Tomoc and Robbie

Emergency Fusion/FlexLink Maintenance

Tonight, April 12, starting at 11:59PM, we will be performing intrusive emergency maintenance on equipment serving Fusion and FlexLink customers in the Cotati area. We estimate this maintenance will be completed within two hours.

-Tomoc and Clay

Heartbleed update

Hello,

All vulnerable keys have now been replaced. We still recommend changing your account password to mitigate the damage if this exploit was attacked on our system. The systems included in this last round of updates where:

  • legacy-webmail.sonic.net
  • forums.sonic.net
  • wiki.sonic.net

– Grant and SOC