Category: Uncategorized

San Francisco Fusion Fiber Service Interruptions

Early this morning, we updated firmware on equipment serving some San Francisco Fusion Fiber subscribers. This new firmware has proven unstable and has caused intermittent service interruptions throughout the day. We have been working with our vendor to resolve the issue as quickly as possible and ensure it does not continue in the future.

 

-Tomoc and Brandon

 

Update (10:00pm)  We have isolated and fixed the issue, service should now be restored/stable.

-Brandon

Sebastopol Fusion Fiber Maintenance

Tonight, April 27, starting at 11:59pm, we will be performing software updates on equipment serving Fusion Fiber customers in Sebastopol.  Estimated time of completion is 3am.

Update(12:29am): Maintenance complete

-Tomoc

San Francisco Fusion Fiber Maintenance

Tonight, April 25, starting at 11:59pm we will be performing software updates on equipment serving a subset of Fusion Fiber customers in the Sunset area.  Estimated time of completion is 3am.

Update(3:42am): Maintenance complete.

-Tomoc

Chico ATM Outage

This morning, April 8, at 5:30am, an aggregation circuit serving legacy DSL and Business T subscribers in the Chico area went down. We are currently investigating the issue.

Update(10:32am): We are working with our transport vendor to restore traffic as soon as possible. No ETA at this time

Update(4:26pm): An upstream provider has located an equipment failure and is working to replace the failed equipment. Still no ETR.

Update (4/9/2016 9:40AM): The upstream provider now has the replacement parts on site at the location of the failure, and an engineer is currently working on replacing the failed equipment. We will update as soon as we have more information.

Update (4/9/2016 1:21PM): The failed equipment has been replaced by the upstream provider, but the circuit is still down. We are continuing to work with our upstream provider to resolve this.

Update (4/9/2016 3:28PM): Our upstream provider has identified a second failure, and is actively working to replace the failed equipment. We hope to have an ETR soon.

Update (4/9/2016 6:04PM): Our upstream provider has the replacement part in transit, and the ETR is now 3AM PST.

Update (4/10/2016 5:58AM): Our upstream provider has experienced delays in transporting the replacement equipment to the failure area, and they have given us a new ETR of today at 1:00PM PST.

Update (4/10/2016 1:33PM): The replacement equipment is now on site, and technicians are currently working on installing it.

Update (4/10/2016 3:45PM): The equipment has been replaced, but the issue persists. Our transit provider is continuing to troubleshoot the cause of the outage. There is no ETR at this point.

Update (4/10/2016 6:35PM): The upstream provider has identified and resolved multiple equipment failures in their network causing this outage.  All services were restored as of 6:00PM PST.

-Tomoc and Michael

San Francisco Outage

A small subset of Fusion customers served in the San Francisco area experienced an interruption of service this morning. The issue appears to have stemmed from a code upgrade that was done on our equipment last night. We are working with our vendors to ensure that upgrades in the future will no longer cause an issue.

 

  • NOC

Webmail Upgrade

On Friday, April 15th at approximately 11 PM, webmail.sonic.net will be temporarily unavailable while it is upgraded.  We expect this to take approximately 1-1.5 hours to complete.  Mail services such as IMAP and POP will remain available and webmail-beta.sonic.net can be used during this time to retrieve mail.

Please see this forum post for more information.  If you haven’t already, we encourage you to try the beta and report any issues regarding features or functionality that you use in the current version.  Please note that settings between beta and production are separate, but all mail and folders are synchronized between the two.

SOC

Fusion/FlexLink Router Maintenance

Tonight, April 6, starting at 11:59pm, we will be performing maintenance on equipment serving subsets of Fusion and FlexLink customers in the following areas. Expected downtime is 20 minutes.

Affected areas include Forestville, Pittsburg, Oakley, San Carlos

Update(12:54am): Maintenance complete

-Tomoc

System Maintenance

UPDATE: Maintenance complete.

Beginning at 11:59pm this evening, Operations will be performing routine system maintenance. The following services may experience brief interruptions:

  • OpenVPN
  • IMAP/POP3
  • IPv6 Tunnels
  • Membertools
  • Custom Hosting Containers
  • Custom Web

Thanks,

SOC

Flexlink Long Range Outage in Albany, CA

This morning Apr 05 at 7:20am a router aggregating Flexlink Long Range customers out of Albany, CA developed problems. We have dispatched a technician to resolve the issue.

Michael

 

Update: 10:25am – Service is now restored.

-Brandon

Sonic Telecom Router Maintenance

Tonight, April 4, starting at 11:59pm, we will be performing maintenance on equipment serving subsets of Fusion and FlexLink customers in the following areas. Expected downtime is 20 minutes.

Affected areas include Brentwood, Forestville, Pittsburg, Oakley, San Carlos, Clayton, Antioch and Los Angeles

Update(3:16am): Due to a software issue, equipment serving Brentwood and Clayton customers are currently unreachable. We are dispatching to restore connectivity immediately.

-Tomoc

 

Update (6:30am): All service has been restored.

-NOC