Category: Uncategorized

Bay Area and Los Angeles ATM Maintenance

Tonight, August 29, starting at midnight, we will be performing a maintenance reload of equipment serving some legacy DSL and Business-T customers in the greater bay area and Los Angeles. Expected downtime is 15 minutes.

Update(12:15am): Maintenance complete

-Tomoc

San Francisco – Fusion Fiber Outage

A small group of people in the San Francisco area may be experiencing a loss of internet connectivity after last nights maintenance.  We are working with our dispatch and tech support teams to get the affected customers fixed as soon as we can. The cause of the outage is unknown at this time as we are also working with our vendor for insight into the issue.

Brandon

Update 10:48am: The outage is affecting a greater number of customers.  We are working aggressively with our vendor.  We will update when more information is available.

Update 2:46pm We have repaired the majority of the fiber customers impacted by the outage.  Additional affected customers have been identified and we will be reaching out to schedule replacent of the failed ONTs onsite.  The cause of outage was narrowed down to the combination of a previously unidentified software defect in the ONT, and an unlikely load issue upstream (that has been remedied).

Scheduled Maintenance – Hosting Database Migration

Update: This maintenance is now complete.

Update: This maintenance is being pushed out one day to Wednesday, August 31st at 11:59pm.

On Tuesday, August 30th, beginning at 11:59pm we will be migrating the customer hosting MySQL database(s) to a new server.  The maintenance is expected to take up to 1 hour during which time access to the custsql servers will be interrupted.

-Dexter K, Sonic System Operations

North Bay Fusion Maintenance

Tonight, August 4, starting at 11:59pm, we will be performing maintenance on equipment serving a subset of Fusion and FlexLink subscribers in Healdsburg and San Francisco. Expected downtime is 15 minutes.

Update(12:31am): Maintenance complete

-Tomoc

System Maintenance

UPDATE: Maintenance complete.

 

Starting at 11:59pm SOC will be running system updates. The following services may experience brief interruptions:

  • Webmail
  • Membertools
  • Sonic.com website
  • SMTP/POP3/IMAP

Interruptions to individual services will be brief, and the Maintenance window is expected to last until 1am.

-Grant and SOC

Webmail Maintenance

On Wednesday, August 3rd at approximately 11:59 PM we will be performing upgrades on webmail.sonic.net.  The upgrades will take approximately 30-45 minutes to complete during which time webmail will be unavailable.  IMAP/POP/SMTP will not be affected and legacy-webmail can still be used during that time.

These are primarily maintenance fixes which include updating Roundcube to the latest version.

-SOC

Update: 2016/08/04 @ 12:22 AM

Maintenance has completed.

Web cluster instability

UPDATE (07/22/16 2:30PM): We have been dealing with a sporadic attack towards value hosting customers during the last two days. Value hosting services were restored at approximately 10PM last night. These attacks also affected colocation services, and NOC has taken steps to reduce the impact if another denial of service attack occurs.

UPDATE: NOC is currently blocking parked.sonic.net on our network edge, since this is the destination of the attack. This is resulting in reachability issue for customers with the value hosting service, and we are reviewing other options.

Our web cluster is currently experiencing a denial of service attack. Stability may fluctuate while Operation mitigates the attack.

 

SOC

Incoming mail configuration error

Recently senders using a google.com address began receiving bounce messages when sending to sonic.net users. This was due to an erroneous configuration on our end, and the problem has been resolved.

 

SOC

Webcluster Outage.

Starting at 1:30 pm we experienced a service outage for a large segment of our shared hosting cluster. Services have been restored, and operations is investigating the cause of the issue.

Grant and SOC

System Maintenance

UPDATE: Maintenance complete.

Starting at 11:59pm SOC will be running system updates. The following services may experience brief interruptions:

  • Webmail
  • Membertools
  • Sonic.com website
  • FTP service

Interruptions to individual services will be brief, and the Maintenance window is expected to last until 1am.

-Grant and SOC