Month: June 2013

Sonic Telecom Maintenance

Tonight, June 24, starting at 11:59pm, we will be performing maintenance on our northern California telecom network. We do not expect any customer down time from this maintenance.

Update: Due to unforeseen circumstances, some telecom customers may have experienced trouble reaching certain web destinations for a brief period. This maintenance window is now closed.

-Tomoc

Legacy ATM Outage – Salinas

We are currently experiencing an issue with our ATM backhaul for legacy DSL in the Salinas area. The issue is still under investigation. We will update this MOTD as we work towards a resolution.

– Robbie and Tim

Update 14:48 – We are working with our transport provider to isolate and repair the issue. There is no ETR at this time but we will provide further updates as soon as possible.

Update 17:37 – The issue has been resolved as of 16:45. We are awaiting an RFO from the carrier.

Database Server Maintenance

UPDATE:

All work has been completed on the database server as of 12:02am, June 22nd.

Tonight, June 21st, from 11:40pm to 11:59pm, System Operations will be replacing a failed component in one of our internal database servers. During the maintenance window, some web-based services such as webmail and Member Tools may be unavailable.

— Joe and the System Operations Center

Database Server Upgrades

The System Operations group will be performing general maintenance and upgrades on Sonic.net’s internal database servers this Thursday, June 20th, at 11:40pm. Work is expected to be complete by 11:59pm. During this time, services such as Member Tools may be temporarily unavailable.

UPDATE:

Upgrades are complete as of 1:00AM on Friday, June 21st.

 

— Joe and the System Operations group

Unreachable Sonic.net Services

This morning, from around 2:10AM to 2:25AM, a large DDoS attack targeting an IP within our Santa Rosa datacenter caused reachability problems to some Sonic.net member and hosting services. The attack has been blocked and all reachability restored.

– Robbie

Facebook is back up.

Yes indeed — for whatever reason, Facebook.com is down.

There are a couple of places to verify this, independent of Sonic.net:

http://www.isitdownrightnow.com/facebook.com.html

http://downforeveryoneorjustme.com/facebook.com

(Please note that these sites, too, will be sporatic, as everybody checks to see if Facebook is, indeed, down.)

These things happen sometimes, and I’m sure they appreciate your patience as they scramble like crazed bunnies to fix things. 🙂

-Scott

Update: Facebook.com is back up.

 

Utility Power Distruption

At about 8:15 AM this morning a utility pole providing power to our 200 Paul Avenue datacenter in San Francisco caught on fire and took the building off of utility power. The datacenter is currently operating on generators and did not have any disruption in service during the switchover. While emergency crews are currently working on the damaged pole, they expect the repair to be a somewhat slow process and we are in high alert until the situation has been resolved.

– NOC

Sonic Flexlink Voice Outage

We are presently investigating reported issues of unreachability for business customers on the Sonic Flexlink Voice platform.  We will post further update once we have determined the scope of the problem and/or resolved it.

-Sonic NOC

Update: A configuration change to our network caused an unintended side effect of making Flexlink Voice customers unable to receive inbound calls.  The change was quickly reverted to restore service and safeguards are being put in place to prevent future repeat.

 

Network Issue

We are currently experiencing a power issue affecting some portion of our customers served out of our San Francisco point of presence. We are working with the facility to determine the cause of the issue and an estimated time of resolution. This outage is affecting Fiber DIA customers and may have caused some temporary DNS resolution issues for other customers. We will provide further updates shortly.

 

-Tim, Nathan and the NOC

Update: Power has stabilized as of around 10AM. All affected customers should be back up. We are working further with facility technicians to determine what caused the outage.

Upstream Routing Issue

We are hearing scattered reports of reachability problems affecting a small
number of remote endpoints behind a specific service provider. Only a subset
of our network appears to be affected by this. We have been in touch with
this service provider, and they have diagnosed the issue to a bad line card
in a particular router. They are presently working with their vendor to get
that card repaired.

Due to the makeup of the Internet we’re unable to effectively re-route
traffic around this failing infrastructure. We’ll make sure that this
provider drives to resolution.

-Nathan and the NOC