Tonight, June 24, starting at 11:59pm, we will be performing maintenance on our northern California telecom network. We do not expect any customer down time from this maintenance.
Update: Due to unforeseen circumstances, some telecom customers may have experienced trouble reaching certain web destinations for a brief period. This maintenance window is now closed.
-Tomoc
We are currently experiencing an issue with our ATM backhaul for legacy DSL in the Salinas area. The issue is still under investigation. We will update this MOTD as we work towards a resolution.
– Robbie and Tim
Update 14:48 – We are working with our transport provider to isolate and repair the issue. There is no ETR at this time but we will provide further updates as soon as possible.
Update 17:37 – The issue has been resolved as of 16:45. We are awaiting an RFO from the carrier.
UPDATE:
All work has been completed on the database server as of 12:02am, June 22nd.
—
Tonight, June 21st, from 11:40pm to 11:59pm, System Operations will be replacing a failed component in one of our internal database servers. During the maintenance window, some web-based services such as webmail and Member Tools may be unavailable.
— Joe and the System Operations Center
The System Operations group will be performing general maintenance and upgrades on Sonic.net’s internal database servers this Thursday, June 20th, at 11:40pm. Work is expected to be complete by 11:59pm. During this time, services such as Member Tools may be temporarily unavailable.
UPDATE:
Upgrades are complete as of 1:00AM on Friday, June 21st.
— Joe and the System Operations group
This morning, from around 2:10AM to 2:25AM, a large DDoS attack targeting an IP within our Santa Rosa datacenter caused reachability problems to some Sonic.net member and hosting services. The attack has been blocked and all reachability restored.
– Robbie
Yes indeed — for whatever reason, Facebook.com is down.
There are a couple of places to verify this, independent of Sonic.net:
http://www.isitdownrightnow.com/facebook.com.html
http://downforeveryoneorjustme.com/facebook.com
(Please note that these sites, too, will be sporatic, as everybody checks to see if Facebook is, indeed, down.)
These things happen sometimes, and I’m sure they appreciate your patience as they scramble like crazed bunnies to fix things. 🙂
-Scott
Update: Facebook.com is back up.
At about 8:15 AM this morning a utility pole providing power to our 200 Paul Avenue datacenter in San Francisco caught on fire and took the building off of utility power. The datacenter is currently operating on generators and did not have any disruption in service during the switchover. While emergency crews are currently working on the damaged pole, they expect the repair to be a somewhat slow process and we are in high alert until the situation has been resolved.
– NOC
We are presently investigating reported issues of unreachability for business customers on the Sonic Flexlink Voice platform. We will post further update once we have determined the scope of the problem and/or resolved it.
-Sonic NOC
Update: A configuration change to our network caused an unintended side effect of making Flexlink Voice customers unable to receive inbound calls. The change was quickly reverted to restore service and safeguards are being put in place to prevent future repeat.
We are currently experiencing a power issue affecting some portion of our customers served out of our San Francisco point of presence. We are working with the facility to determine the cause of the issue and an estimated time of resolution. This outage is affecting Fiber DIA customers and may have caused some temporary DNS resolution issues for other customers. We will provide further updates shortly.
-Tim, Nathan and the NOC
Update: Power has stabilized as of around 10AM. All affected customers should be back up. We are working further with facility technicians to determine what caused the outage.
We are hearing scattered reports of reachability problems affecting a small
number of remote endpoints behind a specific service provider. Only a subset
of our network appears to be affected by this. We have been in touch with
this service provider, and they have diagnosed the issue to a bad line card
in a particular router. They are presently working with their vendor to get
that card repaired.
Due to the makeup of the Internet we’re unable to effectively re-route
traffic around this failing infrastructure. We’ll make sure that this
provider drives to resolution.
-Nathan and the NOC