The System Operations group will be performing general maintenance and upgrades on Sonic.net’s internal database servers this Thursday, June 20th, at 11:40pm. Work is expected to be complete by 11:59pm. During this time, services such as Member Tools may be temporarily unavailable.
UPDATE:
Upgrades are complete as of 1:00AM on Friday, June 21st.
— Joe and the System Operations group
This morning, from around 2:10AM to 2:25AM, a large DDoS attack targeting an IP within our Santa Rosa datacenter caused reachability problems to some Sonic.net member and hosting services. The attack has been blocked and all reachability restored.
– Robbie
Yes indeed — for whatever reason, Facebook.com is down.
There are a couple of places to verify this, independent of Sonic.net:
http://www.isitdownrightnow.com/facebook.com.html
http://downforeveryoneorjustme.com/facebook.com
(Please note that these sites, too, will be sporatic, as everybody checks to see if Facebook is, indeed, down.)
These things happen sometimes, and I’m sure they appreciate your patience as they scramble like crazed bunnies to fix things. 🙂
-Scott
Update: Facebook.com is back up.
At about 8:15 AM this morning a utility pole providing power to our 200 Paul Avenue datacenter in San Francisco caught on fire and took the building off of utility power. The datacenter is currently operating on generators and did not have any disruption in service during the switchover. While emergency crews are currently working on the damaged pole, they expect the repair to be a somewhat slow process and we are in high alert until the situation has been resolved.
– NOC
We are presently investigating reported issues of unreachability for business customers on the Sonic Flexlink Voice platform. We will post further update once we have determined the scope of the problem and/or resolved it.
-Sonic NOC
Update: A configuration change to our network caused an unintended side effect of making Flexlink Voice customers unable to receive inbound calls. The change was quickly reverted to restore service and safeguards are being put in place to prevent future repeat.
We are currently experiencing a power issue affecting some portion of our customers served out of our San Francisco point of presence. We are working with the facility to determine the cause of the issue and an estimated time of resolution. This outage is affecting Fiber DIA customers and may have caused some temporary DNS resolution issues for other customers. We will provide further updates shortly.
-Tim, Nathan and the NOC
Update: Power has stabilized as of around 10AM. All affected customers should be back up. We are working further with facility technicians to determine what caused the outage.
We are hearing scattered reports of reachability problems affecting a small
number of remote endpoints behind a specific service provider. Only a subset
of our network appears to be affected by this. We have been in touch with
this service provider, and they have diagnosed the issue to a bad line card
in a particular router. They are presently working with their vendor to get
that card repaired.
Due to the makeup of the Internet we’re unable to effectively re-route
traffic around this failing infrastructure. We’ll make sure that this
provider drives to resolution.
-Nathan and the NOC
Akamaihd.net and rackcdn.com both ended up one of the RPZ zones that we subscribe to this morning and were whitelisted by our staff as soon as it came to our attention. Â These listings could have prevented the proper loading of many popular websites. Â We’re working with our vendor to ensure that listing like this do not happen again.
-Sonic Ops Team
Tonight, June 6th, starting at 11:59 PM, we will be performing power work on a small subset of equipment serving legacy products in one of our San Francisco datacenters. We do not expect any  customer impact from this procedure.
– Robbie
Update: This maintenance is now complete
Sonic.net Support and Sales will close at 10:00pm Friday May 17th for our employee movie night. Support and Sales will open at the regular time Saturday morning.
– John F. and the Sonic.net Staff.