Today, starting at around 9am, a software bug caused intermittent connectivity issues to telecom customers in the San Pedro area. An emergency reload of this equipment has stabilized all connections, and we will be working with our hardware vendors to ensure this does not happen again.
-Tomoc
Tonight, July 3, beginning at 11:59PM, we will be performing minor maintenance on routers that serve Biz-T and FRATM customers in both northern and southern California. Expected down time is less than 10 minutes.
Update: Maintenance complete.
-Tomoc
Tonight, June 24, starting at 11:59pm, we will be performing maintenance on our northern California telecom network. We do not expect any customer down time from this maintenance.
Update: Due to unforeseen circumstances, some telecom customers may have experienced trouble reaching certain web destinations for a brief period. This maintenance window is now closed.
-Tomoc
We are currently experiencing an issue with our ATM backhaul for legacy DSL in the Salinas area. The issue is still under investigation. We will update this MOTD as we work towards a resolution.
– Robbie and Tim
Update 14:48 – We are working with our transport provider to isolate and repair the issue. There is no ETR at this time but we will provide further updates as soon as possible.
Update 17:37 – The issue has been resolved as of 16:45. We are awaiting an RFO from the carrier.
UPDATE:
All work has been completed on the database server as of 12:02am, June 22nd.
—
Tonight, June 21st, from 11:40pm to 11:59pm, System Operations will be replacing a failed component in one of our internal database servers. During the maintenance window, some web-based services such as webmail and Member Tools may be unavailable.
— Joe and the System Operations Center
The System Operations group will be performing general maintenance and upgrades on Sonic.net’s internal database servers this Thursday, June 20th, at 11:40pm. Work is expected to be complete by 11:59pm. During this time, services such as Member Tools may be temporarily unavailable.
UPDATE:
Upgrades are complete as of 1:00AM on Friday, June 21st.
— Joe and the System Operations group
This morning, from around 2:10AM to 2:25AM, a large DDoS attack targeting an IP within our Santa Rosa datacenter caused reachability problems to some Sonic.net member and hosting services. The attack has been blocked and all reachability restored.
– Robbie
Yes indeed — for whatever reason, Facebook.com is down.
There are a couple of places to verify this, independent of Sonic.net:
http://www.isitdownrightnow.com/facebook.com.html
http://downforeveryoneorjustme.com/facebook.com
(Please note that these sites, too, will be sporatic, as everybody checks to see if Facebook is, indeed, down.)
These things happen sometimes, and I’m sure they appreciate your patience as they scramble like crazed bunnies to fix things. 🙂
-Scott
Update: Facebook.com is back up.
At about 8:15 AM this morning a utility pole providing power to our 200 Paul Avenue datacenter in San Francisco caught on fire and took the building off of utility power. The datacenter is currently operating on generators and did not have any disruption in service during the switchover. While emergency crews are currently working on the damaged pole, they expect the repair to be a somewhat slow process and we are in high alert until the situation has been resolved.
– NOC
We are presently investigating reported issues of unreachability for business customers on the Sonic Flexlink Voice platform. We will post further update once we have determined the scope of the problem and/or resolved it.
-Sonic NOC
Update: A configuration change to our network caused an unintended side effect of making Flexlink Voice customers unable to receive inbound calls. The change was quickly reverted to restore service and safeguards are being put in place to prevent future repeat.