This Thurday, June 16 from 12:01 AM to 3:00 AM, we will be inserting new cards into the backbone routers for our CLEC network to add capacity. This work should not cause any impact to customers, but if something goes awry, impact should be minimized to about 60 seconds of Internet connectivity loss and the possibility of a lost phone call.
-Jared and Matt
Update: All hardware maintenance has been completed with no impact to customers.
The network operations hotline for business support is currently experiencing an outage. We are working with the phone provider to restore service as fast as possible, but there is no ETR at this time. If you presently need to contact the NOC, please call the main Sonic 707-522-1000 number and request to be transferred.
-Matt and the NOC
Update: Service has been restored for this number and the NOC may again be reached normally.
We’ve seen a big up-tick in traffic here due to the coverage of our Fiber deployments. In the last few days, the Press Democrat, Ars Technica, ZDNet, MaximumPC and Engadget have all written about Sonic.net. Check out the coverage:
http://www.pressdemocrat.com/article/20110607/ARTICLES/110609527?p=all&tc=pgall
http://arstechnica.com/tech-policy/news/2011/06/1gbps-fiber-for-70in-america-yup.ars
http://www.zdnet.com/blog/btl/sonicnet-launching-1gbps-fiber-to-home-internet-for-70-per-month/50441
http://www.maximumpc.com/article/news/1gbps_internet_access_70_america_sonic_says_yes
http://www.engadget.com/2011/06/13/sonic-net-starts-trial-of-1gbps-fiber-to-the-home-internet-in-ca/
Don’t worry though – we’ve got plenty of capacity to handle an influx of Fusion subscribers, and web and provisioning traffic loads are not a problem. If you want Fusion, sign up now, and help us build the momentum toward Fiber.
At approximately 9:15 am this morning an OC12 serving a roughly a quarter of our legacy DSL connections in the Bay Area went down. Our Network Operations Center responded to the problem quickly but the outage cleared on it’s own within 7 minutes. We have a ticket open with our vendor to determine the root cause.
-Chris
Tonight, June 8 at 12:01 AM we will be performing hardware maintenance on one of our ATM switches in San Francisco. This ATM switch serves DSL in the Bay Area and has been experiencing hardware problems recently. The maintenance will involve replacing the suspected problem cards in the chassis to restore normal service. Expected downtime is 5-10 minutes.
-Jared
Update: The hardware replacement has been completed and all affected customers are back online.
At approximately 10:33pm tonight one of the devices serving a small subset of DSL connections in the Bay Area went down. Our Network Operations Center responded to the problem, and service was restored by 11:08pm. We’re presently working to isolate the root cause of this failure. -Nathan
Tonight, Saturday May 28 at 12:01AM, we will be performing several minor DSL maintenance tasks. This work will disrupt some DSL service in the LATA 1 (Bay Area) and LATA 8 (Salinas area) for 5-10 minutes.
-Jared, Nathan and Matt
Tonight at about 6:15 PM the ATM switch serving our newest ATM OC12 suffered a software failure and had to be rebooted to restore service. The reboot caused 5-10 minutes of downtime for affected customers. The switch is back up and operating normally at this time, and we are continuing investigation into the software error.
-Jared
One of our authoritative name servers, b.auth-ns.sonic.net, had a hardware failure last Friday and services were restored on temporary hardware shortly thereafter. (We don’t think these kinds of non-service impacting events warrant an MOTD.) Rather than fix the original hardware, we decided it was a good opportunity to further increase the geographic and network diversity of our authoritative name servers and have moved it to a facility in Texas. Our authoritative name servers are now located in three different networks, all with IPv6, in California, Texas, and New York. -Kelsey
The article numbering server in our news cluster suffered a catastrophic failure a couple of hours ago. All services have been restored on to the backup server but there may be some delay in new articles as the servers catch up. It is possible that some articles were lost and cannot be re-fed into the system.
Update: Due to an error promoting a reader slave (this is something we’ve only had to do once before, many years ago) to the article numbering master server, the article numbers being assigned to new articles were grossly incorrect. As such, all articles received between when services were brought back online and the article numbering server was fixed, are lost to our readers. I’m sorry we didn’t catch this sooner.
Update: The news cluster continues to have some issues that we are working to iron out. The replacement article numbering server is having trouble keeping up with the feed and we’ve had to force it to catch up by flushing out its back log (loosing the articles) several times. Please note that this only affects our overviews (article lists) and not our ability to retrieve articles by message-id from our spools. We’re hope to have the situation stabilized soon.
-Kelsey