Thu Oct 28 05:50:47 PDT 2004 — Packet loss to Equinix. We are currently experiencing moderate delay and packet loss from Santa Rosa to our location in Equinix San Jose. The vendor of our circuit between those locations has been notified, and they are working to correct the issue as soon as possible. -Nathan
Update Thu Oct 28 06:09:03 PDT 2004: The packet loss issue has been resolved. We are closely monitoring the circuit for any further issues.
Wed Oct 27 11:00:00 PDT 2004 — Value Hosting migrated to new web cluster. As of 11am this morning, Value Hosting sites are now hosted on our new web cluster, and all of their FrontPage webs have been converted to be compatible with FrontPage 2003.
We currently have someone checking each site to see if they come up, but they are not visiting the sites in-depth…so if you are a Value Hosting customer, we ask you to please test your website. Any reports are welcome at support@sonic.net or news://news.sonic.net/sonic.help.www -Scott, Kelsey, and John F.
Tue Oct 26 12:13:20 PDT 2004 — Broadlink TMAC tower repair complete. Broadlink reports that repairs were completed at 11:33. Some folks may need to power-cycle their Broadlink gear (at the power outlet) to restore service. Thanks to Jason and Linda from Broadlink for working the problem. -Scott, and Broadlink’s Jason and Linda
Tue Oct 26 11:59:13 PDT 2004 — Network Event. One of the SUP1A supervisor modules in one of our core switches failed causing packet loss and latency in our network. Once the failure was identified we swapped to the secondary supervisor and services were restored immediately. -Kelsey, Nathan, Russ and Zeke.
Tue Oct 26 11:02:56 PDT 2004 — Broadlink TMAC tower update. Broadlink reports an ETR of 1 hour, they are on site and working with Cisco engineers to repair the outage. -Scott, and Linda and Jason (from Broadlink)
Tue Oct 26 10:43:17 PDT 2004 — Value Hosting migration to new web cluster. Tomorrow, October 27th, at 11am, we will swing Value Hosting sites to the new web cluster. This should be a seamless migration.
Value Hosting customers using FrontPage will find their site migrated to the latest FP server extensions, which means their sites will now be compatible with FrontPage 2003.
After the migration, we will send an email to Value Hosting customers inviting them to test their web sites. We encourage these customers to test their sites at that time.
If you have questions or concerns regarding this migration, please visit the news://news.sonic.net/sonic.help.www newsgroup, or email support.
-Scott
Tue Oct 26 10:09:37 PDT 2004 — Broadlink TMAC backhaul down. At 9:43am today, the backhaul serving Broadlink customers in North Santa Rosa stopped passing traffic. Broadlink technicians are working on restoring service at this time. No other services should be affected, and we will update this status message as soon as possible. — John F
Tue Oct 26 16:19:59 PDT 2004 — Apache patched. Apache on our new web cluster is now patched to guard against a recently-released exploit. No downtime was incurred with the upgrade, and the exploit wasn’t used against us. -Scott and Augie
Tue Oct 26 14:24:43 PDT 2004 — Route flap. At 1:40pm, we had a layer 2 link to San Jose Equinix flap, causing momentary unreachability to some Internet destinations. -Scott and John
Mon Oct 25 17:28:00 PDT 2004 — Network Maintenance. Tonight at Midnight tonight we are going to be replacing some forwarding hardware on one of our new routers in San Francisco. While the upgrade is being conducted there will be degraded performance and potentially a few short service interruption as routing protocols converge. -Kelsey, Nathan and John
Update: We have confirmed that our previous issues were due to a bad forwarding card. After a quick replacement, everything looks great. The replacement itself required moving traffic off the router, which resulted in around 45 seconds of degraded service to a subset of Internet destination. This completes the first phase of Sonic.net’s new network build out — as of tonight, we have over 1.2 Gbps of capacity out of Santa Rosa! -Nathan and John