Month: November 2005

News server issues and maintenance.

Sun Nov 27 11:23:43 PST 2005 — News server issues and maintenance. Another configuration problem revealed itself over the past few days with the new news cluster. During troubleshooting some of the local sonic.* groups accidentally had articles older than 30 days purged from the overviews. The affected groups are sonic.announce, sonic.antispam, sonic.general, sonic.help.colo, sonic.help.www, sonic.os.unix and sonic.net. I should be able to restore all articles in the groups but it is impossible to match all of the article numbering previously assigned. As a result, all posts made after the migration to the new cluster will show as new again. While the restoration is taking place the feeds to the reader boxes will be paused and no new news will arrive. Posts made locally will be fed to all of our peers but will not appear until the internal feeds are restarted. -Kelsey

Update: Sun Nov 27 14:00:16 PST 2005 — The overviews for the affected groups have been rebuilt. However, this was not a perfect or seamless process. While the archives have been rebuilt there are some apparent issues. First, a few posts from this morning seem to have been lost. Second, some of the groups will exhibit duplicate articles for the past couple of weeks. Additionally, some users may need to unsubscribe and resubscribe or otherwise purge their client’s caches on the affected groups. I apologize for the bumps in the deployment of the new news cluster. -Kelsey

Shell server upgrade.

Wed Nov 23 07:54:14 PST 2005 — Shell server upgrade. The new shell server is currently on the Net as “newshell.sonic.net”. After a suitable migration period, it will become “Bolt.sonic.net” (and “Shell.sonic.net”). The server allows customers to execute Linux commands, compile Linux programs (which might be cgi-bin’s), and perform operations generally expected on a Unix timeshare system.

The system sports dual hyper-threaded 2.8GHz Xeon processors bolted in to a motherboard with 2GB of RAM. The system is running our standard production Linux distribution, Redhat 7.3 (with patches to make the system current.)

While we’ve made efforts to provide a shell environment similar to that found on the current Bolt.sonic.net, there will be differences. Indeed, many programs have been updated, while some we did not install on the system to reduce cruft. The idea: anything missing that a customer would like installed on the new system can be installed upon request. Discussions regarding such requests — as well as any problems encountered or questions — can be found on the following Sonic.net newsgroup:

news://news.sonic.net/sonic.net

(Note: not all browsers understand the news:// protocol. One browser that will read news:// is Mozilla/Thunderbird or Mozilla/Mail and News. www.mozilla.org )

-Scott … and Dane, who is a serious shell user.

Support hours for Thanksgiving.

Mon Nov 21 18:58:58 PST 2005 — Support hours for Thanksgiving. On Thursday, Nov 24, Sonic.net technical support will be closed after 5 pm in observance of Thanksgiving. We will return to our regular schedule (7am to 11pm) on Friday. Happy Holidays! -Eli and the Support team

News Server Issues.

Mon Nov 21 13:21:52 PST 2005 — News Server Issues. A flawed configuration on one of the processes in the new news server cluster led to the discarding of a large number of articles. This flaw only became apparent recently now that the systems have been in service for some time. We’ve corrected the problem and from now on there should be no further completion issues with the new news server. For more information please see news://news.sonic.net/sonic.general -Kelsey

DSL outage.

Sat Nov 12 22:50:51 PST 2005 — DSL outage. One of our DSL termination devices in Santa Rosa has crashed and requires manual intervention. A technician is en route to fix the trouble. ETR is approximately 30 minutes. Sorry for the inconvenience, and remember that all DSL accounts come with dialup access — perfect for emergency access during short outages! -Nathan

Update: Sat Nov 12 23:10:12 PST 2005 — DSL customers in Santa Rosa are currently up, but we have diagnosed a failure in the termination box’s hardware. We will be replacing the faulty card with an on-site spare in approximately 3 minutes. Additional downtime should be around 10 minutes. -Nathan

Update: Sat Nov 12 23:21:52 PST 2005 — All DSL customers in Santa Rosa are back on-line after the hardware swap, and everything looks fine. Sorry for the outage! -Nathan and Kelsey

A line card in one of our Cisco routers in…

Thu Nov 10 11:52:57 PST 2005 — A line card in one of our Cisco routers in Santa Rosa crashed. This currently only affects customers dialing our ATG dial pools in Santa Rosa. We anticipate services will be restored to these customers in 10 to 15 minutes. -Kelsey, Nathan and Zeke

Update: Thu Nov 10 11:59:06 PST 2005 — All services have been restored. The ATG pop was down for a total of 8 minutes.

Update: Thu Nov 10 12:10:20 PST 2005 — We will be performing maintenance early tomorrow morning at approx. 12:01AM on the affected router which will result in 15-20 minutes of intermittent connectivity to our ATG POP in Santa Rosa. – Network Operations

Update: Thu Nov 10 15:53:27 PST 2005 — During the maintenance tonight we will also take the opportunity to implement some network improvements to our shell server, shell.sonic.net. We expect less than 5 minutes of network downtime to complete this upgrade.

Update: Fri Nov 11 01:02:47 PST 2005 — All network maintenance is completed. The affected line card has been replaced without incident and is functioning properly. shell.sonic.net was unavailable for less than 2 minutes as it was moved to a more redundant network.

Intermittent Broadlink outages continue.

Wed Nov 9 15:34:01 PST 2005 — Intermittent Broadlink outages continue. Broadlink operations is working on correcting the problem and hope to have it resolved soon. -Support

Update: — Broadlink has replaced the red-c and service has been restored as of 8pm. -Support

Broadlink Outage.

Tue Nov 8 11:06:51 PST 2005 — Broadlink Outage. A key piece of Broadlink’s head-end equipment has experienced a fault. They are currently working to correct the problem now and hope to have services restored shortly. -Kelsey

Update: Tue Nov 8 11:13:26 PST 2005 — Broadlink services have been restored. -Kelsey