Author: admin

Tue Sep 4 16:07:16 PDT 2001 — Routing hiccup. We were resolving a minor BGP configuration error in our three core routers, mega, gamma, and delta, and unexpectedly prevented customers connected via mega from being able to access anything outside of our network. The problem existed for approximately ten minutes before it was, brought to our attention. This effected T3, T1, Frame Relay, and remote pop users. We’ve identified the problem that led to this and are working to fix it so we can go back to fixing BGP. We apologize for this brief service interruption. -Nathan, Scott, Eli and Kelsey.

Mon Sep 3 18:33:35 PDT 2001 — News Server downtime. News.sonic.net was offline between 5:25p and 6:25p this afternoon. The NNTP software package has been failing intermittently over the last 2-3 days, and in this case a delay in our notification package led to excessive downtime. We’ve tightened up monitoring, and will continue to work out the issues with the Typhoon software. – Eli, Kelsey

UPDATE: Typhoon, the news server software, has decided that it’s best action is to exit and dump core. We are in contact with the software vendor and waiting for a call back. Gale, our news feeder server is capable of back-logging about 6 hours of news for typhoon so as long as we are able to get it up within that time frame, no news posts should be lost. – Kelsey and Eli

UPDATE: We are seeing what appears to be some overview database corruption on typhoon. This manifests itself as garbled subject lines and/or the wrong post being returned for a given message id. In the two years that we’ve been running typhoon we’ve never seen anything like this before, in fact, the server had an uptime of 282 days until tonight. We are going to continue to work with the software vendor for a prompt resolution to this problem which will hopefully not involve the loss of our historical news spools. In the meantime if a group that you are reading exhibits the corruption use our backup news server, “supernews.sonic.net” in place of “news.sonic.net” -Kelsey, Dane, Nathan and Eli.

Fri Aug 31 18:18:07 PDT 2001 — Emergency maintenance on NetApp filer. One of the loop resiliency circuit cards in one of our NetApp filers has failed. While this is redundant equipment and didn’t cause any immediate impact, our redundancy is compromised, so we consider this a very high priority repair.

Saturday morning at 1:00AM, we will be shutting down freezer.sonic.net, our NFS filer hosting all storage for email, shell and ftp. This means that these services will not be available for about five to ten minutes. Normal maintenance of these filers does not require any downtime, and a second CPU/head allows for full redundancy in a failure or normal maintenance, however, in this case, the LRC card is what supports the redundant filer CPU/head. -Kelsey, Nathan and Dane

Fri Aug 31 17:14:54 PDT 2001 — Sonic.net has deployed a new network time protocol (NTP) server for customer use. You may synchronize your system clocks with the NTP protocol to time.sonic.net, aka ntp.sonic.net. “time” is a stratum 2 atomic synchronized time server for use by Sonic.net customers.

Client software is available for a variety of platforms. For Windows users, we recommend AboutTime, available at the following URL:

www.arachnoid.com/abouttime/index.html

For additional NTP resources, see www.ntp.org/.

-Eli and Dane

Fri Aug 31 13:54:24 PDT 2001 — One of our Santa Rosa pops was returning a ‘All Circuits are Busy’ error today, we narrowed it down to a PRI card that was acting up and rebooted it. -Steve

Update: Rebooting the PRI card did not clear up the problem as I had thought it would. I have replaced the card and tested it, and all looks well now. -Steve

Update #2: The All Circuits are Busy message came back. We opened a ticket with PacBell and also did a hard reboot on NAS21, which is the front of our 1003 dialup pool. This caused some people to be disconnected. The hunt group appears to be taking calls just fine now I will be watching it for the rest of the day. -Steve

Thu Aug 30 11:39:32 PDT 2001 — Update on BroadLink tower outage. Estimated time of repair is 3:00pm today. 100 BroadLink/Sonic.net customers are affected by this outage in the Southern Santa Rosa area.

On Monday morning, the licensed frequency 23 Ghz equipment serving the backhaul from the South SR tower failed. It was replaced with spare gear, which worked for a few hours, failed again Monday afternoon, then worked through Tuesday. It failed completely on Wednesday morning around midnight.

The vendor for this equipment is out of business, and BroadLink was unable to obtain technical assistance or any additional equipment beyond the spare that failed.

On Wednesday, BroadLink worked to bring up a 2.4 Ghz link (802.11b) to backhaul traffic from this tower site, but at around a 1.5 Mbps speed, this link was completely unable to handle the load requirements.

This AM, BroadLink has taken equipment out of service that was serving a link from their Dutton main office, and is moving it to downtown Santa Rosa to serve the backhaul for the South tower site. This 5 Ghz link has a max speed of 8 Mbps in this configuration, and will be adequate to support the traffic from the tower.

Technicians are currently on site in downtown Santa Rosa and at the tower site, and hope to have the equipment fully online by 3pm.

BroadLink asked me to pass along their apologies for the extended outage to the 100 affected customers. They’ve had an unforeseen situation where spare equipment failed after a brief time in service, and the vendor is no longer in business. (Co-incidence? Hmm..) The long term goal is to deploy backup connectivity to each tower site in this market to assure reliability in case of a link failure. -Dane, and from BroadLink, Chris, Shane, Randy and Tim.

Update: BroadLink has repaired their backhaul link, and has in fact made the network more efficient by removing one hop from the connection.

Thu Aug 30 02:04:43 PDT 2001 — Maintenance on the 9811 dialup pool has been completed. Thank you for your patience while we worked on this gear. -Steve and Jeff.

Wed Aug 29 09:05:32 PDT 2001 — BroadLink is having problems with the link to their South Santa Rosa tower site, and customers served by that tower currently are offline. BroadLink reports that they have a technician on site, and are working to troubleshoot the problem now. -Dane

Update, Wed Aug 29 17:06:43 PDT 2001 BroadLink’s licensed RF backhaul has failed, and they’re working to bring a replacement on line. No ETR is available at this time.

Update, Wed Aug 29 18:00:00 PDT 2001 A replacement link is online, but performance may be slower than usual. We will provide additional updated information here on Thursday morning.

Wed Aug 29 17:06:01 PDT 2001 — Sonic.net is now offering free additional IP addresses for PacBell DSL connected customers. For customers with multiple PCs, this allows for sharing of the DSL link between up to four computers simultaneously. Previously priced at $12 per month, Sonic.net is now including a block of four IP addresses for all PacBell DSL customers at no charge. Eight IP addresses are available for $12 per month. (Existing customers who have four or eight IP blocks will see pricing moved down to $0 or $12, respectively on their next invoice)

Using an Ethernet hub, this allows customers to share their DSL connection with multiple PCs in the home or office. No additional equipment is needed, and setup is simple.

Sonic.net is committed to continuing to deploy additional features and services on both our broadband and dialup services. We hope that this new feature adds value to our DSL offering.

You can migrate from the single IP initially assigned to a new block of four or eight IP addresses using our online member tools, linked from the home page at www.sonic.net/. This is a new standard ongoing service offering, so please, to conserve IP address space, only migrate if and when you have multiple machines that you need to address. For any questions or discussion about this new feature, please see news:sonic.dsl

Note that BroadLink wireless service uses equipment which already provides access for multiple PCs, so this change does not affect BroadLink customers.

-Kelsey, Nathan, Dustin and Dane

Mon Aug 27 15:10:06 PDT 2001 — BroadLink has had problems with the backhaul link that serves Southeastern Santa Rosa. The link failed this morning between 10:23a and 10:57a, and a replacement link was brought up. This afternoon @ 1:30p, a configuration error in the backup link caused intermittent outages for some customers between 1:30p and 3:00p. Broadlink’s engineers are currently monitoring the link to prevent further issues, and may well schedule some maintenance on the equipment late this evening. Watch this space for updates on this situation. -Eli, Broadlink’s Support/NOC staff