Month: August 2001

Emergency maintenance on NetApp filer.

Fri Aug 31 18:18:07 PDT 2001 — Emergency maintenance on NetApp filer. One of the loop resiliency circuit cards in one of our NetApp filers has failed. While this is redundant equipment and didn’t cause any immediate impact, our redundancy is compromised, so we consider this a very high priority repair.

Saturday morning at 1:00AM, we will be shutting down freezer.sonic.net, our NFS filer hosting all storage for email, shell and ftp. This means that these services will not be available for about five to ten minutes. Normal maintenance of these filers does not require any downtime, and a second CPU/head allows for full redundancy in a failure or normal maintenance, however, in this case, the LRC card is what supports the redundant filer CPU/head. -Kelsey, Nathan and Dane

Sonic.net has deployed a new network time…

Fri Aug 31 17:14:54 PDT 2001 — Sonic.net has deployed a new network time protocol (NTP) server for customer use. You may synchronize your system clocks with the NTP protocol to time.sonic.net, aka ntp.sonic.net. “time” is a stratum 2 atomic synchronized time server for use by Sonic.net customers.

Client software is available for a variety of platforms. For Windows users, we recommend AboutTime, available at the following URL:

www.arachnoid.com/abouttime/index.html

For additional NTP resources, see www.ntp.org/.

-Eli and Dane

One of our Santa Rosa pops was returning a…

Fri Aug 31 13:54:24 PDT 2001 — One of our Santa Rosa pops was returning a ‘All Circuits are Busy’ error today, we narrowed it down to a PRI card that was acting up and rebooted it. -Steve

Update: Rebooting the PRI card did not clear up the problem as I had thought it would. I have replaced the card and tested it, and all looks well now. -Steve

Update #2: The All Circuits are Busy message came back. We opened a ticket with PacBell and also did a hard reboot on NAS21, which is the front of our 1003 dialup pool. This caused some people to be disconnected. The hunt group appears to be taking calls just fine now I will be watching it for the rest of the day. -Steve

Update on BroadLink tower outage.

Thu Aug 30 11:39:32 PDT 2001 — Update on BroadLink tower outage. Estimated time of repair is 3:00pm today. 100 BroadLink/Sonic.net customers are affected by this outage in the Southern Santa Rosa area.

On Monday morning, the licensed frequency 23 Ghz equipment serving the backhaul from the South SR tower failed. It was replaced with spare gear, which worked for a few hours, failed again Monday afternoon, then worked through Tuesday. It failed completely on Wednesday morning around midnight.

The vendor for this equipment is out of business, and BroadLink was unable to obtain technical assistance or any additional equipment beyond the spare that failed.

On Wednesday, BroadLink worked to bring up a 2.4 Ghz link (802.11b) to backhaul traffic from this tower site, but at around a 1.5 Mbps speed, this link was completely unable to handle the load requirements.

This AM, BroadLink has taken equipment out of service that was serving a link from their Dutton main office, and is moving it to downtown Santa Rosa to serve the backhaul for the South tower site. This 5 Ghz link has a max speed of 8 Mbps in this configuration, and will be adequate to support the traffic from the tower.

Technicians are currently on site in downtown Santa Rosa and at the tower site, and hope to have the equipment fully online by 3pm.

BroadLink asked me to pass along their apologies for the extended outage to the 100 affected customers. They’ve had an unforeseen situation where spare equipment failed after a brief time in service, and the vendor is no longer in business. (Co-incidence? Hmm..) The long term goal is to deploy backup connectivity to each tower site in this market to assure reliability in case of a link failure. -Dane, and from BroadLink, Chris, Shane, Randy and Tim.

Update: BroadLink has repaired their backhaul link, and has in fact made the network more efficient by removing one hop from the connection.

BroadLink is having problems with the link to

Wed Aug 29 09:05:32 PDT 2001 — BroadLink is having problems with the link to their South Santa Rosa tower site, and customers served by that tower currently are offline. BroadLink reports that they have a technician on site, and are working to troubleshoot the problem now. -Dane

Update, Wed Aug 29 17:06:43 PDT 2001 BroadLink’s licensed RF backhaul has failed, and they’re working to bring a replacement on line. No ETR is available at this time.

Update, Wed Aug 29 18:00:00 PDT 2001 A replacement link is online, but performance may be slower than usual. We will provide additional updated information here on Thursday morning.

Sonic.net is now offering free additional IP…

Wed Aug 29 17:06:01 PDT 2001 — Sonic.net is now offering free additional IP addresses for PacBell DSL connected customers. For customers with multiple PCs, this allows for sharing of the DSL link between up to four computers simultaneously. Previously priced at $12 per month, Sonic.net is now including a block of four IP addresses for all PacBell DSL customers at no charge. Eight IP addresses are available for $12 per month. (Existing customers who have four or eight IP blocks will see pricing moved down to $0 or $12, respectively on their next invoice)

Using an Ethernet hub, this allows customers to share their DSL connection with multiple PCs in the home or office. No additional equipment is needed, and setup is simple.

Sonic.net is committed to continuing to deploy additional features and services on both our broadband and dialup services. We hope that this new feature adds value to our DSL offering.

You can migrate from the single IP initially assigned to a new block of four or eight IP addresses using our online member tools, linked from the home page at www.sonic.net/. This is a new standard ongoing service offering, so please, to conserve IP address space, only migrate if and when you have multiple machines that you need to address. For any questions or discussion about this new feature, please see news:sonic.dsl

Note that BroadLink wireless service uses equipment which already provides access for multiple PCs, so this change does not affect BroadLink customers.

-Kelsey, Nathan, Dustin and Dane

BroadLink has had problems with the backhaul…

Mon Aug 27 15:10:06 PDT 2001 — BroadLink has had problems with the backhaul link that serves Southeastern Santa Rosa. The link failed this morning between 10:23a and 10:57a, and a replacement link was brought up. This afternoon @ 1:30p, a configuration error in the backup link caused intermittent outages for some customers between 1:30p and 3:00p. Broadlink’s engineers are currently monitoring the link to prevent further issues, and may well schedule some maintenance on the equipment late this evening. Watch this space for updates on this situation. -Eli, Broadlink’s Support/NOC staff

We will be performing maintenance on the…

Fri Aug 24 11:34:05 PDT 2001 — We will be performing maintenance on the focal dialup gear on Thursday morning at 12:00am to 2:00am. This will cause random disconnects on any dialup line that ends with 9811. I would like to point out that most areas have redundant dialup numbers and you can find a backup number to use during this maintenance period by visiting our pop finder tool at www.sonic.net/cgi-bin/pops.pl Thanks for your patience. -Steve

Move of the Sebastopol POP and dialups.

Fri Aug 24 09:47:52 PDT 2001 — Move of the Sebastopol POP and dialups. On Wednesday morning at 6am, we are planning to relocate our equipment serving the West county. Expected downtime is about one hour. Note that in addition to this PacBell POP, we serve the same area with our PacWest and also Focal dialup numbers, so full redundancy is available during this planned downtime. -Dane

Update: Because of problems with the backhaul links from Santa Rosa to Sebastopol, this move is being rescheduled.