Author: admin

Fri May 25 16:23:45 PDT 2001 — Bolt upgrade. Our shell server, Bolt.sonic.net, is running a new kernel. We have also upgraded several servers on our network with the same kernel. Ping, traceroute, ssh, mtr, and screen are all back in service on Bolt. More details regarding the importance of this upgrade will be posted to the MOTD within the next week or so. -Scott, Dane, Dustin, Steve

Fri May 25 15:24:20 PDT 2001 — Pacific Bell’s statewide ATM network is still having some very significant problems. Our own statistics show that about 25% of our PacBell connected DSL customers are currently offline. Here’s the statement from PacBell, including the timeline for repairs.

We’ve experienced a hardware failure within select Lucent ATM switches. Your ATM service may be affected during the restoration process. Service will be brought down shortly after 12:01 a.m. PDT Saturday, May 26. All service should be restored by approximately 6:00 a.m. PDT Saturday, May 26. As this is a progressive restoration process, expected down time may be up to 8 hours. Service restoration to ISPs will receive very high priority, behind emergency services and banks. Every effort will be made to minimize down time.

In the event an ATM circuit serving a DSL application is affected, DSL end users may need to re-boot their equipment upon restoration.

We were told by a PacBell rep that somewhere around a million DSL customers in California have been affected by this huge PacBell outage. Sonic.net will work to assure that PacBell resolves the issues affecting our customers as quickly as possible.

Note that Sonic.net and BroadLink have build a backup link to resolve this issue for BroadLink connected customers, freeing them from PacBell’s currently broken network. Note also that all DSL accounts include dialup access for backup purposes, so you may dialup in situations like this one. See our support page for dialup numbers and setup. -David and Dane

Fri May 25 13:23:26 PDT 2001 — Pacific Bell’s ATM network problems caused all BroadLink customers to go offline. While Pacific Bell works to repair their issues, we’ve reconfigured for a physical cross connection from BroadLink’s cabinet colocated in our data center over to a spare interface on our RedBack SMS 1800.

Pacific Bell reports that many customers are being affected state-wide, and it was great to have a quick contingency plan in place in case of trouble. As Anne Robinson of the BBC would say, “Pacific Bell, you are the weakest link. Good bye.”

We will schedule some brief downtime in the future to migrate back onto Pacific Bell’s network once it’s stable. -Eli, Scott, Dane and Shane R. (BroadLink)

Thu May 24 22:33:36 PDT 2001 — We’ve made some software changes on the shell server which have broken a few utilities temporarily. The items which are currently unavailable include ping, traceroute, ssh and screen.

Sorry for the inconvenience, we will work to get these back online shortly. Please use your own workstation for these tools in the mean time.

In other shell server news, we found that a cable was causing some network errors, causing slow performance, particularly with NFS recently. After some investigation, the cable has been replaced, and we’re back up to full speed. -Dane, Scott and Nathan (the cool cable sleuth)

Thu May 24 20:55:43 PDT 2001 — We will be performing maintenance on the 1003 dialup group this evening starting at 11:30pm. Some users who are connected to the 1003 dialup number may be disconnected. The window for this work is between 11:30pm and 12:30am. This maintenance includes moving a number of T1 PRI circuits from copper to our fiber facilities.

Update: All completed. -Steve and Kevan

Thu May 24 17:03:30 PDT 2001 — Due to a cable connector problem, we’ve just suffered about 15 minutes of intermittent performance. The connection between our core switch and one of our two redundant core routers was severed temporarily, apparently due to problems with a cable connector latch in the patch bay. In theory, the second connection to the second core router should have taken on all of the traffic, and it’s redundant connection to the affected router would have carried the traffic. This deployment is still in progress, and hasn’t been fully tested.

We’re sorry about the poor performance, and we’ll take steps to wrap up the testing of the redundancy to assure that a minor failure like this can’t impact things badly. -Dane, Nathan, Eli, Steve and Kevan

Wed May 23 02:09:26 PDT 2001 — Since Friday afternoon ape, the core switch, had been intermittently exhibiting some strange behavior delaying ICMP traffic (pings and traceroutes) on some ports while still running at wire speed for other IP traffic. In some cases the delay introduced into ICMP traffic could be as much as 2 seconds. Because of the problem did not appear to be affecting normal traffic, we left the switch in this state in order to facilitate the troubleshooting and debbuging process with Extreme Networks. Extreme Networks finished the information gathering process tonight and we went ahead and rebooted the switch at 1:25AM to clear the problem. The switch booted cleanly and appears to be fully functional. We are working closely with Extreme Networks to resolve the ongoing problems that we are having with their equipment. -Kelsey

Tue May 22 12:01:29 PDT 2001 — Some non-intrusive testing of our 1003 dialup gear turned out to be more intrusive then we expected. This caused the 1003 dialup group to return either a fast busy or a ‘All circuits are busy’ message. In order to clear up this issue we needed to reset some modems. This caused a few people to be disconnected. Sorry for any problems this may have caused. -Steve

Wed May 16 15:19:40 PDT 2001 — Sonic.net is now offering Sun Netra X1’s on our leased colocation plan! For as low as $195 a month, you can colocate a Sun server with a 400 MHz UltraSPARC processor and from 128-512 Megabytes of RAM. For more details, see our sales page at www.sonic.net/sales/sun/

-Kevan

Wed May 16 14:16:13 PDT 2001 — We completed the upgrades of our local DNS servers yesterday afternoon. We now have three new dedicated DNS servers on our local network and are currently securing a facility for a fourth off-site server on the East Coast to increase the resiliency of our DNS services, especially in the event of network outages here at our NOC — or the West Coast in general.

The new DNS servers are considerably more responsive than the two old servers and have significantly more robust hardware beneath them. You may notice faster DNS resolution times and more responsive web browsing. We’ve published a brief white paper on the new DNS servers at www.sonic.net/network/ along with a link to our public network statistics. If you have any questions we’ll be happy to respond to them in news:sonic.net

-Kelsey