Month: May 2001

Wed May 30 15:31:08 PDT 2001 — We’ve had a security incident affecting bolt.sonic.net, our unix shell server. No other servers or services appear to be affected. We’re aware of 126 remote user account logins and passwords were captured during the break-in, but all recent unix shell users should review any remote hosts they use for signs of abuse.

The shell server is our most vulnerable server due to the nature of interactive shell services. With this in mind, it’s been designed to limit the scope of potential damage due to an intrusion. It has no passwords on board; authentication is done remotely via RADIUS. The shell server also has no customer records of any sort. The shell server is a non-trusted component of the network, and lives on it’s own private network segment to prevent network sniffing. Additionally, NFS filesystems are mounted ‘squashed’, so that there are zero privileges on those filesystems housing end-user files.

The user names and passwords which were captured were from a trojaned version of the ‘ssh’ client binary. If you’ve used ssh from bolt toward a remote system in the past few weeks, your username and password on the remote system you connected to may be compromised. Notification emails are being sent to the affected remote accounts, plus the security address at the remote site. Telephone notification has already been made to any remote sites for which a ‘root’ or ‘admin’ password was captured.

Connecting to remote systems via a public shell server is a “Bad Idea” from a security perspective. Please use caution and common sense – if you need to establish a remote session, it should be done directly from your workstation to the remote host.

We’re sorry about any inconvenience this has caused the affected unix shell users. We’re committed to continuing to provide shell services, and we will work to assure that bolt.sonic.net remains secure on an ongoing basis. Please post to news:sonic.os.unix if you have any additional questions.

-Dane, Scott, Eli, Nathan, Kelsey, Steve and Russ

Wed May 30 13:33:13 PDT 2001 — We have put up new web-enabled versions of ping, traceroute and mtr at stats.sonic.net/cgi-bin/icmp.cgi We’ve also replaced the respective binaries on bolt with scripts that gateway your queries to the web enabled tools to provide basic icmp troubleshooting from the shell server. Note that ‘screen’ is still unavailable for security reasons. -Kelsey and Dane

Tue May 29 12:06:25 PDT 2001 — We just had to reboot one of the three USR TC hubs that handles the 522-1003 dial-up group. One of it’s line cards had been generating repeated errors interfering with the ability for users to successfully connect. Reseting the single card did not resolve the problem, nor did taking the card out of service. The only other choice we had was to reboot it. It appears to be up and taking calls just fine now. The box had been exhibiting this failure for about 10 minutes before we had it services restored. -Eli, Russ and Chris.

Fri May 25 11:40:58 PDT 2001 — The ATM network appears to have returned to normal. Customers started coming back online gradually between 10:15 and 11:30, and we have no outstanding problems at this time. – Support

Update – new trouble has affected the PVCs serving BroadLink’s customers.

Fri May 25 09:50:40 PDT 2001 — Pacific Bell’s ATM network is experiencing serious problems right now, specifically in the LATA that covers North Western California. This affects Pacific Bell ADSL customers, and FRATM T1 customers. Pacific Bell has not given us an estimated time of repair, but they are working as quickly as possible to correct this massive outage. – The Sonic OPS/Support Team

Fri May 25 16:23:45 PDT 2001 — Bolt upgrade. Our shell server, Bolt.sonic.net, is running a new kernel. We have also upgraded several servers on our network with the same kernel. Ping, traceroute, ssh, mtr, and screen are all back in service on Bolt. More details regarding the importance of this upgrade will be posted to the MOTD within the next week or so. -Scott, Dane, Dustin, Steve

Fri May 25 15:24:20 PDT 2001 — Pacific Bell’s statewide ATM network is still having some very significant problems. Our own statistics show that about 25% of our PacBell connected DSL customers are currently offline. Here’s the statement from PacBell, including the timeline for repairs.

We’ve experienced a hardware failure within select Lucent ATM switches. Your ATM service may be affected during the restoration process. Service will be brought down shortly after 12:01 a.m. PDT Saturday, May 26. All service should be restored by approximately 6:00 a.m. PDT Saturday, May 26. As this is a progressive restoration process, expected down time may be up to 8 hours. Service restoration to ISPs will receive very high priority, behind emergency services and banks. Every effort will be made to minimize down time.

In the event an ATM circuit serving a DSL application is affected, DSL end users may need to re-boot their equipment upon restoration.

We were told by a PacBell rep that somewhere around a million DSL customers in California have been affected by this huge PacBell outage. Sonic.net will work to assure that PacBell resolves the issues affecting our customers as quickly as possible.

Note that Sonic.net and BroadLink have build a backup link to resolve this issue for BroadLink connected customers, freeing them from PacBell’s currently broken network. Note also that all DSL accounts include dialup access for backup purposes, so you may dialup in situations like this one. See our support page for dialup numbers and setup. -David and Dane

Fri May 25 13:23:26 PDT 2001 — Pacific Bell’s ATM network problems caused all BroadLink customers to go offline. While Pacific Bell works to repair their issues, we’ve reconfigured for a physical cross connection from BroadLink’s cabinet colocated in our data center over to a spare interface on our RedBack SMS 1800.

Pacific Bell reports that many customers are being affected state-wide, and it was great to have a quick contingency plan in place in case of trouble. As Anne Robinson of the BBC would say, “Pacific Bell, you are the weakest link. Good bye.”

We will schedule some brief downtime in the future to migrate back onto Pacific Bell’s network once it’s stable. -Eli, Scott, Dane and Shane R. (BroadLink)

Thu May 24 22:33:36 PDT 2001 — We’ve made some software changes on the shell server which have broken a few utilities temporarily. The items which are currently unavailable include ping, traceroute, ssh and screen.

Sorry for the inconvenience, we will work to get these back online shortly. Please use your own workstation for these tools in the mean time.

In other shell server news, we found that a cable was causing some network errors, causing slow performance, particularly with NFS recently. After some investigation, the cable has been replaced, and we’re back up to full speed. -Dane, Scott and Nathan (the cool cable sleuth)

Thu May 24 20:55:43 PDT 2001 — We will be performing maintenance on the 1003 dialup group this evening starting at 11:30pm. Some users who are connected to the 1003 dialup number may be disconnected. The window for this work is between 11:30pm and 12:30am. This maintenance includes moving a number of T1 PRI circuits from copper to our fiber facilities.

Update: All completed. -Steve and Kevan