Author: admin

Tue Feb 20 11:06:47 PST 2001 — Congestion avoidance. A colo site had incorrect permissions on some of their directories, which prompted scurrilous ruffians to upload DIVX files for illicit distribution. The resulting demand caused saturation on our T3’s, with as much as 3% packet loss on our Internet circuits. While this does not seriously interfere with ordinary use of TCP/IP, it does cause TCP to enter “congestion avoidance”, which means TCP streams slow down to prevent packet loss. We have corrected the situation; the saturation has stopped, and traffic is now moving normally. Please visit news:sonic.net for questions about congestion avoidance. -Scott, Matt, Rob, and Mike

Tue Feb 20 13:39:26 PST 2001 — Emergency admin server downtime. We had to reboot our core administrative server. This could have resulted in slow DNS response while the server was down but should not have affected other normal Internet usage. Membertools were also unavailable while the host was down. -Scott & Kelsey

Fri Feb 16 02:27:58 PST 2001 — Night Operations: We completed all of the security updates and network maintenance. No significant downtime was incurred. -Kelsey, Scott, Chuck and Matt.

Thu Feb 15 16:35:06 PST 2001 — Night Operations: Tonight we are performing a series of security updates throughout our network. We will be rebooting a number of our servers to bring them up on new Linux kernels and doing ACL work on our border routers to help mitigate the impact of a DOS on our network. We do not expect any significant down time on any of our services and, due to our load balancing, major services like email and web should be unaffected.

We will also be finishing the upgrades to our core switch, Ape, and we may need to reboot it when we are finished. If we do need to restart it, there will be a 90 sec complete network outage as it restarts.

-Kelsey, Scott, Chuck and Matt.

Wed Feb 14 10:35:13 PST 2001 — Update on mail servers. We have isolated the source of the attack and it appears that it is a mailbomb attack targeted at specific users. We have escalated the attack to the remote upstream providers and have successfully blocked the attack. -Scott, Steve, Kelsey, and Russ

Wed Feb 14 10:22:39 PST 2001 — Attack on mail servers. Our entire mail server array has been impacted by what appears to be either a) a rude or misconfigured mail host, or b) a denial of service attack. We decided to treat the situation like an attack, isolate the source, block the attack, and then harden our servers to prevent future attacks of this nature. We have completed all but the latter step, which pends further analysis. -Scott, Steve, Kelsey, and Russ

Wed Feb 14 13:24:53 PST 2001 — Though there has been no official word from Pacific Bell, DSLAM 1 in Santa Rosa is now consistently showing good performance with no problems. – Eli

Tue Feb 13 11:00:54 PST 2001 — Update on DSLAM 1. It appears that PacBell resolved the issues with DSLAM 1 here in Santa Rosa. If you still are experiencing difficulties with your DSL please let us know. -Dave

Tue Feb 13 15:24:23 PST 2001 — Update on DSLAM 1. DSLAM 1 is now completely offline. We do not have an ETR but will post one as soon as we hear from PacBell. This only affects some DSL customers in Santa Rosa. -Dave

Mon Feb 12 11:20:52 PST 2001 — Update On DSLAM 1. PacBell confirms that they are having trouble with one of the DSLAMs here in Santa Rosa. They have not given us an ETA yet on it’s repair. Please don’t forget that if your DSL is down you may still dial-in using a modem. -Kelsey & Dave Mon Feb 12 20:43:55 PST 2001 – Update: we’re seeing much lower packet loss now, but the trouble is ongoing. -Dane