Month: July 2000

Mon Jul 31 23:48:34 PDT 2000 — CW’s NOC is still troubleshooting the packet loss and latency within their network. To facilitate that, I’ve shut down the UUNet circuit. The loss appears to be on one link — an OC3 — which is connected to their Palo Alto Internet Exchange router. There, CW peers with several ISP’s, including Verio. Connectivity to CW’s PAIX peers, will be intermittent and sporadic as we track down this problem. -Scott

Mon Jul 31 20:26:37 PDT 2000 — CW security has confirmed that the packet loss and latency within their network is not a DoS attack. CW’s NOC is troubleshooting. -Scott

Mon Jul 31 19:27:08 PDT 2000 — UUNet has solved the packet loss and latency trouble within their network. They did this by rebooting the OC12 interface on their Cisco 7500 (which doesn’t let know what the problem was, but at least it’s fixed). I just spoke with CW’s NOC, and they are investigating the problem in their network from both a technical and security perspective. Updates as they come in… -Scott

Mon Jul 31 18:55:11 PDT 2000 — More net latency and packet loss. We also found a UUNet link that exhibits high latency and packet loss. Nathan was able to correlate the beginning of the high latency on this link with Sunday morning’s DoS attack. It is possible (though we don’t know yet) that this is an ongoing DoS attack that is being filtered at both CW and UUNet’s border routers. (The traffic is not making it to Sonic.net.) However, we just got off the phone with UUNet security, who says that the DoS would have to be filling an OC12 (the equivalent of 12 T3’s) to be causing this problem — unlikely, but they are checking that out.

Another correlation: the packet loss and latency are manifesting themselves on routers that are directly connected to exchange points, even though the traffic doesn’t necessarily traverse the exchange points. UUNet is also investigating this on their end now. We are still waiting for a call from CW. -Scott, Nathan, Kelsey, Steve

Mon Jul 31 17:01:11 PDT 2000 — Net slowness through the Palo Alto Internet Exchange. We are seeing intermittent high latency and packet loss for some sites we reach across Cable and Wireless. We characterized the problem (a saturated link on CW’s network), and CW is working on it. -Scott

Sun Jul 30 03:37:00 PDT 2000 — Denial of Service attack. Sonic.net suffered a Denial of Service attack this morning that consumed all inbound bandwidth of one of our network providers. The network provider’s security force _almost_ had the attack traced to the source, when the attack stopped. Both they and we are monitoring the situation. -Scott

Mon Jul 24 11:34:29 PDT 2000 — Nas22.sonic.net rebooted this morning at around 11:30am this would have caused some people who where dialed into the 707-522-10003 dialup group to be disconnected. I am currently investigating as to why this happened. -Steve

Sun Jul 23 12:03:05 PDT 2000 — The remote access equipment that serves the 522-1002 dial-up group blew a circuit breaker a half and hour ago as the load grew this morning. Eli made it down to our NOC to redistribute the equipment across a second circuit and has restored all access to the 522-1002 number. PostOffice, our web based email client, was broken in the ssl server upgrade. Nathan and Scott are working on it now and should have both PostOffice and its replacement, Twig, working shortly. -Kelsey

Sun Jul 23 03:54:10 PDT 2000 — Night operations are complete, and went well. Kelsey and Scooter upgraded the news server to 1 gig of RAM, and it has been moved into a new cabinet location. They also moved mistral, our third web server, into a cabinet and rearranged power for our core LAN rack so that we now have diverse dual-UPS power for the load balancing switches. Steve moved the older analog dial group to new power outlets from our second huge APC Symmetra UPS. Nathan and Scott worked to upgrade our SSL server with a new OS on a new disk. The BroadLink guys moved their DSLAM into their dedicated rack here in the data-center. Then we had doughnuts, but the BroadLink folks went missing before eating theirs, and we’re not quite sure why. -Dane (and a bunch of sleepy ops)

Thu Jul 20 13:59:16 PDT 2000 — This coming Sunday morning at 12:30am we will be performing maintenance and upgrades on a number of facilities here.

The older analog 522-1001 dialup group will be moving to our new power plant, and any connected customers will be disconnected. It’ll be down for just a few minutes as we move power cables.

The news server will be moved to a new location in the data center, and we’ll be putting a full gig of RAM in it. In addition, we’re hoping to get a new disk array online for faster performance.

Broadlink will be doing DSLAM maintenance and upgrades, and wireless DSL customers will be offline for a little while beginning at 1am.

We’ll be working on the configuration in our SMS DSL ATM router, and all DSL customers will be offline briefly while we reset the equipment with the new configuration.

SSL hosted customers on ssl.sonic.net will be down briefly while that system is upgraded with new hardware. The new deployment has been in test at ssl2.sonic.net, and all customers have been notified to test their CGI configurations.

We hope to be finished in three to four hours with all of these tasks. -Dane, Scott, Eli, Steve, Kelsey, Nathan and Scooter