Author: admin

Tue Aug 13 18:18:35 PDT 2002 — Intermittent mail delays. We’ve had difficulties with intermittent delay of local email, resulting in suboptimal queue sizes. This has become quite noticeable the past few days, especially today. This was a main agenda topic at today’s operations meeting, where we developed a multi-pronged plan of attack to get the situation under control. Current queue sizes for mail servers in our pool are low, and we have confirmed that the majority of these messages are store-and-forward to networks that have not yet collected. We will post updates as we work further to improve email, our most important application. -Scott, Dane, Kelsey, Russ, Matt

Mon Aug 12 01:41:32 PDT 2002 — The circuit that is used for our Covad DSL uplink has been unstable since 1:17AM. It’s gone through over 2000 carrier transitions in the past half hour! We our en route to the data center to investigate the cause the flapping. In the meantime we have administratively shutdown the interface to prevent routing instability in our network. We hope to have service restored to our Covad DSL customers shortly. -Kelsey

Update: The circuit appears to have stabilized; all service has been restored. We’ll keep a close eye on the circuit for the next few hours.

Thu Aug 8 13:45:35 PDT 2002 — CW outage. At about 1:30pm CW dropped our T3 to them. It is now back up again, and engineers are monitoring it for further signs of trouble. -Scott and Nathan

Thu Aug 8 13:34:35 PDT 2002 — CW does it AGAIN. Cable and Wireless has once again killed one of our T3’s. We have escalated the ticket and are working with their NOC to get the circuit up as soon as possible. Meanwhile, our UUNet T3 is fine, but is groaning under the load. -Scott

Wed Aug 7 09:29:29 PDT 2002 — This morning our authentication servers experienced problems with authenticating users. After spending some time trouble shooting the issue we were able to get it back on line. This caused dialup customers to get an invalid password message when trying to log in. We apologize for the inconvenience this may have caused. – Ops Team

Mon Aug 5 10:37:06 PDT 2002 — At approximately 8:30AM we lost our BGP session with our Cable and Wireless upstream router. As a result all traffic flowing to the Internet was routing over our UUNet circuit causing high latency. We worked with Cable and Wireless’ NOC to get the circuit back up as fast as possible. The ckt is now up and passing traffic. -Scott, Kelsey, and Nathan

Mon Aug 5 09:05:15 PDT 2002 — At approximately 8:30AM this morning we lost out BGP session with our Cable and Wireless upstream router. As a result all traffic flowing to the Internet is routing over our UU.net circuit which is flooded under the load. We are currently in working with Cable and Wireless’ NOC to get our BGP session back on line as soon as possible. -Kelsey and Nathan.

Update: Cable and Wireless has an engineer working on the situation now. -Scott

Mon Aug 5 04:17:00 PDT 2002 — Night Operations Complete: We have completed the redundant L2 core reconfiguration of the SMS and 1800 and our 7507 customer router. Extensive testing shows that the redundant L2 networks are functioning as expected.

We also overhauled gale, our Diablo-based NNTP feeder server with some additional disks for it’s spools and a third ethernet card. The changes have made a remarkable difference in the quality of our inbound NNTP feeds. If the current trends hold, we expect to process more than 300GB today, which is significantly higher than we’ve been able to handle in the past. Gale’s increased performance should result in a higher multi-part completion rate on our news server.

Zeke, Kevan and Tony relocated all of our leased Sun Cobalt colocations. They are now up and operating normally.

-Kelsey, Nathan, Zeke, Kevan and Tony

Sat Aug 3 02:49:11 PDT 2002 — We have just finished moving our network monitoring and notification servers to our new facility. This leaves only a few odds and ends at our old facility which we expect to decommission by the end of this month once all customer colocations have been moved.

Earlier today we brought up our first three peers on the public switch at our colocation in Equinix’s San Jose IX. A direct peer to Yahoo! should be online by Monday morning. Peering at Equinix decreases utilization on our two T3 transit links while also providing decreased latency and improved throughput.

Monday morning as 12:01 AM we will be migrating Mega, one of our 7507 customer routers, as well as our SMS 1800 which terminates all PacBell DSL to our redundant meshed L2 core. We do not anticipate any loss of service while we bring up the second links to these two customer routers. -Kelsey, Nathan and Zeke

Fri Aug 2 18:21:58 PDT 2002 — Leased Cobalt customer move. Monday morning at 12:01 AM we will be moving our leased Cobalt customers to the new datacenter. This will not affect any of our other services or customers. Downtime for the leased servers is expected to be less than two hours. -Zeke, Nathan, Kevan and Jared