Month: July 2002

Sat Jul 20 21:54:50 PDT 2002 — Night Operations. Tonight we will be moving our NetApp filers, web, mail and support servers. We will begin by moving half of our load balanced web and mail servers. This pre-move of systems will not create a service outage. NetApps are scheduled to go off-line at 1am and should be down for no longer than 4 hours. During this time, mail and pop service will be off-line as well as locally hosted web sites. This operation will not affect connectivity to the Internet. -Matt, Nathan and Kelsey

Fri Jul 19 16:17:34 PDT 2002 — “R” Section Rooftop restored. The Rohnert Park “R” section Rooftop circuit had an outage earlier today. We opened a ticket with PacBell, and service was restored at approx. 3:15pm. PacBell reports that a backhoe “bumped” some of their equipment, which needed some TLC to come up again. -Scott and Steve

Fri Jul 19 13:23:21 PDT 2002 — “R” Section Rooftop down. The Rohnert Park “R” section Rooftop circuit is down. We have a ticket open with PacBell, and they are currently testing the circuit. -Scott, John, and Jen

Update: PacBell can’t see the Rohnert Park end of the circuit, so they have rolled a truck to see what’s going on. -Scott and Steve

Fri Jul 19 00:16:07 PDT 2002 — Over the next few hours we will be working on our primary administrative server. During this time member tools will be unavailable. -Kelsey and Nathan

Thu Jul 18 19:08:38 PDT 2002 — In the past 10 minutes, we’ve had a number of reports from DSL customers in Sebastopol that they are offline. We have opened trouble tickets with ASI, and will followup to work out what the situation is (likely down DSLAM) and estimated time to resolution is. -Dane, Zeke and Jared

Update: It appears that the DSLAM is back up and operational. -Kavan and Scott

Thu Jul 18 17:37:35 PDT 2002 — Problems on our 522-1003 dial group. In order to reduce downtime in our xxx-9811 Focal dial group we shuffled equipment out of our 522-1003 dial group so it could be relocated to San Francisco. In order for this migration to work properly, the gear had to be reconfigured on both ends. The new xxx-9811 gear was properly transitioned to its new configuration but the 522-1003 wasn’t properly configured. The MPIP server address wasn’t changed, which prevented MLPPP (Multilink PPP) from negotiating properly. As a result, all attempted MLPPP sessions to 522-1003 failed to negotiate multiple channels until this was resolved this afternoon. Overall, this only affected a small number of customers, primarily those with ISDN. -Steve

Thu Jul 18 04:07:17 PDT 2002 — Our news server migration went smoothly as planned. Total downtime for news.sonic.net was less than 45 minutes as we relocated the boxes to our new datacenter. -Kelsey and Nathan

Wed Jul 17 08:20:45 PDT 2002 — Brief busy signals on dial-up numbers ending with 9811. We discovered a misconfigured card in the middle of our dial group which caused the 9811 numbers to give false busy signals for about 5 minutes. The problem has been corrected and we will monitor this POP closely all day. -Matt and Steve

Wed Jul 17 18:32:48 PDT 2002 — Traffic to and from the Focal xxx-9811 dial up servers destined to some parts of the Internet was prevented from reaching its destination by one of our old edge routers, delta. A slight misconfiguration in a ACL used to prevent DOS attacks was catching some of the traffic sent by people using these numbers. This may have resulted in some users being unable to reach some destinations off of our network. Oddly enough, our other border router had the same incorrect ACL installed but due to what appears to be a Cisco IOS bug, it wasn’t affecting traffic. We will be consulting with TAC to aide in resolving the apparent bug in the other router. -Kelsey, Nathan and Chris B.

Wed Jul 17 17:00:26 PDT 2002 — Night Operations: Tonight at 1AM we will be moving our news servers to our new data center. We do not anticipate more than one hour of downtime as we migrate to the new location. During this time, we encourage our customers to use our alternate news service, supernews.sonic.net, which will not be affected by the migration. -Kelsey