Month: April 2002

Tue Apr 16 08:42:18 PDT 2002 — Even more issues with UUNet. Early this morning, their upstream router (the new one we migrated to last night) stopped ‘listening’ to our route announcements, making for some highly sub-optimal routing and poor performance. This has been repaired, and we’re watching these guys like hawks. – Eli, Matt

Mon Apr 15 22:56:07 PDT 2002 — UUNet migrated us to an alternate Cisco GSR 12000 in their SFO hub just now, and it took a little less than five minutes for the circuit to come up and all the BGP feeds to converge. We’re hoping this brings an end to this trouble, but we’ll be monitoring the circuit pro-actively tonight to confirm this.

208.201.224.115/smokeping/smokeping.cgi?target=NSPs.UUNet.UpstreamR

– Eli, Nathan, Matt.

Mon Apr 15 21:57:50 PDT 2002 — Yet more UUNet issues. The fix prescribed on the morning of the 12th mitigated the problem, but we’re still seeing problems, and at this time we’re observing heavy packet loss on the UUNet circuit. UUNet has agreed to re-home us on a different router port, and we have decided to allow this migration to happen during a period of heavy traffic because the impact on our network is rather significant in the present. The migration will certainly cause a 5-10 minute period of degraded connectivity, but it’s our assessment that we need to implement this now. – Eli, Nathan, Kelsey, Dane.

Thu Apr 11 16:28:37 PDT 2002 — More UUNet trouble. While last night’s reconfiguration went as planned, we still see inklings of the problem. Because of this, UUNet has decided to move our T3 to a different card in an adjacent router. The move has been scheduled for 2am on April 12th, and will result in about 5 minutes of degraded performance.

Update: After monitoring the circuit closely, we have determined that the fix was successful. We will continue to keep watch over this link to catch future problems quickly.

-Nathan

Tue Apr 9 20:09:27 PDT 2002 — ASI upgrading RT’s in 831 area code. We’ve determined that ASI has been upgrading remote terminals in the 831 area code, with the result that some folks are getting a sync-no-surf (green sync light, but no ability to access the Internet) condition. They said that the process would be completed by the morning. -John Kerr and Scott

Tue Apr 9 16:31:10 PDT 2002 — UUNet saga continues. The latency problems we’ve seen on the UUNet T3 should be solved, if all went as planned. We will be monitoring the circuit this evening closer than usual to quickly detect problems. One place to monitor this is through our public graph located at:

208.201.224.115/smokeping/smokeping.cgi?target=NSPs.UUNet.UpstreamR

Update: The problems continue, and UUNet has determined another fix. Unfortunately, the necessary changes require them to take down our T3 for approximately 2 minutes. This event is scheduled for April 11th at 3:00am. While we will still have Internet connectivity via our Cable and Wireless T3, access may be slow during this period.

-Nathan and Scott

Tue Apr 9 16:22:01 PDT 2002 — Add-on mailbox upgrade. On Sunday at 2pm we moved add-on mailboxes to our load-balanced redundant cluster. This did not go as smoothly as we expected, but all mailboxes were operating correctly as of 5pm on Sunday.

One problem we still have: we have reports of some Twig users losing stored email on the server. We are still trying to track down what could have happened to cause this, and hopefully restore the email. -Scott, Russ, and Nathan

Fri Apr 5 20:36:08 PST 2002 — Ongoing UUNet Connectivity Issues. Our UUNet T3 is currently experiencing intermittent latency and packet loss. UUNet has tracked the issue to the buffering system on the line card that we connect to. They have determined a course of action that should fix the problem and expect to put it in place tomorrow morning. The effects of this problem can be seen at:

208.201.224.115/smokeping/smokeping.cgi?target=NSPs.UUNet.UpstreamR

-Nathan

Thu Apr 4 10:59:15 PST 2002 — The sms which terminates Pac Bell DSL and Broadlink WDSL mysteriously rebooted this morning at about 10:45am. These customers experienced a 5 minute interruption in service while the diagnostic information was collected and the router rebooted. We are not yet sure if this is related to Sundays issues but will be working with Redback today to determine the cause. -Matt, Russ, Steve and Eli

Thu Apr 4 16:57:18 PST 2002 — Add-on mailbox upgrade. On Sunday, April 7th, at 2pm, we will be moving add-on mailboxes from the current host to our load-balanced redundant cluster. This change will be transparent to folks with mailboxes, with the exception of about 30 minutes of downtime.

One advantage of this move will be the ability of mailbox accounts to tag spam using Spamassassin, our flagship anti-spam tool. -Russ and Scott