Year: 2002

Fri Apr 19 23:25:55 PDT 2002 — The new SBC Pacific Bell phone book is likely to hit your porch shortly. Be sure to check out the new Sonic.net index tab advertisement in the Internet Guide section. It’s big, full color and card stock, and you can’t miss it. Tell a friend! -Dane

Fri Apr 19 23:23:31 PDT 2002 — C&W bandwidth upgrade from 15Mbps to 45Mbps (full T3). We’re planning to add 50% more bandwidth to our network connections on on either Monday or Tuesday morning. We’ll be making the change between six and eight AM, and during the change there is likely to be a brief slowdown as routes re-converge. -Ops staff

Wed Apr 17 14:08:56 PDT 2002 — Emergency maintenance on Broadlink wireless network. Broadlink will be performing maintenance on the backhaul serving the Rincon Valley tower site at 2:15pm today. This will interrupt service to Broadlink customers on the North East side of Santa Rosa. Maintenance is expected to take 10-15 minutes. -Broadlink and Sonic.net

Tue Apr 16 08:42:18 PDT 2002 — Even more issues with UUNet. Early this morning, their upstream router (the new one we migrated to last night) stopped ‘listening’ to our route announcements, making for some highly sub-optimal routing and poor performance. This has been repaired, and we’re watching these guys like hawks. – Eli, Matt

Mon Apr 15 22:56:07 PDT 2002 — UUNet migrated us to an alternate Cisco GSR 12000 in their SFO hub just now, and it took a little less than five minutes for the circuit to come up and all the BGP feeds to converge. We’re hoping this brings an end to this trouble, but we’ll be monitoring the circuit pro-actively tonight to confirm this.

208.201.224.115/smokeping/smokeping.cgi?target=NSPs.UUNet.UpstreamR

– Eli, Nathan, Matt.

Mon Apr 15 21:57:50 PDT 2002 — Yet more UUNet issues. The fix prescribed on the morning of the 12th mitigated the problem, but we’re still seeing problems, and at this time we’re observing heavy packet loss on the UUNet circuit. UUNet has agreed to re-home us on a different router port, and we have decided to allow this migration to happen during a period of heavy traffic because the impact on our network is rather significant in the present. The migration will certainly cause a 5-10 minute period of degraded connectivity, but it’s our assessment that we need to implement this now. – Eli, Nathan, Kelsey, Dane.

Thu Apr 11 16:28:37 PDT 2002 — More UUNet trouble. While last night’s reconfiguration went as planned, we still see inklings of the problem. Because of this, UUNet has decided to move our T3 to a different card in an adjacent router. The move has been scheduled for 2am on April 12th, and will result in about 5 minutes of degraded performance.

Update: After monitoring the circuit closely, we have determined that the fix was successful. We will continue to keep watch over this link to catch future problems quickly.

-Nathan

Tue Apr 9 20:09:27 PDT 2002 — ASI upgrading RT’s in 831 area code. We’ve determined that ASI has been upgrading remote terminals in the 831 area code, with the result that some folks are getting a sync-no-surf (green sync light, but no ability to access the Internet) condition. They said that the process would be completed by the morning. -John Kerr and Scott

Tue Apr 9 16:31:10 PDT 2002 — UUNet saga continues. The latency problems we’ve seen on the UUNet T3 should be solved, if all went as planned. We will be monitoring the circuit this evening closer than usual to quickly detect problems. One place to monitor this is through our public graph located at:

208.201.224.115/smokeping/smokeping.cgi?target=NSPs.UUNet.UpstreamR

Update: The problems continue, and UUNet has determined another fix. Unfortunately, the necessary changes require them to take down our T3 for approximately 2 minutes. This event is scheduled for April 11th at 3:00am. While we will still have Internet connectivity via our Cable and Wireless T3, access may be slow during this period.

-Nathan and Scott

Tue Apr 9 16:22:01 PDT 2002 — Add-on mailbox upgrade. On Sunday at 2pm we moved add-on mailboxes to our load-balanced redundant cluster. This did not go as smoothly as we expected, but all mailboxes were operating correctly as of 5pm on Sunday.

One problem we still have: we have reports of some Twig users losing stored email on the server. We are still trying to track down what could have happened to cause this, and hopefully restore the email. -Scott, Russ, and Nathan