Wed Feb 15 17:32:58 PST 2006 — DSL Outage. One of our DSL concentrators in our San Francisco POP went down for 10-15 minutes this evening. We are working with the vendor on analyzing the software core dump so that we can prevent such failures in the future. All affected customers should be back online at this time. – Nathan and Jared

Fri Feb 10 15:14:18 PST 2006 — Planned Dialup Maintenance. On Tuesday 2/14 at approximately 10AM we will be migrating dialup connectivity served from our 200 Paul Ave. POP in San Francisco to a new Cisco Access Server architecture. This upgrade is expected to enhance performance while reducing the number of disconnects. This will cause a brief service interruption for customers dialing 707-823-8812 as well as any of our dialup numbers ending in 5. Typically we would perform such maintenance during off-peak hours, however we feel it is important to perform this maintenance when our Technical Support and Operations staff are available. As always, alternate dialup numbers may be obtained via our Pop Finder tool located at www.sonic.net/popf/ – Russ and Network Operations

Tue Feb 7 21:39:19 PST 2006 — DSL Outage in Santa Rosa. The DSL terminator in Santa Rosa just rebooted. Most customers should already have service restored by the time this MOTD is finished. We will investigate the cause of the failure and take the required steps to resolve it. -Nathan, Jared and Kelsey

Thu Feb 2 23:22:18 PST 2006 — DSL Outage in Santa Rosa. Tonight at approximately 10:55pm, we started seeing DSL customers in the Santa Rosa area reporting sync-no-surf. This appears to be on AT&T’s side of the network, but we are monitoring the situation. We have no ETR at this time. -John F and Technical Support

Thu Feb 2 10:14:34 PST 2006 — Additional DSL Maintenance in San Francisco. Tonight shortly after Midnight we are going to reconfigure the other OC-3 uplinked RedBack SMS 1800 to the new architecture to eliminate the performance issues that have started to affect it as well. Downtime for customers will be brief and the total operation should take less than 30 minutes. -Kelsey and Nathan

Mon Jan 30 13:37:31 PST 2006 — DSL Outage Redux – All affected customers appeared to come back up last night, but we never received an official cause from SBC/AT&T – At approx. 13:10hrs today, we received reports that a significant number of DSL customers in Santa Rosa went down again, and while we have once again received confirmation from SBC/AT&T that it is a problem on their end we still do not have an ETR or official cause. We will continue to contact SBC/AT&T until we have an answer. – Eli, Support

UPDATE: SBC/AT&T appear to have successfully resolved this afternoon’s DSL outage.

Sun Jan 29 21:45:35 PST 2006 — DSL Outage. There appears to be a widespread DSL outage that is affecting a large number of subscribers. We’ve traced it back to SBC/AT&T’s network and are in the process of contacting them now. There is no ETR at this time. -Kelsey, Nathan and Jared

Sat Jan 28 14:20:32 PST 2006 — DSL Maintenance. Tonight, shortly after Midnight, we are re-architecting the termination of the DSL customers that are currently experiencing degraded performance in San Francisco to take advantage of equipment we put in place to terminate the new ATM OC-12 circuit. This is a complex procedure that will result in the subscribers split between two new RedBack SMS 1800s. We will do everything possible to minimize downtime but anticipate some customers may experience outages as long as 30 minutes while the work is underway. Once finished, there will be no more load related performance issues for the affected customers. We apologize for the short notice. -Kelsey and Nathan

Update Sun Jan 29 04:11:38 PST 2006 — The migration has gone surprisingly well. We’ve distributed enough load onto the new DSL termination equipment to mitigate the performance issues. We may do additional maintenance over the next few days if warranted. -Kelsey and Nathan

Wed Jan 25 20:43:59 PST 2006 — Broadlink TMAC outage. Broadlink reports an problem with their TMAC site — they report estimated time of repair should be sometime between 9pm and 11pm. -Scott, Tony, and Jason from Broadlink

Wed Jan 25 16:26:26 PST 2006 — DSL Latency and Loss. We continue to have ongoing problems with some DSL customers served out of San Francisco. The issue manifests itself as intermittent latency and loss, which often results in poor download performance or choppy streaming media. We have identified that the issue is caused by an overloaded router inside of our San Francisco network. Our attempts to mitigate the overload have met with limited success despite repeated calls to the router vendor and extensive in-house troubleshooting. This router was scheduled to be replaced on 12/30 as part of our upgrade to an ATM OC-12 from AT&T. Unfortunately, AT&T was unable to deliver the ATM OC-12 on time, and has now given us a 1/30 date of installation for the new circuit. Shortly after this link is installed we will be moving all of the affected customers over to new routers with ample capacity.

We apologize for the delay in the upgrade and will continue to work diligently with AT&T to improve service quality.

-Nathan, Dane, Eli and Kelsey