Fri Feb 16 18:30:26 PST 2007 — DSL Service disruption. One of the OC-12s that terminates DSL subscribers in LATA 1 (most of Northern California) is offline. Our NOC staff is already onsite at the POP and is working with AT&T and the facility providers to ensure a speedy fix. -Kelsey, Alan and the 7065 NOC.

Update Fri Feb 16 19:40:40 PST 2007 — Service restored. As of approximately 7:30pm all services on the affected OC-12 have been restored. The problem was determined to be faulty cross-floor fiber patching at our 200 Paul Ave. facility. To expedite service restoration we brought some otherwise idle gear into service, which allowed us to bring customers back by eliminating intermediate fiber connections. We will be scheduling a maintenance window in the near future to clean the cross-floor run and return the circuit to its original configuration. We apologize for the late update to this entry, not to mention the downtime! -Nathan, Zeke, Matt and Juston

Thu Feb 15 21:33:58 PST 2007 — Transport bounce. Between 9:05 and 9:20pm this evening, we experienced a number of brief transport outages resulting in packet loss. Customers may have experienced transient connectivity problems to some sites. We are currently working to mitigate and abate this. — Nathan, Zeke, Matt, and John F

Thu Feb 15 17:40:41 PST 2007 — Changes to SuperNews NNRP Services. Today we have implemented the authentication changes to our SuperNews NNRP access that we announced on Feb 6. Users connecting to supernews.sonic.net will need to supply their username in the format of ‘username@sonic.net’ and normal account password. -Kelsey and Jared

Tue Feb 13 09:06:29 PST 2007 — More traffic disruption. The same problem we experienced with one our our upstream providers network dropping large packets on Friday has cropped back up. We’re currently working with the provider now and hope that they are able to have it resolved quickly. -Kelsey and Nathan

Update Tue Feb 13 10:29:43 PST 2007: Pending full resolution from the upstream provider we rerouted affected traffic. We’ll continue to keep a very close eye on things. -Nathan and Kelsey

Fri Feb 9 11:53:08 PST 2007 — Internet traffic disruption. We’re currently tracking an issue inside one of our upstream provider’s networks that manifests itself as an inability to receive packets larger than a certain size. 1476 byte packets appear to be forwarded correctly, while 1477 byte and longer packets are dropped. We’ve identified the broken equipment in the provider’s network and have re-routed traffic to mitigate the problem. We are also working with the provider to solve the issue. We’ll post an update when more information is available. -Nathan and Kelsey

Update Fri Feb 9 14:42:10 PST 2007: Our upstream provider has fixed the trouble, which turned out to be a partially failed router in San Jose. -Nathan

Fri Feb 9 09:15:51 PST 2007 — Inbound mail delays. We are seeing delays in the delivery of inbound mail due to unusually high spam volume overnight. We are bringing additional resources to bear, and expect resolution over the course of the day. – Eli, Operations

Tue Feb 6 11:34:04 PST 2007 — Changes to SuperNews NNRP Services. Friday morning we are going to change our SuperNews NNRP access to use username based authentication in place of the existing IP based access control lists. Users connecting to supernews.sonic.net will need to supply their username in the format of ‘username@sonic.net’ and normal account password. For more information please see www.sonic.net/features/news/ -Kelsey

Wed Jan 31 10:25:29 PST 2007 — CRAC Testing and Evaluation. We will be conducting some tests on our current data center and colo facility CRAC (Computer Room Air Conditioning) to see how our redundant systems will behave in a failure scenario. This should not have any impact on our customers. -Kelsey, Nathan and Russ

Sun Jan 28 19:18:00 PST 2007 — Hotspot DSL outage. Currently, we are experiencing difficulties with the hardware that terminates our Hotspot-enabled DSL lines. Customers who have Hotspot-enabled DSL lines will have no connectivity at this time. We are working on repair or replacement of the affected hardware, estimated time of repair is one hour. We apologize for the inconvenience. -Nathan and Jared

Update Sun Jan 28 19:53:52 PST 2007 — Hotspot service has been restored. Sorry for the downtime! -Nathan and Jared

Wed Jan 17 21:14:30 PST 2007 — Customer router reload. One of our customer routers in Santa Rosa crashed and reloaded at approximately 9:10 this evening. Only customers with T1s served by this router would have seen 5 minutes of downtime. We will be investigating the cause of this crash and taking action as necessary. -Jared