Sat Apr 12 09:08:51 PDT 2008 — Los Angeles DHCP Failure. One of our DHCP servers in our Los Angeles PoP suffered a failure that prevented it from providing DHCP leases at approx. 2:30 AM today. This problem would have interrupted connectivity for dynamic IP DSL subscribers in the Los Angeles area until the problem was resolved 6 hours later at 8:40 AM. We apologize for the outage, and will be investigating the root cause of the problem our DHCP server experienced. -Jared and Tim

Tue Apr 8 22:02:22 PDT 2008 — Custsql MySQL replication upgrade tonight. This evening, at 10:00PM PDT, I will be performing this upgrade which will make our system much more robust and add yet another layer of data protection to your databases! For a very short period, you may experience your sessions timeout; they will be back shortly. -Don

Mon Apr 7 23:29:19 PDT 2008 — @Mail Webmail Service Interruption. Earlier this evening some routine server maintenance caused @Mail to go down unexpectedly. We are sorry for the delays in returning it to service; it took an unusual quantity of sleuthing to find the errors introduced last week leading to the failure tonight. At this time @Mail is fully operational. -Kelsey, Don and Augie

Update Thu Apr 3 02:22:47 PDT 2008 — Emergency ATM Switch Maintenance. When we performed our card reload earlier tonight, we apparently hit a bug which caused an interface to stop forwarding ATM cells, but appear to be functioning perfectly. This caused a loss of connectivity for DSL some subs in LATA1 with odd-numbered VPs. Upon investigation of the bug, the only way to resolve it was to reload the entire ATM switch, which took place at 2:15 AM, and caused a 5-minute outage for all DSL subs connected to it. This includes a large amount of LATA1, as well as all of Chico, San Diego, and Stockton. Currently all traffic levels look normal on all interfaces, and we will be closely monitoring this switch for the near future. -Jared and Nathan

Update Thu Apr 3 00:40:57 PDT 2008 — ATM Switch Maintenance Complete. We have completed our reload of the affected card and everything appears to be operating normally. There should have been no customer impact. -Jared

Wed Apr 2 10:40:07 PDT 2008 — ATM Switch Maintenance. A backup management card in one of our ATM switches began experiencing errors this morning. Tonight at midnight we will be reloading the card to clear these errors. Since this card is the backup redundant management card, there should be no customer impact during the reload. -Jared and Nathan

Fri Mar 28 17:16:54 PDT 2008 — GigaMAN maintenance. Tonight we will be performing maintenance with AT&T on one of our two redundant GigaMAN circuits. We have removed all traffic from the affected circuit, so there should be no customer impact. During the maintenance, however, our Santa Rosa POP will no longer be redundantly connected to the network. We do not anticipate any problems with this maintenance. -Jared and Nathan

Update Tue Mar 25 16:09:49 PDT 2008 — Correction, resolution is expected in 2 hours, minimum. – Eli, John F, and John K

Update Tue Mar 25 15:15:28 PDT 2008 – AT&T has identified the problem and they are working to correct it. We expect resolution in the next 20-30 minutes. – Eli, Support

Tue Mar 25 14:04:31 PDT 2008 — Intermittent DSL outage, Santa Rosa. DSL customers in Santa Rosa are experiencing intermittent outages that last for 5-10 minutes at a time. We are working with AT&T to resolve the issue, but we do not have an ETR at this time. – Eli