Month: June 2006

Network maintenance complete.

Wed Jun 21 01:03:05 PDT 2006 — Network maintenance complete. The scheduled network maintenance tonight went off without a hitch. Affected colo customers would have experienced less than 10 minutes of total downtime as we reloaded our distribution switch onto new code, and migrated it to gigabit uplinks. -Jared, Zeke, Nathan, and Matt

Brief System Event.

Tue Jun 20 14:16:21 PDT 2006 — Brief System Event. An administrative tool gone awry briefly impacted the file server that handles /home. A large number of systems depend upon /home being available and snappy including inbound mail, POP3/IMAP and webmail, as well as shell and FTP. -Jared

Upcoming network maintenance for colo.

Mon Jun 19 12:57:48 PDT 2006 — Upcoming network maintenance for colo. We will be performing a software upgrade on our primary colo distribution switch in Santa Rosa at 12:01 AM Wednesday, June 21. Most colo customers will experience 5-10 minutes of intermittent connectivity as we reload the switch onto new code. -Zeke, Jared, and Nathan is currently experiencing network…

Wed Jun 14 08:40:42 PDT 2006 — is currently experiencing network instability we are investigating the problem and will be working to resolve it shortly. Operations

Update: Wed Jun 14 09:07:45 PDT 2006 — This morning a major DoS attack created instability in part our network in Santa Rosa. Colocation and DSL customers served out of the Santa Rosa facility were affected. A reboot of one of our distribution switches halted the attack and we will continue to investigate. -Matt, Nathan, Augie, Clay and Dan

Emergency customer router maintenance.

Wed Jun 7 15:24:57 PDT 2006 — Emergency customer router maintenance. We will be replacing hardware on one of our customer routers this evening at 12:00. The hardware replacement should only cause a few minutes of downtime for 20 of our T1 customers. We apologize for any inconvenience this may cause. -Jared

Update: Thu Jun 8 00:38:58 PDT 2006 — Router maintenance complete. The maintenance went as expected. The card replacement and all associated downtime should have taken less than 5 minutes. All T1 lines on the affected hardware have been confirmed up and running. -Jared and Matt

DSL outage in LA.

Tue Jun 6 11:33:55 PDT 2006 — DSL outage in LA. Our ATM DS3 that serves DSL to the Los Angeles region went down for 5-10 minutes at approx 11:15 this afternoon. The circuit is back up now and all customers should be up and running again. We are investigating the cause with the provider of the DS3. -Jared, Russ, and Nathan

Update: Wed Jun 7 13:56:21 PDT 2006 — The LA region DSL circuit began to show errors again for about 30 minutes this afternoon. The transport provider was able to see the problem and has their engineers working on the problem now. At this time all customers should be up and running. -Jared

Update: Sat Jun 10 16:12:44 PDT 2006 — The LA region’s circuit is once again showing errors. We’re working with our transport provider to fix the link. Unfortunately, we don’t have an ETR, but are hopeful the problem will be resolved soon. -Nathan

Update: Sat Jun 10 20:35:16 PDT 2006 — The circuit has been repaired. Our transport provider found a failure in one of their long-haul transmission cards. The card has been replaced and we expect no further interruption on this link. -Nathan