Month: August 2009

DHCP Server Failure

This morning at approximately 8 AM our Support staff became aware that one of our DHCP servers was not serving DHCP leases properly, in such a way that our monitoring systems were not able to detect the problem. Our Support team escalated the problem to the Operations staff and we were able to remedy the problem by rebooting the partially-locked server. Service was restored by approximately 8:28 AM. We apologize for this outage and will be looking into ways of extending our monitoring so this unusual type of failure will not be missed again.

-Jared and the Support team

Update: The DHCP server has begun to fail again, so we have removed it from service. DHCP should be functioning normally again. If you are having problems obtaining a DHCP lease, please try rebooting your modem and any computers or routers that are having problems.

Updates on E-mail maintenance.

A couple of issues showed up this morning that were not picked up by our monitoring systems or testing after the night-op.  First, some users migrated to the new message stores were unable do use IMAP or webmail to access their e-mail.  This problem was fixed shortly after Tech Support reported it this morning.  Second, one the servers in our inbound mail cluster was left in the old configuration and delivered some new mail to the old message stores.  This was also fixed shortly after Tech Support reported it and any misdirected email has been redelivered to the correct location.  -Kelsey

E-mail service interuption.

For about an hour this afternoon SSL connections to our POP servers (pop.sonic.net) were unavailable; we have remedied the problem and are investigating the cause.

During this time non-SSL connections to pop.sonic.net still worked, and both SSL and non-SSL connections to our IMAP cluster (imap.sonic.net) were unaffected.

–Augie, Kelsey, and William.

E-mail infrastructure maintenance.

Update : E-mail maintenance finished a bit ago, just cleaning up; total observed down time around 15 minutes. -Augie

This Saturday (29, August) at 12:01 AM we will be performing the first of four service impacting e-mail migrations from our existing e-mail storage infrastructure to our new (and greatly improved) storage system.

When it is completed this migration will mark a substantial improvement in our e-mail systems performance and availability for all of our customers.

During the maintenance window (12:01am – 1:00am) all customer e-mail systems will be un-available; this includes POP, IMAP, and Webmail.

–Augie

Sacramento ATM Maintenance

This Friday, August 21 at 12:01 AM we will be performing maintenance to upgrade an ATM aggregation circuit. We are doing this work to clear a capacity problem that the existing link faces during peak times. Sorry for the short notice.

-Jared and Nathan

Update: The migration is now complete. Downtime for affected customers was approximately 5 minutes, except for 4 folks that many have seen as much as 15 minutes worth of outage. -Nathan

ATM OC12 Maintenance

This Wednesday, August 19 at 12:01 AM we will be performing fiber maintenance on one of our OC12 circuits that aggregates DSL service in the Bay Area. We will be moving the OC12 to a new in-building cross-connect. Expected downtime is less than 5 minutes as we move the fiber from one cross-connect to another. This maintenance will affect some DSL and Business-T customers in the Bay Area.

-Jared and Nathan

Update: This maintenance is now complete. Actual downtime was approximately 4 minutes and 40 seconds.

bolt security update

We upgraded the kernel running on ‘bolt’ (our customer shell server) today to address a potential security issue.
This required a reboot of the system. The system is backup and running smoothly.
-William

Webcluster Security Changes

We have completed the migration to our new web hosting security configuration. The new system improves our already robust web site hosting platform with fully compartmentalized security and eliminates the need for cgiwrap configurations. For more information please see news://news.sonic.net/sonic.help.www -Kelsey and Nathan

ADSL2+ Speed Reprofiling

This Wednesday, August 5 at 12:01 AM, we will be re-profiling all of our ADSL2+ lines. This will cause all ADSL2+ lines to have to re-sync, causing a minute or two of downtime for any Fusion or Flexlink ADSL2+ customer. If you have problems with your ADSL2+ line after this operation, please reboot your modem.

-Jared

Update: We have completed the re-profiling of all our ADSL2+ lines. All affected lines should be up and running at the new speeds.