Month: September 2008

San Jose POP Retrofit

This Thursday, Oct 2 at 12:01 AM we will begin work retrofitting the backbone of our San Jose POP. This retrofit will enhance the redundancy and capacity of this POP to accomodate our growth and new CLEC-based products. Currently this POP does not directly serve any customers, so there should be no customer impact. We will route customer traffic away from the POP, so you may see longer routes to Internet hosts than normal during this time. We expect the operation to take 60-90 minutes.

-Jared and Nathan

DSL Subscriber Rebalancing

On Tuesday September 30 at 12:01 AM we will be moving several hundred DSL subscribers from one of our DSL aggregation routers to another, to balance network load. This move should take less than 5 minutes and will only cause a brief service interruption.

-Jared

Update: Rebalancing completed. Downtime for affected customers was under 5 minutes.

IMAP performance problems solved.

We believe we have fixed the recent IMAP performance problems; let us know if you have any trouble with imap.sonic.net still.

For the past few days we have received a number of reports from customers concerned about periodic slow IMAP performance to imap.sonic.net; the problem would manifest as a mail client behaving very slowly when reading e-mail or timing out all together.

The periodic nature of the problem made it very difficult to troubleshoot as by the time a customer would call to alert us to the problem it would then mysteriously go away.

We believe we have solved the problem and have since implemented much more detailed monitoring of the imap.sonic.net IMAP cluster.

Note: we have seen a handful of reports of slow Webmail performance; those servers are completely separate from the imap.sonic.net cluster, so if you are seeing Webmail performance problems, please note the time it occurred, the error you saw if any and send an e-mail to support@sonic.net .

Updates to password changing tool

In the interest of customer security we have updated our password changing tool to email main account holders when either the main account, or any mailbox under that account has the password changed.  The email also will go to any “invoice to email” addresses you have on the account.  We hope that this is useful in notifying customers immediately if their account or mailboxes have been compromised.

System Authentication Failure

This afternoon, around 4:30PM we experienced a problem in our main authentication system. This problem effected a handful of user accounts and associated mailboxes. Account and mailbox owners would have seen authentication failures and possible timeouts when attempting to connect to services such as shell, pop/imap, and webmail services. The underlying problem has been fixed.  Customers may have seen approximately 10 minutes of authentication failure. If you are experiencing problems with authentication, please call Sonic.net Support at (707) 522-1000. We appologize for the inconvenience this has caused. –Don and Jared

Mail Service Interuption.

Early this morning around midnight one of our clustered NetApp filers suffered a critical failure which caused one of its Ethernet Interfaces to lock up.  Some customers may have noticed timeouts or other errors when while trying to check their Mail. The total downtime for the service was around 15 minutes. We will be investigating the problem further with our vendor in the waking hours.  –Augie and Don.

Intermittent mail delivery problem from Customer Web Cluster.

Today we fixed a problem on the Customer Web Cluster that would have effected mail flow from Customer scripts on our Web Cluster that were using the local Sendmail program to send e-mail. The problem began on Sunday and would have appeared as an intermittent failure to Customer scripts as only two of the servers in our Web Cluster displayed the problem.

We will be crafting more detailed monitoring so that we can resolve this problem much faster in the future.

–Augie and William.

VPN Concentrator Certificate Expiry

Today the identity certificate for our VPN concentrator expired. This would have prevented new VPN sessions from establishing. We have renewed the certificate, and the VPN concentrator is accepting connections normally now.

-Jared and Nathan

Upstream Routing Issue

Starting this morning, we observed a transient problem with one of our upstream providers. Impact would have been inability to connect to certain websites. We have taken action to correct this issue. -Nathan, Tim and the NOC.

SpamAssassin Outage

SpamAssasin may have stopped filtering mail for some users this afternoon. No mail was lost during this period. Service has restored to normal at this time and mail is being filtered for all users again. We apologize for any inconvenience this may have caused. -William