We are currently tracking a network event that is disrupting network connectivity for several of our DSL aggregation routers. We are working to identify and resolve this event as quickly as possible.

-Jared and Nathan

Update:  We believe the issue is a backplane problem on the ATM switch serving many of our DSL customers.  A reload of the affected device should resolve the trouble — we’ll let you folks know how it goes. -Nathan and Jared

Update: Things are looking much better after a reload of the ATM switch. We’re still working to ensure that all services are up and functional, and will be working with the equipment vendor to diagnose the trouble that we’re having. Sorry to all affected! -Nathan and Jared

One of our ATM OC-12s suffered a five minute outage, we are investigating, and a further update will follow.

Update: The ATM OC-12 has remained stable and we have tickets open with the provider on the circuit to ensure that the circuit will not have further problems. The outage we had from 16:55 to 17:00 would have affected some of our DSL, Business-T and FRATM customers in the bay area.

-Jared

Update : 5:20pm all services have been restored.

We suffered an internal database failure around 4:30pm today; the data is currently being restored, and the repair time is estimated to be 20 minutes.

You may see errors when trying to use some of our Member Tools because of this.

Updates will be published as they happen.

We will be making changes to directory permissions tomorrow which will limit the ability of users to list the contents of some directories.  You will be able to change directory through them, but you will not be able to list the contents.

This will for example affect users connecting via FTP to upload files – you can go to the directory you own, but you will not be able to list all the directories one level up.  This is a cosmetic change, you can still get to the directory, but you must know its name.

If you have any questions after this change is made, please post to news://news.sonic.net/sonic.os.unix

This Sunday, April 4 at 11PM we will begin power maintenance at our San Jose POP. This maintenance involves moving the A and B sides of our redundant power to a new physical configuration. We will be taking every precaution to ensure that nothing loses power, however, when doing work of this nature, outages can occur. We expect the work to be completed by midnight.

-Jared and Nathan

Some e-mail clients may receive an error about the SSL certificate for pop.sonic.net when checking e-mail beginning this morning around 08:30; we are working to fix the problem and will have a fix shortly.

No e-mail has been lost and you may still check and receive e-mail.

–Augie

This evening, between 12:01AM and 6:00AM, we will be performing intrusive maintenance on equipment serving Fusion and FlexLink ADSL2+ customers in the Pacific Heights, Mission Terrace, Excelsior, and Bayview areas of San Francisco. Affected customers should experience less than 15 minutes of downtime.

-Tim and Juston

This evening, between 12:01AM and 6:00AM, we will be performing intrusive maintenance on equipment serving Fusion and FlexLink ADSL2+ customers in the Financial District, Castro, SoMa, and Mission areas of San Francisco. Affected customers should experience less than 15 minutes of downtime.

-Tim and Juston

Tonight at midnight, we will be performing card insertions and replacements at our San Francisco POP. These operations are being performed on our redundantly-built backbone systems, and traffic will be routed around each device as the work is being done. This should cause no interruption of service to customers served by this POP. This work is to support further DSL expansion, and our new core backbone.

-Jared, Nathan, Matt, and Jacob

The backbone maintenance for tonight has been completed. There were no major incidents, but customers may have noticed 15-30 seconds of poor connectivity as links were removed from service for the maintenance.