LATA1 AT&T ATM Outage

At 5:03 PM today, all of our DSL and Business-T subscribers in LATA1 went offline. The problem appeared to be internal to AT&T’s ATM network. As of 5:12 PM, the problem in AT&T’s network appears to have been resolved, and DSL and Business-T customers are back online. We are continuing to monitor the situation and work with AT&T to find out what happened in their network. More details will be forthcoming as we get them.

-Jared, and everyone in the NOC and Support

Update: Word from AT&T and ASI (AT&T’s ATM division) is that they suffered a major fiber issue in one of their major hub Central Offices in San Francisco. Traffic has been routed around the problem, and they are currently working on resolving the fiber issue. There is a small chance that traffic may be interrupted as AT&T works on their issue, but they will be doing everything they can to prevent that. We will be monitoring our ATM links very closely for the next 24 hours to mitigate any potential problems.

Update: AT&T has determined that the outage was caused by a failing ATM fabric card. Service was restored when traffic was moved to the spare fabric card. The failed card has been replaced, with no further service interruption.

11 comments for “LATA1 AT&T ATM Outage

  1. Graham Freeman
    September 4, 2008 at 5:34 pm

    Hi, folks,

    Thanks for the effective and prompt communication!

    Would this outage have affected Business T customers? I ask because we’re about to move to a new office with Sonic.net Business T as the primary connection instead of our current conventional DSL.

    thanks,

    Graham

  2. John
    September 4, 2008 at 5:38 pm

    I’m back online after the outage, but my 3Mbps line seems to be maxing out at around 1Mbps now.

  3. Graham Freeman
    September 4, 2008 at 6:08 pm

    Nevermind, I just noticed the part in your post where you said “and Business-T”.

    Please disregard my earlier comment.

    -Graham

  4. jared
    September 4, 2008 at 6:31 pm

    Graham – This would have affected any ATM-based access product, which includes DSL, Business-T, FRATM, and native ATM PVCs.

    John – I’m not seeing any problems with the overall traffic levels on our ATM circuits, so I do not believe AT&T’s network would be congested. I would recommend that you give our support department a call at your convenience to troubleshoot your DSL line.

  5. dane
    September 4, 2008 at 6:35 pm

    Graham, consider one of our optical MPLS products to have diversity from the ATM network. Or, if you’re lucky and are in a location we’re building our own facilities, FlexLink.

  6. Anonymous
    September 5, 2008 at 8:13 am

    Looks like they are having problems again…

  7. kavan
    September 5, 2008 at 9:02 am

    We haven’t heard reports of widespread problems this morning. Please give our support department a call if you are experiencing problems with your connection.

  8. Eric
    September 5, 2008 at 5:54 pm

    I’m seeing the same. My system was apparently affected by yesterday’s outage. It has been inaccessible now for about four hours.

  9. adame
    September 5, 2008 at 7:23 pm

    Eric: I can pass ATM traffic through to your DSL modem, and I can see consistent traffic between your router and our DSL gateway. I’d suggest power-cycling your router, and if you’re still unable to get online, try giving our Support department a call.

  10. Adam
    September 8, 2008 at 8:11 pm

    Ok – where’s LATA1?

  11. September 8, 2008 at 9:40 pm

    @adam: LATA 1 is a region of California that stretches from the immediate San Francisco Bay area (north of Monterey) along the coast to the Oregon border. On this map, it’s the area marked as LATA 722.

Leave a Reply

Your email address will not be published. Required fields are marked *

*