At 5:03 PM today, all of our DSL and Business-T subscribers in LATA1 went offline. The problem appeared to be internal to AT&T’s ATM network. As of 5:12 PM, the problem in AT&T’s network appears to have been resolved, and DSL and Business-T customers are back online. We are continuing to monitor the situation and work with AT&T to find out what happened in their network. More details will be forthcoming as we get them.
-Jared, and everyone in the NOC and Support
Update: Word from AT&T and ASI (AT&T’s ATM division) is that they suffered a major fiber issue in one of their major hub Central Offices in San Francisco. Traffic has been routed around the problem, and they are currently working on resolving the fiber issue. There is a small chance that traffic may be interrupted as AT&T works on their issue, but they will be doing everything they can to prevent that. We will be monitoring our ATM links very closely for the next 24 hours to mitigate any potential problems.
Update: AT&T has determined that the outage was caused by a failing ATM fabric card. Service was restored when traffic was moved to the spare fabric card. The failed card has been replaced, with no further service interruption.
Hi, folks,
Thanks for the effective and prompt communication!
Would this outage have affected Business T customers? I ask because we’re about to move to a new office with Sonic.net Business T as the primary connection instead of our current conventional DSL.
thanks,
Graham
I’m back online after the outage, but my 3Mbps line seems to be maxing out at around 1Mbps now.
Nevermind, I just noticed the part in your post where you said “and Business-T”.
Please disregard my earlier comment.
-Graham
Graham – This would have affected any ATM-based access product, which includes DSL, Business-T, FRATM, and native ATM PVCs.
John – I’m not seeing any problems with the overall traffic levels on our ATM circuits, so I do not believe AT&T’s network would be congested. I would recommend that you give our support department a call at your convenience to troubleshoot your DSL line.
Graham, consider one of our optical MPLS products to have diversity from the ATM network. Or, if you’re lucky and are in a location we’re building our own facilities, FlexLink.
Looks like they are having problems again…
We haven’t heard reports of widespread problems this morning. Please give our support department a call if you are experiencing problems with your connection.
I’m seeing the same. My system was apparently affected by yesterday’s outage. It has been inaccessible now for about four hours.
Eric: I can pass ATM traffic through to your DSL modem, and I can see consistent traffic between your router and our DSL gateway. I’d suggest power-cycling your router, and if you’re still unable to get online, try giving our Support department a call.
Ok – where’s LATA1?
@adam: LATA 1 is a region of California that stretches from the immediate San Francisco Bay area (north of Monterey) along the coast to the Oregon border. On this map, it’s the area marked as LATA 722.