AT&T and Microsoft scale trillion-token workloads with Microsoft Foundry and AMD

When you’re processing a trillion tokens—that’s roughly equivalent to reading Shakespeare’s complete works 2 million times over—you need infrastructure that can keep up without breaking the bank. AT&T recently did exactly that while developing OTel2.0 (OpenTelemetry 2.0), a critical observability framework, using Microsoft Foundry’s Managed Compute platform paired with AMD and NVIDIA GPUs. This wasn’t a one-off experiment; it represents a fundamental shift in how enterprises approach large-scale AI workloads in production environments. The story here isn’t just about the size of the computation—it’s about the flexibility and cost-effectiveness that made it possible.

Let’s break down what actually happened technically. Microsoft Foundry provides a managed compute layer that abstracts away much of the infrastructure complexity while maintaining access to raw performance. AT&T leveraged this to run open AI models (models not locked into proprietary ecosystems) across heterogeneous GPU infrastructure—mixing both AMD and NVIDIA accelerators. This is the key detail: most enterprises commit to one GPU vendor and live with that decision for years. By building workloads that could run on multiple GPU types, AT&T gained negotiating power, avoided vendor lock-in, and could select hardware based on availability and cost at any given moment. They paired this with Python-based training pipelines and API-driven model serving, architectures your team likely already understands. The trillion-token scale was managed through distributed training, chunking massive datasets into manageable pieces across multiple compute nodes.

Why does this matter for your infrastructure decisions? First, the cost implications are substantial. Running a trillion-token workload on a single vendor’s hardware could easily run into seven figures; flexibility in hardware selection can reduce that by 20-30% depending on market conditions. Second, this demonstrates that production-grade AI doesn’t require proprietary platforms or closed ecosystems. Open models with open infrastructure are reaching feature parity with their closed counterparts, and AT&T’s success with OTel2.0 proves the business case. Third, this is directly applicable to your own workloads—whether you’re fine-tuning models for customer support, building recommendation engines, or processing logs at scale, the pattern holds: managed compute layers plus flexible model choices equal sustainable economics.

The practical takeaway is straightforward: if you’re planning major AI infrastructure investments, the days of single-vendor commitments are ending. Start exploring multi-GPU support in your training pipelines now, investigate managed platforms that don’t lock you into one hardware vendor, and test with open models from the Hugging Face ecosystem. AT&T’s trillion-token achievement wasn’t about raw innovation; it was about smart architectural choices that allowed proven technology to scale sustainably. That’s a blueprint you can follow today.