Microsoft has come swinging in the battle of custom hyperscale silicon, debuting its “AI inference powerhouse” Maia 200 accelerator.

Built on Taiwan Semiconductor Manufacturing Company's (TSMC) 3nm process, the hyperscaler claims Maia 200 offers three times the FP4 performance compared to AWS’ Trainium3 and FP8 performance stacked against Google’s seventh-gen ‘Ironwood’ TPU.

In a bold attack on its hyperscale rivals, Microsoft claims its new accelerator is 30% cheaper to run than any other silicon on the market today, yielding what Scott Guthrie, Microsoft’s EVP for cloud and AI, described as "the best token, per watt, per dollar."

Microsoft optimizes the network stack for scale-up

In terms of its networking stack, Microsoft continues the growing industry acceptance of two-tier network topologies for scale-up.

Where traditional topologies have involved three-tier designs, which allow for more cabinets to be connected at the cost of engineering headaches due to their inherent complexity, the rise of two-tier designs aims at drastically reducing latency.

The two-tier scale-up network that powers interconnected Maia 200s is built on standard Ethernet – another win for the resurgent protocol.

A custom transport layer combined with an on-die integrated NIC (network interface card) provides some 2.8 TB/s (terabit-per-second) of bidirectional bandwidth across clusters of up to 6,144 accelerators.

Each tray features four Maia accelerators, with every chip connected with direct, non‑switched in-tray links to keep high‑bandwidth communication local for optimal inference efficiency. Such an approach means its in-tray connected chips avoid “inefficiencies” caused by external switches, which tend to add hop latency and power, along with port and cabling overprovisioning.

For intra-rack and inter-rack networking, Microsoft then employs its AI Transport Layer (ATL) protocol, the hyperscaler’s own specialized, Ethernet-based protocol to deliver low-latency communication across nodes. A technical blog revealed that the hyperscaler’s custom, in-house network controller also supports features like packet spraying, multipath routing, and congestion-resistant flow, meaning Maia 200 workloads can be controlled directly in the transport layer to maximize throughput and stability.

The scaleup hardware stack is then supported at the communication layer by the Microsoft Collective Communication Library (MCCL), a software library built on top of Nvidia’s NCCL to make AI training and inference workloads running on Azure faster.

The software library aims to maximize Maia workloads by providing tools like hierarchical collectives to automate network traffic management. The tools can directly improve the speed and responsiveness of distributed inference tasks and help to reduce incast congestion, where parallel senders overwhelm a single receiver, causing packet loss, stalling, and worst of all from a cost perspective: idling GPUs.

“Together, the interconnect hardware and MCCL software deliver a tightly integrated, inference optimized scaleup platform capable of supporting the next generation of large-scale, low-latency AI deployments,” Microsoft silicon VPs Saurabh Dighe and Artour Levin wrote in a blog post.

Maia’s memory magic

Hungry to get your hands on Maia 200? The hardware has already been deployed in Microsoft’s U.S. Central data center region near Des Moines, Iowa. And it’s set for expansion, with Microsoft revealing that next in line to receive the custom hardware will be its U.S. West 3 region near Phoenix, Arizona.

The hyperscaler’s so-called Superintelligence team is also set to use Maia 200s for synthetic data generation and reinforcement learning to improve the creation of in-house models. The group, formed last November, is led by former DeepMind co-founder turned Microsoft exec Mustafa Suleyman as a counter to Meta’s Superintelligence Labs, which aims to develop the next-generation of ‘frontier’ AI models.

But in addition to a highly optimized network stack powering its top-level performance, Maia 200’s entirely redesigned memory subsystem helps to push it to the next level.

Made up of 272 MB of on-die SRAM (static random access memory) combined with 216 GB of HBM3e (high bandwidth memory 3E), the reimagined memory component helps Microsoft’s Maia line offer 7 TB/s of HBM bandwidth capacity, making it more than suitable to handle even the most intensive workloads.

But the redesign subsystem also means the chip line can better minimize off-chip traffic and reduce HBM demand – an integral addition considering the commodity has become an ever-increasing luxury in today’s market.

The accelerators’ Direct Memory Access (DMA) subsystem is also tightly coupled with its Network-on-Chip (NoC) unit to further boost performance levels.

Dighe and Levin explained: “The DMA engines are architected for multichannel, high-bandwidth transfer and support 1D/2D/3D strided movement, enabling common ML tensor layouts to be moved efficiently between on-chip SRAM, HBM, and external interfaces while overlapping data movement with compute.

“Meanwhile, the NoC provides scalable, low-latency communication across clusters and memory subsystems and supports both unicast and multicast transfers, an important capability for distributing tensor blocks and coordinating parallel execution.”

A very early Christmas present

In addition to showcasing its all-conquering custom silicon, Microsoft also debuted the software development kit (SDK) for Maia 200.

Featuring support for PyTorch, a simulator tool, and cost calculator, Microsoft hopes that providing early SDK access will entice AI developers looking for access to high-end hardware to get tinkering ahead of general availability (a timeline for which wasn’t revealed).

Guthrie said the SDK “gives developers fine-grained control when needed while enabling easy model porting across heterogeneous hardware accelerators.”

Eager developers can sign up for a preview of the Maia 200 SDK now.

The Maia SDK launch comes hot on the heels of Microsoft adding new features to its quantum application development platform. The Quantum Development Kit (QDK) received additions to support developers and researchers working on near-term quantum hardware.