NVIDIA Dynamo: Supercharge AI Now

NVIDIA has launched Dynamo into production, marking a pivotal advancement in AI inference at scale. This open-source framework acts as the operating system for AI factories, enabling seamless deployment of generative AI models across massive GPU clusters with unprecedented efficiency and low latency.

What is NVIDIA Dynamo?

Dynamo is a high-throughput, low-latency inference-serving framework designed for distributed environments. It supports generative AI and reasoning models by disaggregating inference phases, such as prefill and decode, across multiple GPUs. This approach optimizes resource use, boosts throughput, and reduces costs in data center-scale deployments.

Key to its design is intelligent resource scheduling, which dynamically allocates GPUs based on demand. It also features LLM-aware request routing to minimize redundant computations by leveraging existing KV caches, where the model state is reused for similar queries.

Core Innovations Driving Performance

Dynamo introduces several breakthroughs that set it apart from traditional inference servers.

• Disaggregated Serving: Separates prefill (processing user queries) and decode (generating responses) stages onto different GPUs, allowing independent optimization for higher throughput.
• Dynamic GPU Scheduling: Adds, removes, or reallocates GPUs in real-time to handle fluctuating workloads and traffic in multi-model pipelines.
• LLM-Aware Routing: Directs overlapping requests to GPUs with matching KV caches, avoiding recomputation and slashing latency.
• Accelerated Data Transfer: Uses the NVIDIA Inference Transfer Library (NIXL), a low-latency library for fast KV cache movement between GPUs, memory tiers, and storage via GPUDirect RDMA, NVLink, or EFA.
• KV Block Manager (KVBM): Manages cost-effective KV cache offloading to system memory or storage, freeing GPU resources while preserving performance.

Broad Compatibility and Real-World Gains

Dynamo integrates seamlessly with popular open-source engines like vLLM, SGLang, TensorRT-LLM, and PyTorch. It runs on NVIDIA GPUs from Ampere onward, including Hopper and Blackwell architectures, and works in heterogeneous environments by supporting existing inference stacks.

Performance benchmarks show dramatic improvements: up to 30x more requests served on Blackwell with DeepSeek-R1 models compared to Hopper, and doubled inference speed for Llama models on Hopper systems. Major adopters include AWS, Cohere, CoreWeave, Google Cloud, Meta, Microsoft Azure, and others accelerating AI inference in production.

From Announcement to Production Deployment

Revealed at GTC 2025 as the successor to Triton Inference Server, Dynamo has rapidly entered production. Available via GitHub and NVIDIA container images (NIMs), it simplifies large-scale serving with modular components for custom needs, including logging, monitoring, and security.

Why Dynamo Defines the Future of AI Factories

Dynamo transforms AI factories by maximizing token revenue through efficient scaling and cost reductions. As reasoning models demand more compute, its disaggregated architecture and smart optimizations ensure faster responses and higher utilization. This framework positions NVIDIA at the forefront of production-grade AI inference, empowering enterprises to deploy advanced models at data center scale.

In summary, NVIDIA Dynamo's production rollout delivers the tools needed for next-generation AI deployments, blending innovation with broad accessibility for developers and operators alike.