Amazon SageMaker Inference Unlocks GPU Efficiency with Multi-Instance GPU Support

Amazon Web Services (AWS) has announced a significant enhancement to its Amazon SageMaker Inference service, introducing support for NVIDIA Multi-Instance GPU (MIG). This new capability allows customers to partition a single physical GPU into multiple isolated instances, dramatically improving GPU utilization and enabling more efficient deployment of machine learning models.

The Chenab Times has learned that traditionally, when deploying models to GPU endpoints, users were constrained by the smallest unit of GPU available, often requiring an entire accelerator even for models with modest resource needs. This limitation led to inefficient resource allocation, where instances with multiple GPUs might host only a few models, or where models that could benefit from newer, more powerful GPUs were relegated to older, less capable hardware due to cost or availability, resulting in either underutilization or overpayment.

Maximizing GPU Utilization and Cost Efficiency

NVIDIA MIG addresses these challenges by dividing a physical GPU into smaller, independent partitions. Each partition is allocated a dedicated share of the GPU’s compute units and memory, with hardware-enforced isolation. This partitioning allows users to host multiple distinct models on a single GPU, each operating within its allocated resources. The flexibility extends to selecting precisely the fraction of a GPU needed, rather than committing to a full accelerator, thereby optimizing costs and performance.

Previously, users seeking GPU partitioning had to resort to self-managed Kubernetes solutions, which involved considerable operational overhead in managing driver versions, partition configurations, and scheduling across a fleet of instances. Amazon SageMaker Inference now integrates MIG directly, abstracting away this complexity and allowing users to leverage hardware-isolated GPU sharing on a managed service.

How MIG Enhances Inference Deployments

NVIDIA MIG offers several key advantages for machine learning inference:

• Predictable Latency: Dedicated memory bandwidth for each partition ensures that traffic spikes from one model do not impact the performance of others.
• Fault Isolation: Issues or crashes within one partition are contained, preventing them from affecting other models running on the same GPU.
• Right-Sized Capacity: Smaller and medium-sized models can be allocated partitions that precisely match their requirements, avoiding the waste associated with whole GPUs.
• No Cold Starts: Each model maintains its own partition and remains loaded, eliminating the load penalty associated with on-demand model loading.
• Tenant Isolation: Hardware-enforced memory protection between partitions simplifies security and compliance reviews for multi-tenant environments.
• Reliable Planning and Chargeback: Consistent performance across partitions simplifies load testing and allows for more accurate cost allocation to specific models, teams, or tenants.

The partitioning capability builds upon SageMaker’s existing inference components, which allow for the deployment of multiple models to a single endpoint. Users can now specify an AcceleratorPartitionLayouts for their endpoint configuration and then select a specific AcceleratorPartitionType for each inference component.

Performance Gains and Cost Benefits

To demonstrate the value of MIG, AWS conducted performance comparisons. In one test, a single mig-3g.40gb partition of an NVIDIA H100 GPU was compared against a full, unpartitioned NVIDIA A100-40GB GPU. The results indicated that the H100 partition consistently outperformed the A100 across various traffic shapes and concurrency levels, delivering higher output tokens per second and lower inter-token latency. Furthermore, the partition offered cost savings of 18-34 percent per million output tokens for certain workloads.

However, the benefits are nuanced. For low-concurrency or short-input workloads, the whole A100 GPU remained more cost-effective. The study suggests that users should treat available GPU fractions as distinct GPU types and benchmark them against their specific workloads to determine the optimal configuration. Partitioning is most beneficial for scenarios such as hosting a portfolio of small to mid-size models, multi-tenant model serving, multi-stage inference pipelines on a single instance, or running auxiliary models like safety classifiers.

Implementation and Availability

GPU partitioning is available on SageMaker’s P4d, P5, G7, and G7e instance families. Users can configure partitions during the endpoint creation process. While the hourly cost of the underlying instance remains the same, the ability to better utilize the GPU capacity can lead to significant cost efficiencies and performance improvements. Models that genuinely require a full accelerator or multiple accelerators can still be deployed as such, as partitioning does not span across multiple physical GPUs.

The new feature is accessible in several AWS Regions, including US East (N. Virginia), US East (Ohio), US West (Oregon), Asia Pacific (Singapore), Asia Pacific (Tokyo), Europe (Frankfurt), and Europe (Ireland). AWS provides detailed documentation and sample notebooks to help users get started with implementing GPU partitioning.

Salesforce, an early participant in the development, highlighted MIG’s potential to consolidate workloads safely, improve GPU utilization, reduce inference costs, and maintain predictable performance with strong isolation. Richard Chang, Software Engineering Architect at Salesforce, noted that this capability would help scale AI Inference more efficiently across different use cases and deliver meaningful technical and business value for enterprise customers.

---

---

Follow us on

---

(We don't allow anyone to copy content. For Copyright or Use of Content related questions, visit here.)