NVIDIA’s Run:ai GPU Fractioning, integrated with Kubernetes, offers a novel approach to sharing Graphics Processing Units (GPUs) within Amazon Elastic Kubernetes Service (Amazon EKS). This technology builds upon existing methods like Multi-Instance GPU (MIG) and time-slicing, introducing per-Pod memory contracts and enforcement to optimize GPU utilization for AI workloads.
The Chenab Times has learned that NVIDIA Run:ai GPU Fractioning provides advanced scheduling capabilities for AI workloads orchestrated through Kubernetes. By enabling shared access to GPUs on Amazon EKS, it addresses the challenge of idle GPU capacity, particularly for tasks like interactive notebooks, model evaluations, and smaller inference services that might otherwise leave teams waiting.
The core innovation of NVIDIA Run:ai GPU Fractioning lies in its integration with Kubernetes scheduling, coupled with enforced GPU memory limits and per-Pod metrics. This system is designed for NVIDIA GPUs that meet specific hardware and software requirements, including those that do not support MIG. In a validated scenario, a single NVIDIA A10G Tensor Core GPU was successfully shared between two Pods. Each Pod was allocated 6 GiB of memory within an 8 GiB limit, demonstrating the system’s ability to manage and enforce memory contracts.
NVIDIA announced the open-source release of NVIDIA Run:ai as part of its DSX AI Factory platform on October 13, 2026. The implementation detailed in this walkthrough utilizes two key components: the KAI Scheduler and the GPU Fractioning Operator. While other components like KAI Resource Management and Workload Map exist for organizational governance, they are outside the scope of this specific example.
The process involves building a custom Amazon Linux 2023 (AL2023) image and configuring GPU memory sharing on Amazon EKS. The validated walkthrough, using an Amazon EC2 g5.xlarge instance with an NVIDIA A10G GPU, establishes a baseline for whole-GPU use, then configures enforced 8 GiB memory contracts per Pod. It verifies successful 6 GiB allocations, inspects per-Pod metrics, and tests the failure of allocations exceeding the defined limit.
Understanding GPU Sharing Approaches
Various methods exist for GPU sharing, each with distinct isolation and utilization characteristics. MIG partitions supported GPUs at the hardware level. Device-plugin time-slicing offers GPU sharing without strict memory or fault isolation. Multi-Process Service (MPS) allows concurrent CUDA processes, with fault containment dependent on configuration. NVIDIA Run:ai GPU Fractioning, however, enhances software-based sharing by adding enforced per-container memory limits. It supports both time-slicing (default) and sm-sharing compute modes. While NVIDIA’s documentation outlines separate workload error domains for its default mode and a shared domain for sm-sharing, these are not presented as security guarantees. This specific walkthrough focuses on memory limit enforcement rather than fault containment or compute performance, noting that a memory limit does not reserve proportional compute power.
Amazon EKS itself offers several GPU sharing options, including MIG, time-slicing via the NVIDIA device plugin, and time-slicing with the NVIDIA DRA driver and Karpenter. For guidance on selecting the most appropriate method, Amazon EKS Best Practices Guide provides detailed recommendations. GPU fractioning is particularly recommended when enforced per-Pod GPU memory limits are required on GPUs lacking MIG, and when the deployment utilizes fixed-size GPU node groups.
Dynamic Resource Allocation (DRA) offers Kubernetes APIs for requesting and allocating devices, with GPU sharing and isolation dependent on the specific device driver implementation and enabled features. Both KAI Scheduler and the GPU Operator support DRA. However, this walkthrough employs NvFractions annotations rather than DRA for configuration.
The benefits of GPU fractioning are particularly evident for interactive notebooks with periods of inactivity, short evaluation jobs, GPU compatibility tests, and small inference services. The recommended approach involves measuring peak usage plus overhead for each GPU memory request and then testing latency under concurrent load. While the demonstrated density improvement is notable, it does not inherently establish a throughput gain or a universal cost reduction.
Solution Architecture and Implementation
The validated single-GPU environment was configured in the US East (Ohio) Region (us-east-2). The setup utilized AWS Command Line Interface (AWS CLI), eksctl, Helm, and Packer for building and operating the environment. The Amazon EKS control plane, running Kubernetes version 1.36, resided within an AWS-managed Amazon Virtual Private Cloud (Amazon VPC), separate from the customer VPC, and connected to EC2 worker nodes in private subnets.
A system node, an m7i.large instance, hosted the operator controllers and KAI Scheduler. The KAI Scheduler was responsible for placing both Pods onto a single NVIDIA A10G GPU within a g5.xlarge instance, which possesses 23,028 MiB of GPU memory. Each Pod was configured with an 8 GiB GPU memory limit and successfully allocated 6 GiB. The GPU worker leveraged a custom AL2023 image equipped with the NVIDIA R615 driver branch, NVIDIA Container Toolkit, and containerd. It ran fractiond with Node Resource Interface (NRI) integration and MPS for sharing, while the metricsd daemon exported per-Pod GPU memory metrics.
The implementation walkthrough, found in the AWS Samples repository, requires specific prerequisites including permissions for Amazon EKS, AWS CloudFormation, Amazon VPC, Amazon EC2, Amazon Elastic Block Store (Amazon EBS), AWS Identity and Access Management (IAM), and AWS Systems Manager. Users must also ensure they have AWS CLI v2, eksctl, kubectl, Helm, Packer, Git, jq, and curl installed.
The process involves several steps: cloning the repository and configuring environment variables; creating a disposable Amazon EKS cluster and system node; building a custom AL2023 image with the necessary NVIDIA drivers; adding a GPU managed node group utilizing this custom image; installing the GPU Operator and inspecting the host configuration; establishing a whole-GPU baseline to demonstrate pre-fractioning scheduling constraints; installing KAI Scheduler and the GPU Fractioning Operator; running a positive sharing test to confirm concurrent workloads within memory limits; verifying the rejection of allocations exceeding the set limits; and finally, cleaning up all provisioned resources.
During the walkthrough, a custom AL2023 image was built to ensure compatibility with the GPU Fractioning Operator’s requirement for the R615 driver branch, as the standard EKS-optimized AL2023 NVIDIA image did not include it at the time of validation. The installation of the GPU Operator, version 26.7.1, provided essential Kubernetes integrations, including the device plugin, while preserving the image-owned driver and toolkit.
The baseline test demonstrated that before fractioning was enabled, only one Pod could run per physical GPU. Following the installation of KAI Scheduler and the GPU Fractioning Operator, two Pods, each with an 8 GiB GPU memory limit, were successfully deployed and ran concurrently on the same A10G GPU. Each Pod reported a successful allocation of 6,144 MiB (6 GiB) and metrics confirmed a used memory of approximately 6.24 GiB per Pod, accounting for overhead.
A critical test verified the enforcement of memory limits. A Pod attempting to allocate 9 GiB when its limit was 8 GiB was successfully rejected with an out-of-memory error, confirming the functionality of the memory contract enforcement and demonstrating allocation rejection rather than container termination.
Cleanup procedures involve capturing the VPC ID and then removing the test namespace, Helm releases, and the disposable cluster. While the custom AMI and its EBS snapshot are preserved by default, a separate command is available for complete removal of all created resources.
Production Considerations and Conclusion
For production environments, NVIDIA Run:ai GPU Fractioning is presented as a behavior validation. It necessitates the use of trusted workloads, adherence to least-privilege IAM principles, namespace controls, approved images, and admission rules for fraction requests. For scenarios demanding stronger isolation, the recommendation is to opt for whole GPUs or MIG on supported hardware.
The privileged nature of the fractioning daemons, which employ privileged containers, host PID access, and NRI/MPS host mounts, requires strict controls over their deployment and configuration, restricting them to trusted administrators. Careful consideration must be given to headroom for CUDA contexts, caches, batching, and runtime overhead. Performance testing, including latency and throughput under concurrent load, is essential, alongside ongoing monitoring of allocation failures, MPS health, NRI connectivity, and GPU errors.
Pinning the base AMI, driver, toolkit, device plugin, operator, scheduler, and workload images as a tested combination is crucial. Rebuilding images and conducting rollback tests before rollout are recommended practices. The current walkthrough focuses on pre-provisioned GPU capacity and does not configure or validate GPU-fraction-aware autoscaling, as fractional Pods selecting kai-scheduler do not request nvidia.com/gpu resources and target a node group fixed at one node. Support for secondary schedulers within Karpenter remains an open feature request.
The custom image used in this walkthrough is incompatible with Amazon EKS Auto Mode-managed nodes, which manage their own AMIs. While clusters can mix Auto Mode-managed nodes with managed node groups, this configuration was not validated. Teams using Karpenter can reference the custom AMI in an EC2NodeClass, but the autoscaling limit will still apply. The walkthrough did not validate this specific path.
When KAI Scheduler is adopted, organizations using Kueue must decide which component will manage queueing for GPU workloads. For monitoring, the Amazon EKS node monitoring agent is recommended on GPU nodes. When used with the custom R615 image and MPS sharing, the agent reported healthy GPU conditions. It’s advised to configure the agent to utilize the Data Center GPU Manager (DCGM) host engine already running via GPU Operator, rather than its bundled one.
In conclusion, this post demonstrates the configuration of GPU memory fractioning on Amazon EKS using a custom AL2023 image with the G5/A10G as a validated example. The approach can be extended to other compatible NVIDIA GPUs following prerequisite checks and configuration validation. The process involved establishing a whole-GPU baseline, running dual bounded workloads on a single GPU, inspecting per-Pod memory usage, and confirming the rejection of over-limit allocations. GPU fractioning offers a means to enhance access to existing capacity when workload memory requirements are met. Successful production adoption, however, hinges on implementing robust isolation controls and comprehensive performance testing.
Users can replicate this experiment by cloning the AWS Samples repository and comparing their results with the provided evidence. Further exploration of the GPU Fractioning Operator documentation and Amazon EKS accelerated AMI guidance is recommended when adapting the example to specific workloads.

The Chenab Times News Desk

