AWS Simplifies Distributed AI Training with Ray Train DLC on EKS

Amazon Web Services (AWS) is streamlining the process of distributed artificial intelligence training for developers by integrating the Ray Train Deep Learning Container (DLC) with Elastic Kubernetes Service (EKS) and Elastic Fabric Adapter (EFA).

Streamlined AI Training Workflow

According to details received by The Chenab Times, a significant hurdle in scaling out PyTorch training to clusters has traditionally been the complexity of container setup. This includes ensuring compatibility across CUDA runtime, PyTorch builds, NVIDIA Collective Communications Library (NCCL), Elastic Fabric Adapter (EFA) libraries, and the AWS OFI NCCL plugin. AWS aims to eliminate this overhead with the Ray Train DLC, a pre-configured container image that bundles all necessary components, tested as a single unit and maintained by AWS.

The Ray Train DLC simplifies distributed training by providing a unified, pre-built environment that includes Ray, PyTorch, CUDA, NCCL, and EFA libraries. This eliminates the need for developers to build and manage complex container images themselves, allowing them to focus on model development and training.

KubeRay Operator for Kubernetes Management

On Kubernetes, the orchestration of Ray clusters is managed by KubeRay, an open-source Kubernetes operator. KubeRay introduces a custom RayCluster resource, enabling users to define head and worker groups within a single manifest. The operator then handles the creation and maintenance of the Ray cluster, including starting Ray in the appropriate role within each pod and ensuring workers can connect to the head node.

This approach ensures that a single, tagged image can serve as both the Ray head and worker nodes, whether deployed on Amazon EKS or Amazon Elastic Compute Cloud (Amazon EC2). Configuration for specific instance types, GPU allocation, network transport, and additional Python packages is managed externally, preserving the image’s universality.

Fine-Tuning Qwen2.5-1.5B as a Case Study

A recent demonstration showcases the effectiveness of this setup by fine-tuning the Qwen2.5-1.5B model using PyTorch Fully Sharded Data Parallel (FSDP) across eight NVIDIA L4 GPUs. The experiment utilized two g6.12xlarge instances within a single EKS cluster. The entire process required minimal custom code—just a single training function and a RayCluster manifest—without any custom image builds.

The integration of the DLC with the EFA stack ensures that inter-instance communications, crucial for FSDP, occur over EFA, significantly boosting performance. The complete setup and code are available in a sample repository, building upon the foundation established in a previous multi-node Ray Serve walkthrough.

Key Components of the Ray Train DLC

The Ray Train DLC consolidates numerous components that would otherwise require individual management:

• Distributed Runtime: Ray with its training libraries, compatible with KubeRay.
• Framework: PyTorch compiled for the included CUDA version.
• GPU Runtime: CUDA, cuDNN, and NCCL versions tested for compatibility.
• Networking: EFA installer, libfabric, and the AWS OFI NCCL plugin.
• Training Libraries: DeepSpeed, flash-attn, Transformer Engine, and Hugging Face libraries.
• Diagnostics: Pre-installed NCCL performance testing tools.

A critical aspect highlighted is the networking layer. Mismatches in NCCL and EFA configurations can lead to performance degradation, as NCCL may fall back to slower TCP sockets. The DLC ensures these libraries are built and tested together, with AWS validating multi-node EFA and KubeRay training.

Advantages of the Pre-built DLC

Beyond the integrated stack, the Ray Train DLC offers several operational advantages:

• No Registry Setup: The public image eliminates the need for credential management and image pushing.
• Driver Compatibility: Automatic handling of NVIDIA driver versions through bundled CUDA forward-compatibility libraries.
• Unified Image: A single image serves for both Ray head and worker roles, configured at runtime.
• Shared Repository: Seamless integration with other Ray DLCs for inference and LLM tasks.

Configuration Through External Settings

Customization for different training jobs is achieved through external configurations rather than image rebuilding. This includes:

• Role Assignment: Managed via KubeRay’s rayStartParams.
• Hardware Allocation: Defined by Kubernetes resource requests for GPUs and EFA.
• Network Transport: Controlled via environment variables like FI_PROVIDER and NCCL_NET_PLUGIN.
• Python Packages: Added at submission time using Ray’s runtime_env.
• Image Version: Specified via versioned tags or digests in DLC_IMAGE.
• Cluster Configuration: Set through variables in environment files for instance types, node counts, and region.
• Training Parameters: Passed as arguments to the training script, such as model ID and batch size.

This approach ensures that users can adapt the deployment to their specific workload needs without modifying the core container image.

Cluster Deployment and Job Submission

The deployment process is guided by a set of scripts that automate the setup of the EKS infrastructure and the submission of the training job. Key steps involve creating the EKS control plane, configuring GPU node groups with EFA support, installing necessary device plugins, and deploying the KubeRay operator. The training job itself is then submitted to the Ray cluster, with logs streamed directly and EFA usage verified.

The infrastructure setup takes approximately 20 to 30 minutes, with scripts designed to be idempotent and to detect existing resources. The sample employs a CPU system node for the EKS control plane, KubeRay operator, and Ray head, while EFA-enabled g6.12xlarge instances serve as GPU worker nodes, each running a Ray worker pod that claims all available GPUs and the EFA interface.

Adapting for Various Workloads

Modifications for different workloads are typically confined to a few configuration files. Changing the model, training parameters, or adding Python packages requires minimal adjustments. The system also supports switching to different GPU instance families, such as p5 instances, without altering the core training code or the DLC image, provided they support GPUDirect RDMA and are compatible with the EFA device plugin.

Resource Management and Cleanup

The cost-effective deployment involves careful management of GPU instances, which are the primary expense. Scripts are provided for tearing down all deployed resources in the reverse order of creation, ensuring that cloud infrastructure is efficiently managed and costs are controlled.

Future steps for users include integrating with data storage solutions like Amazon S3, pinning DLC versions for reproducibility, enabling failure recovery configurations, and scaling to larger models and different GPU hardware families.