Amazon SageMaker Enhances AI Model Training with Multi-Turn Reinforcement Learning

Amazon Web Services (AWS) has introduced a new capability on Amazon SageMaker AI designed to train artificial intelligence models for complex, sequential tasks. The new Multi-Turn Reinforcement Learning (MTRL) feature addresses the limitations of traditional fine-tuning methods by enabling models to learn from the cumulative feedback of an entire sequence of actions, rather than just individual prompts. This advancement is particularly relevant for agentic AI applications that require a series of dependent steps to achieve a final objective, such as automated customer service or complex workflow automation.

SageMaker’s MTRL: A New Approach to Agent Training

According to details received by The Chenab Times, MTRL on Amazon SageMaker AI allows developers to train models by providing a single reward signal at the end of an interaction, reflecting the overall success or failure of a task. This contrasts with supervised fine-tuning, which requires labels for each step, and single-turn reinforcement learning, which struggles to attribute outcomes to earlier decisions. The system works by sending prompts to an agent hosted on Amazon Bedrock AgentCore or user-provided infrastructure like Amazon EKS, EC2, or AWS Fargate. The agent then interacts with an environment, guided by the policy model being trained, and provides a final reward signal to SageMaker AI for learning.

Wordle as a Testbed for Advanced AI Training

To demonstrate the capabilities of MTRL, AWS utilized the popular word-guessing game Wordle as a practical example. The game’s structure—where each guess depends on the feedback from the previous one and the ultimate goal is to correctly identify the word within a limited number of tries—makes it an ideal scenario for MTRL. This approach trains a model on the overall success rate, rather than mimicking specific guessing sequences.

In experiments, MTRL successfully improved the solve rate of the gpt-oss-20b model from 61% to 84% on unseen words when operating at its default reasoning effort. Even at a lower training effort, the model’s solve rate increased from 15% to 69%. The training process for this improvement cost approximately $339 in training tokens.

Technical Implementation and Architecture

The MTRL system comprises two main components: the SageMaker AI training job, which manages the training process and the policy model, and Amazon Bedrock AgentCore, which hosts the agent, the environment (in this case, the Wordle game), and the reward function. During each training rollout, the MTRL service invokes the AgentCore runtime, which then prompts the policy model through an OpenAI-compatible endpoint. The agent plays the game, using a tool to submit guesses and receiving feedback after each turn. Once the game concludes, the agent returns a single reward value, which is then used by the Group Relative Policy Optimization (GRPO) algorithm to update a Low-Rank Adaptation (LoRA) adapter on the policy model.

Key design principles include housing the agent, environment, and scoring function within a single Python file deployed to AgentCore. The training dataset consists solely of the secret word for each task, and the reward mechanism is outcome-dominant, assigning higher rewards for faster solves and penalties for losses or invalid guesses, with a small shaping term to incentivize good intermediate strategies.

Supported Models and Regional Availability

MTRL offers a serverless training experience, allowing users to select a base model, with SageMaker AI provisioning the necessary infrastructure. Currently supported base models include Amazon Nova 2 Lite, GPT-OSS-20B, Gemma 4 31B (instruction-tuned), and Qwen 3.6 27B. These models are trained with a LoRA adapter. The service is available in specific AWS Regions, including US East (N. Virginia) and US West (Oregon) for some models, and exclusively in US West (Oregon) for others.

Pricing and Prerequisites for Implementation

Users are billed based on token usage across three dimensions: prefill (input tokens), sample (generated tokens), and train (tokens processed during model updates). For the example run, which consumed over 1.29 billion tokens, the cost for training tokens was approximately $339, with an additional $79 for AgentCore runtime usage. The pricing structure is designed to be consumption-based, eliminating the need for users to manage instances or cluster sizing.

To implement MTRL, users require Python 3.10 or later, Node 20 or later with the AgentCore CLI, specific versions of the botocore library, and two AWS Identity and Access Management (IAM) roles. An Amazon S3 bucket in the same region as the agent is also necessary.

Step-by-Step Implementation Guide

The guide details a comprehensive process for setting up and running MTRL. This includes creating necessary IAM roles, deploying the agent to Amazon Bedrock AgentCore using the provided sample repository, and verifying the agent’s runtime functionality. The process involves cloning a GitHub repository, configuring deployment settings, and executing commands via the AgentCore CLI.

Data preparation involves creating a dataset where each row contains a task for the agent, typically structured as a JSON object. This dataset is then uploaded to Amazon S3. The training job is launched using the MultiTurnRLTrainer class from the SageMaker SDK, specifying model configurations, ARNs for the agent runtime and roles, dataset locations, and an MLflow application for monitoring metrics.

Monitoring during training occurs in MLflow, where key metrics like reward, token usage, and validation solve rates are tracked. The article highlights the importance of observing the val/reward/pass_at_1 metric. Post-training, an evaluation job is conducted to assess the trained model’s performance under serving settings, comparing it against the base model across various decoding parameters such as reasoning effort and sampling temperature.

Refinements and Best Practices in Reward Design

The deep dive into reward design emphasizes the importance of an outcome-dominant reward structure, distinguishing between a primary reward for task completion and a secondary shaping term for intermediate performance. Initial attempts using summed per-guess scoring led to suboptimal results, such as lost games receiving positive rewards. The revised reward system ensures that solves are consistently prioritized over losses, and faster solves are rewarded more than slower ones, while still using shaping to encourage learning from variance in trajectories.

The guide also addresses potential pitfalls encountered during development, including issues with token concurrency, ensuring rollouts successfully reach the policy model, and managing model drift into plain text outputs. It concludes with a recommendation to evaluate models using the same serving settings employed in production to accurately gauge their real-world performance.

The MTRL capability on Amazon SageMaker AI offers a powerful new avenue for developing sophisticated AI agents capable of handling multi-step, sequential tasks, extending the reach of generative AI into custom business workflows and internal procedures.

---

---

Follow us on

---

(We don't allow anyone to copy content. For Copyright or Use of Content related questions, visit here.)