Amazon Bedrock AgentCore Enhances Agent Evaluation with Decision Models

Amazon Web Services (AWS) is enhancing its Amazon Bedrock AgentCore capabilities by integrating decision models into its AgentCore Evaluations framework. This development aims to provide more precise and reliable scoring of AI agents by leveraging specialized models designed to answer specific, repeatable questions with calibrated probabilities.

The Chenab Times has learned that decision models offer a distinct advantage over general-purpose large language models (LLMs) in evaluation scenarios where a fixed set of answers is required. Unlike LLMs that produce free-form text, decision models are trained to select from predefined options and deliver a numerical confidence score, making them ideal for tasks such as policy adherence checks, tool call safety assessments, and context matching.

Understanding Decision Models in Agent Evaluation

Decision models are particularly well-suited for evaluative questions that have a clear, finite set of possible answers. These can range from simple yes/no queries, like whether an agent refused a request or stayed on topic, to more nuanced assessments such as rating helpfulness on a defined scale. The output is a probability distribution over these options, offering a trustable confidence measure that can be programmatically thresholded and acted upon.

AWS has introduced two decision model backends that can be integrated with AgentCore Evaluations via custom code-based evaluators: Jev, a hosted decision model from TypeSafe, and Strands Decider 2B, an open-source model developed by the Strands Agents team. These models can handle three types of questions: ‘noul’ (yes/no with a probability), ‘choice’ (selecting from up to 255 options with probabilities), and ‘score’ (assigning one of 2 to 10 ordered levels with probabilities).

Integrating Decision Models with AgentCore Evaluations

AgentCore Evaluations, a feature within Amazon Bedrock AgentCore, supports agent scoring throughout its lifecycle. It includes 15 built-in evaluators and allows for custom evaluators, which can be either LLM-as-a-judge or code-based. The integration of decision models is achieved through these code-based evaluators, which run custom logic in an AWS Lambda function. This function can then invoke external systems, including decision models.

The process involves converting the input data from AgentCore Evaluations into a format that the decision model can process, and then mapping the model’s probability outputs back into an AgentCore evaluation result. This allows developers to incorporate the precise judgment of decision models alongside the broader assessments and explanations provided by built-in LLM evaluators.

Comparing Decision Models and LLM-as-a-Judge

While built-in evaluators often use general-purpose LLMs to assess agent responses and provide explanations, decision models serve a complementary role. They are not intended to replace LLM evaluators but to augment them for specific, well-defined checks. LLM-as-a-judge evaluators are valuable for tasks requiring evidence connection, identification of missing information, or in-depth response analysis where the explanation is critical. Decision models, conversely, excel at providing consistent, calibrated confidence scores for narrow, deterministic judgments.

Solution Overview and Implementation

AWS provides an end-to-end example demonstrating the integration of decision models with AgentCore Evaluations using an HR Assistant agent. This agent, built with Strands Agents and deployed on AgentCore runtime, answers employee queries using tools that provide mock data for PTO balances, HR policies, and pay stubs. The sample code allows developers to set up a framework for testing decision models against LLM-as-a-judge evaluators.

The implementation involves deploying deterministic Lambda evaluators for tasks like response length checking and fact validation against mock data. These are registered as code-based evaluators in AgentCore. The system can then run on-demand evaluations combining these with built-in evaluators like Correctness, Helpfulness, and Response Relevance. Decision models, such as Jev and Strands Decider, can be optionally added to the evaluation pipeline using specific flags. These decision models contribute evaluators for groundedness, helpfulness, and goal completion, each operating at either the trace or session level.

Validating Decision Model Evaluators

To ensure the reliability of decision-model evaluators, AWS recommends a rigorous validation process. This involves building a labeled sample of agent traffic, obtaining human reference labels from independent scorers, and running both the decision-model evaluator and the corresponding built-in evaluator on identical inputs. The results are then analyzed by examining agreement rates with human labels (differentiating between easy and hard cases), checking calibration of confidence scores, and performing disagreement analysis to identify and rectify any issues in the rubric or model interpretation.

Choosing the Right Evaluators

AWS suggests a cascaded approach for agent evaluation. Built-in evaluators serve as the foundation for broad quality assessment. Decision-model evaluators are then added for domain-specific, precise checks, and deterministic code-based evaluators handle exact computations. This setup allows decision models to act as an initial screening layer, providing a quick pass/fail signal. Only sessions that fall below a certain confidence threshold are then subjected to the more comprehensive analysis by LLM evaluators, which provide detailed explanations for deeper investigation.

This strategy allows for efficient evaluation by prioritizing detailed analysis for potentially problematic interactions, thereby optimizing resource usage and developer time. The goal is to build a robust evaluation pipeline that combines the speed and precision of decision models with the explanatory power of LLM-as-a-judge evaluators.

---

---

Follow us on

---

(We don't allow anyone to copy content. For Copyright or Use of Content related questions, visit here.)