Use decision models with Amazon Bedrock AgentCore Evaluations

Some of the questions you ask when evaluating an agent are narrow and repeatable. Did this response stay on policy? Is this tool call safe to approve? Does this answer match the retrieved context? Decision models answer exactly that kind of question. They take a structured state and a fixed set of allowed answers and return a calibrated probability over them, rather than free-form text whose confidence drifts from run to run. That makes them a strong fit for the scoring step in an agent evaluation, where you want a label and a number you can threshold and trust.

Amazon Bedrock AgentCore Evaluations, a capability of Amazon Bedrock AgentCore, scores an agent through its entire lifecycle, on demand during development, in batch jobs against stored sessions, and online against production traffic. Alongside 15 built-in evaluators, AgentCore Evaluations supports custom evaluators that you define as either an LLM-as-a-judge or a code-based evaluator. Custom LLM-as-a-judge evaluators apply your instructions and rating scale with a model in Amazon Bedrock. Code-based evaluators run your own logic in an AWS Lambda function, and that logic can call your choice of external system, model, or API, including a decision model.

Decision models suit questions with a fixed set of answers that an informed person could decide at a glance, such as whether the agent refused a request or whether a response stayed on topic. After you validate a question on your own traffic, a decision model can be sufficient for that decision or score, and you can extend the same approach to specific harmfulness or stereotyping categories in your policy. Because both backends can answer several questions about the same state in a single request, you can use them as a screening layer rather than a replacement, running a pass or fail signal across your metrics to select the turns that warrant deeper assessment by the built-in evaluators, whose explanations help you investigate what went wrong.

This post shows how to connect two decision model backends to AgentCore Evaluations through code-based evaluators. The first is Jev, a hosted decision model from TypeSafe. The second is Strands Decider 2B, an open-source decision model you run yourself. By the end of this post, you will know how to convert the AgentCore evaluator input into a decision model request, and the model’s probabilities back into an AgentCore evaluation result. First, though, when should you reach for a custom LLM-as-a-judge evaluator, and when does a decision model fit better? The next section breaks down what a decision model is and the trade-offs involved.

Decision models

A decision model is a small, specialized model trained to select from a fixed set of options and return calibrated confidence scores rather than generating free-form text. A general-purpose large language model (LLM) evaluator produces a score with a rationale. A decision model instead produces a probability distribution over predefined answers. That gives you a direct confidence measure you can threshold, aggregate, and act on programmatically. This makes decision models well-suited for evaluation tasks where the judgment can be stated as a choice among a fixed set of options: “Is this response grounded?” or “Did the agent complete the user’s goal?”

Decision models are most useful when you want a well-calibrated confidence score alongside the label, not just a binary judgment. They are also appropriate when you want to enforce a specific rubric consistently without relying on a general-purpose LLM to interpret it.

Jev, the hosted decision model from TypeSafe, and Strands Decider, an open-source, 2-billion-parameter decision model from the Strands Agents team, both support three types of questions. A noul question poses a yes/no question and returns a probability between 0 (no) and 1 (yes). A choice question asks which of up to 255 options applies, returning the most probable option along with the full distribution and a confidence. A score question asks which of 2 to 10 ordered levels applies, returning a probability-weighted score, the distribution over levels, and a confidence. Many evaluation rubrics fit one of these types. “Is every fact in this response supported by the tool results?” is a yes/no question, and “How helpful was this response, on a four-level rubric?” is a score.

Decision models versus LLM-as-a-judge

The built-in evaluators in AgentCore today use a general-purpose LLM to assess the agent response against a rubric and explain its judgment. This is valuable when the evaluation requires connecting evidence, identifying missing information, or working through a response in depth in tasks where the explanation is as useful as the score. Decision models are not a replacement for LLM evaluators. They’re an addition that the code-based evaluator framework makes possible. Use them for narrow, well-defined checks alongside the built-in evaluators’ broader assessment.

Solution overview

To show how these evaluator types work together, this post builds an end-to-end example of an HR Assistant agent evaluated by deterministic code-based checks, built-in LLM-as-a-judge evaluators, and decision-model evaluators backed by Jev or Strands Decider 2B.

The agent is built with Strands Agents and deployed to AgentCore runtime, a capability of Amazon Bedrock AgentCore. It answers employee questions using five tools that return fixed mock data such as paid time off (PTO) balances, PTO requests, HR policies, benefits, and pay stubs. We use fixed data in this code sample to make it straightforward to verify evaluator behavior from LLM-as-a-judge and decision models. When you additionally have ground truth, you can confirm whether each evaluator catches the right failures. This code sample helps you set up the framework for testing decision models compared to LLM-as-a-judge for your own use cases.

Figure 1: End-to-end architecture of the HR Assistant evaluated by code-based checks, built-in evaluators, and decision-model evaluators backed by Jev or Strands Decider

Prerequisites

To deploy the sample, you need the following:

• An AWS account in a Region where AgentCore Evaluations is available. The sample defaults to US West (Oregon).
• Amazon CloudWatch Transaction Search enabled in the account.
• Access to a foundation model in Amazon Bedrock.
• Python 3.12 or later, the AWS Command Line Interface (AWS CLI), and Docker.
• Jev requires a TypeSafe API key. Visit typesafe.ai for access details.
• Strands Decider requires a host running the Strands Decider server. You can run it locally for testing, or on Amazon Elastic Compute Cloud (Amazon EC2) inside your virtual private cloud (VPC) so the Lambda function can reach it.

Seeing it in action

You can find the full code sample in the AgentCore samples GitHub repository. Two Python scripts handle everything:

• ../utils/deploy.py deploys the HR Assistant to AgentCore runtime.
• evaluate.py deploys evaluators, invokes the agent, and runs the evaluation modes.

Running evaluate.py without flags executes five steps:

1. Creates the Lambda execution role (AgentCoreLambdaEvaluatorRole).
2. Packages and deploys two deterministic Lambda evaluators: hr-response-length (a trace-level evaluator that checks whether the agent’s response stays within length bounds) and hr-fact-checker (a session-level evaluator that regex-validates HR facts against the known mock data).
3. Registers both as AgentCore code-based evaluators through the control plane.
4. Runs on-demand evaluation with EvaluationClient and OnDemandEvaluationDatasetRunner across five labeled scenarios, combining the code-based evaluators with Builtin.Correctness, Builtin.Helpfulness, and Builtin.ResponseRelevance.
5. Creates an online evaluation configuration that continuously scores live sessions at 100 percent sampling.

At this point you have a working evaluation pipeline with deterministic checks and LLM-as-a-judge scoring. Decision-model evaluators are opt-in. You can add them with a flag, and they run alongside the base evaluators with no changes to the existing flow. Using the flags --with-jev --jev-secret-arn deploys jev-evaluator and registers three evaluators: JevGroundedness, JevHelpfulness, and JevGoalCompletion. Using the flag --with-decider deploys strands-decider-evaluator and registers the same three evaluators backed by Strands Decider 2B. Add --decider-ec2 to auto-provision the Amazon EC2 instance, security groups, IAM role, and Lambda VPC configuration without manual console steps.

Figure 2: The evaluation pipeline deployed by evaluate.py, from deterministic checks through built-in and decision-model evaluators

Each decision-model backend contributes three evaluators:

Build the evaluator with decision models

Using the flags --with-jev and --with-decider creates a custom code-based evaluator, where you have complete control of the code you want to run as your evaluator. We create AWS Lambda functions where the decision model is invoked inside an AWS Lambda function and returns a label or a numeric value. This Lambda function is registered as the code-based evaluator in AgentCore Evaluations.

The Lambda that is registered as the evaluator receives OpenTelemetry spans of the agent. The spans.py module normalizes the shapes OpenTelemetry content can arrive in and then formats it appropriately to make the call to the decision model. For trace-level evaluation, only the targeted turn and brief prior history are sent, whereas the full conversation is included for session-level.

Figure 3: How the code-based evaluator Lambda adapts OpenTelemetry spans for the decision model and maps the result back to AgentCore

The adapter converts the model’s probability distribution into a result that AgentCore evaluation expects. For a noul question, the probability of true becomes the value and the most probable answer becomes the label. For a score question, the value is the probability-weighted expected level scaled to [0, 1]. For a choice question, it is the weighted average of predefined option values.

Validate a decision-model evaluator on your own traffic

When you rewrite a rubric as a decision model question, you’ve created a new measurement rather than a copy of the old one. A reframed rubric often lands stricter or looser than the original intended, and you won’t see that drift until you check it against real sessions. Before you gate a release on a decision-model evaluator, validate it the same way you’d validate any new measurement, against a labeled sample of your own traffic. The goal of this section is to walk you through that validation end to end, so you finish with an evaluator you can threshold with confidence.

The procedure has five steps.

Step 1. Build a labeled sample. Pull a set of stored sessions from your agent’s traffic, with enough coverage of the failures and edge cases you care about. A sample that covers only the happy path tells you nothing about where the evaluator breaks.

Step 2. Get human reference labels. Have three independent labelers score each turn for the property your question measures and take the majority vote as the reference label. Keep track of which turns were unanimous and which were split. Although unanimous turns are the clear-cut cases, the split-vote turns are the valuable ones, because they reveal ambiguity in the rubric itself before the model ever sees it.

Step 3. Run both evaluators on the same inputs. Run the decision-model evaluator and the matching built-in evaluator over those same sessions in a single batch evaluation. Scoring identical inputs is what makes the comparison apples to apples.

Step 4. Read the results through three lenses. With the majority-vote labels and the easy and hard split in hand, you can now answer the three questions that tell you whether the evaluator is ready.

• Agreement rate: does it match your humans? Measure how often each evaluator agrees with the majority-vote labels, and do it per question, since one question’s accuracy says nothing about another’s. Then break the numbers out by easy (unanimous) and hard (split-vote) cases. An evaluator that stumbles only on the hard cases is behaving a lot like your human labelers did, which is exactly what you want.
• Calibration: do the confidence scores mean something? Group turns by the evaluator’s reported probability and compare each group’s agreement with the labels. Higher confidence should track higher accuracy. Then check whether easy cases consistently draw higher confidence than hard ones. If they do, the model’s uncertainty is tracking genuine ambiguity, and you can use those groups to pick a decision threshold per evaluator instead of defaulting to 0.5.
• Disagreement analysis: where and why do they differ? Find the turns where the two evaluators disagree with each other, and check whether those are the same turns your humans found ambiguous. Read them. Most disagreements trace back to the wording of the question, and a revised question clears them up. When the errors cluster in the hard cases, the rubric is the problem rather than the model.

Step 5. Re-run the check when anything changes. Treat this as a repeatable check rather than a one-time gate. Any time you reword a question or move to a new model version, run it again. Each result’s explanation records the model version that produced it, so you know which version a given score came from.

Choosing the best set of evaluators for your agent

The built-in evaluators remain the foundation of broad evaluation strategy in AgentCore Evaluations. They assess the response against a rubric and its supporting context, and explain the judgment so developers can investigate what went wrong. For the HR Assistant, a faithfulness evaluation can identify a claim that contradicts a tool result, whereas a helpfulness evaluation can point out that the agent answered a balance question but left a requested action incomplete.

Faithfulness, helpfulness, and correctness can require connecting evidence, identifying missing information, or working through a solution. The preprint JEV-as-a-Judge: Accept When Confident, Escalate When Unsure describes a related distinction as “reading versus deriving.” A narrow decision-model check can flag a turn for attention, whereas the built-in evaluator provides the broader assessment and reasoning needed to review it. Decision models shine brightest when you treat them as a gate rather than a replacement. The cascade pattern works like this. First, run a fast, calibrated decision model on every turn of every session to get a cheap pass/fail signal. Then invoke a full LLM evaluator only on sessions where the signal falls below your threshold.

Figure 4: The cascade pattern, where a calibrated decision model screens every turn and only low-confidence turns escalate to a built-in evaluator

We recommend the following setup:

1. Configure online evaluation with the built-in evaluators you rely on, as the standard measure of quality across your agents.
2. Add decision-model evaluators for domain-specific yes/no and choice questions to the same configuration, along with deterministic code-based evaluators for anything you can compute exactly.
3. Each week, or on whatever schedule suits your release cycle, query the online results for sessions where a decision-model check failed or scored near its threshold, and run batch evaluations on those sessions with the built-in evaluators to get explanations.
4. Gate releases on the built-in evaluators and on the decision-model evaluators you have validated against labeled data, and track how often the two agree.

Clean up

To avoid ongoing charges, run the cleanup.py that deletes the agent hosted on AgentCore runtime, the evaluators and evaluation config, the AWS Lambda functions that are registered as the evaluators, the EC2 instance that hosts the Strands Decider model, AWS Secrets Manager key and the associated log group on CloudWatch.

Conclusion

In this post, you connected two decision models, Jev and Strands Decider, to AgentCore Evaluations using code-based evaluators, and the accompanying code sample gives you a framework for testing those models and seeing where they add value as evaluators. The built-in evaluators remain your baseline. Decision models add narrow, validated checks that come with a measure of certainty you can threshold and act on. The practical path forward is to add decision-model checks alongside the built-in evaluators you already run, measure how often they agree with your reference labels on your own traffic, and expand coverage only once you trust them.

To get started, review the Amazon Bedrock AgentCore Evaluations documentation, integrate custom evaluators using code-based evaluators and take a deeper look into Strands Decider model. Get started with code samples by deploying the sample evaluators from the Amazon Bedrock AgentCore sample repository.

About the authors