AMAZON BEDROCK / SERVICE DETAIL S09
AGENTCORE EVALUATIONS
CHAPTER 21 / 36
IN ONE LINE an automatic report card for your agent
REV 2026.07
How do you know your agent is actually good, and stays good after every change? Evaluations grades it automatically: it takes the agent's recorded runs and scores them on things like "was this helpful?" and "did it use the right tool?", so you catch problems before and after you ship.
New to these words? Read chapter 02 "Key terms" (trace and span, session, model). See also chapter 19 "Observability".
[ 1 ] WHAT IT IS
Evaluations is an automatic grading service for agents. It reads the traces your agent already produces (the same step-by-step records from Observability) and scores them, so you get a clear, repeatable measure of quality instead of eyeballing a few chats. It works with Strands and LangGraph agents instrumented with OpenTelemetry or OpenInference.
THE PROBLEM IT SOLVES
"Looks fine" is not a measure. You need numbers to compare versions and catch a drop in quality.
HOW IT GRADES
An LLM-as-a-Judge: a model reads a run and scores it against a rubric (built-in or your own).
[ 2 ] HOW IT WORKS
STEP 01
RECORD
The agent runs and produces traces (via Observability).
==>
STEP 02
PICK SCORERS
Choose evaluators: built-in ones, or your own custom rubric.
==>
STEP 03
JUDGE
A judge model scores each run at session, trace, or tool level.
==>
STEP 04
SCORE
You get numbers you can compare across versions.
==>
STEP 05
WATCH
Scores flow into CloudWatch, where you can chart them and set alarms.
[ 3 ] TWO KINDS OF SCORER
BUILT-IN ready made
Pre-tuned scorers, named like Builtin.Helpfulness.
Cover common qualities: helpfulness, correctness, faithfulness to the conversation history, tool use, goal completion.
Public and fixed (you cannot change their rubric).
CUSTOM your rubric
Your own scorer with your own instructions and standard.
Private to you, controlled by IAM.
For quality checks specific to your use case.
[ 4 ] TWO TIMES TO RUN IT
ON-DEMAND before you ship
You run it yourself, on past runs or a test dataset.
Great for checking a new version before release.
Batch, dataset, and simulation evaluations exist as sibling types.
ONLINE after you ship
An online evaluation configuration watches new runs and scores them automatically as they happen.
Catches a quality drop in production, in real use.
Scores land in CloudWatch alongside your traces, so you can alarm on a drop in quality. Limits: up to 1,000 evaluation configurations per Region, 100 active at once.