AMAZON BEDROCK / SERVICE DETAIL S09 EVALUATIONS
AGENTCORE EVALUATIONSPRACTICAL
CHAPTER 22 / 36
TOPIC add a grader, score your agent
REV 2026.07
IN PLAIN WORDS
You will add an automatic grader to your agent, run it on past conversations to get scores, and optionally let it grade new conversations as they happen. Terms: evaluator = the grader (a built-in one, or your own); on-demand = you run it now; online = it runs continuously; LLM-as-a-Judge = a model does the grading.
[ 1 ] PREREQUISITES
[x] An AgentCore agent with Observability on (it produces traces)
[x] AWS credentials, the AgentCore CLI
[x] Model access for the judge model (needed for custom evaluators; built-ins are fully managed)
[ 2 ] ADD A GRADER TO YOUR PROJECT
$ agentcore add evaluator # create a custom grader (built-ins are used by Builtin.* id, no add needed)
$ agentcore deploy # apply it
[ 3 ] GRADE NOW, OR GRADE CONTINUOUSLY
ON-DEMAND before you ship
$ agentcore run eval # grade past runs now (e.g. --evaluator "Builtin.Helpfulness")
$ agentcore evals history # view past results (agentcore logs evals to stream)
ONLINE after you ship
$ agentcore add online-eval # score new runs automatically
$ agentcore deploy
$ agentcore pause online-eval / resume online-eval # pause or resume it
[ 4 ] WHERE THE SCORES GO
Pick a built-in evaluator by id (for example Builtin.Helpfulness) or use a custom one. Scores appear in the CloudWatch GenAI Observability dashboard next to your traces, so you can chart quality over time and set an alarm when it drops. Online graders are managed as an "online evaluation configuration" (create, get, list, update, delete).
Tip: run an on-demand evaluation on every new version before release, then keep an online one running in production to catch regressions in real use.