Evaluate agent decisions before go-live.
Find out whether a change to how your agent decides would actually help your users, using the conversations you already have. Add exploration for the strongest evidence.
$ uv add "agentope[llm] @ git+https://github.com/liminalplanes/agentope"1Check what's possible
2Import the traces
3Ask the model again
4Evaluate the change
$ agentope check --traces traces.otlp.json.gztool_choice · 12,000 decisions found in traces.otlp.json.gz Level 1 Logged ✗ no logged propensities Level 2 Sampled ✓ stored inputs: 12,000/12,000 Level 3 Estimated ✓ context has 3 features: customer_tier, has_order_number, intent Best available: Level 2$ agentope import --traces traces.otlp.json.gz -o decisions.jsonltraces.otlp.json.gz · 12,000 traces · 45,351 spans chosen 9,351 from execute_tool spans · 2,649 __none__ (no tool call)wrote decisions.jsonl$ agentope sample decisions.jsonl -o samples.jsonl --n 64 --yessampled 12,000 (decision, policy) pairs → samples.jsonl$ agentope eval decisions.jsonl --samples samples.jsonl --outcomes helpdesk.csv \ --horizon 3d --target policies.py:lookup_when_order_numberEvidence level L2 Sampled (N=64, α=0.5)Logging policy 0.637 realized mean outcomeTarget policy 0.711 SNIPS (IPS 0.755)Difference +0.074 95% CI [+0.058, +0.090]Verdict ok
Excerpts of a real run on simulated traces. The rule would have resolved +0.074 more tickets (95%: +0.058 to +0.090); the true effect, known because it's simulated, is +0.080.
What you can evaluate
A change to one decision your agent makes, with everything after it held as it was.
Which tool the agent calls first
lookup_order before search_kb
Which model a request goes to
short questions → small model
When to hand over to a person
escalate every pro customer
A new prompt or model at that step
system prompt v1 → v2
Not a fit… yet:A whole new agentSeveral changes at onceFree-text answer quality
How much you can trust the answer
It depends on what your traces contain. agentope uses the strongest evidence you have, says which, and lists what the estimate assumes.
- Strongest
L1Logged
A little exploration, recorded on the trace
- Strong
L2Sampled
Traces that include the messages
- Weakest
L3Estimated
Only what the traces recorded, once you agree to its assumption
Every answer ends with a verdict:okweakunreliable
Evidence levels in detailWorks with your stack
reads traces from
asks models through
The docs walk through the example step by step, with exploration, and compare what each evidence level gives on the same agent.