Evaluate agent decisions before go-live.

Find out whether a change to how your agent decides would actually help your users, using the conversations you already have. Add exploration for the strongest evidence.

Get started
$ uv add "agentope[llm] @ git+https://github.com/liminalplanes/agentope"
  1. 1Check what's possible

  2. 2Import the traces

  3. 3Ask the model again

  4. 4Evaluate the change

terminal · support-agent traces
$ agentope check --traces traces.otlp.json.gztool_choice · 12,000 decisions found in traces.otlp.json.gz  Level 1 Logged     ✗  no logged propensities  Level 2 Sampled    ✓  stored inputs: 12,000/12,000  Level 3 Estimated  ✓  context has 3 features: customer_tier, has_order_number, intent  Best available: Level 2$ agentope import --traces traces.otlp.json.gz -o decisions.jsonltraces.otlp.json.gz · 12,000 traces · 45,351 spans  chosen         9,351 from execute_tool spans · 2,649 __none__ (no tool call)wrote decisions.jsonl$ agentope sample decisions.jsonl -o samples.jsonl --n 64 --yessampled 12,000 (decision, policy) pairs → samples.jsonl$ agentope eval decisions.jsonl --samples samples.jsonl --outcomes helpdesk.csv \    --horizon 3d --target policies.py:lookup_when_order_numberEvidence level   L2 Sampled (N=64, α=0.5)Logging policy   0.637  realized mean outcomeTarget policy    0.711  SNIPS   (IPS 0.755)Difference      +0.074  95% CI [+0.058, +0.090]Verdict          ok

Excerpts of a real run on simulated traces. The rule would have resolved +0.074 more tickets (95%: +0.058 to +0.090); the true effect, known because it's simulated, is +0.080.

What you can evaluate

A change to one decision your agent makes, with everything after it held as it was.

  • Which tool the agent calls first

    lookup_order before search_kb

  • Which model a request goes to

    short questions → small model

  • When to hand over to a person

    escalate every pro customer

  • A new prompt or model at that step

    system prompt v1 → v2

Not a fit… yet:A whole new agentSeveral changes at onceFree-text answer quality

How much you can trust the answer

It depends on what your traces contain. agentope uses the strongest evidence you have, says which, and lists what the estimate assumes.

  1. Strongest

    L1Logged

    A little exploration, recorded on the trace

  2. Strong

    L2Sampled

    Traces that include the messages

  3. Weakest

    L3Estimated

    Only what the traces recorded, once you agree to its assumption

Every answer ends with a verdict:okweakunreliable

Evidence levels in detail

Works with your stack

The docs walk through the example step by step, with exploration, and compare what each evidence level gives on the same agent.