leiblerkullback is the open-source harness behind Leibler.
Your traces in.
An environment out.
Traces are the logs your agent already writes. Kullback reads them, rebuilds the tools, data and rules the agent used, and replays the logs to check the rebuild. The claim we are working toward, not made yet: a 2B parameter model post-trained in that environment performs on the real tasks.
The loop. Code decides pass or fail from the final data, not from the transcript. A judge model can take a pass away, never give one.
What a week of traces turns into.
- Data. The records your runs touched, with the values they saw.
- Tools. One function per tool, behaving the way the real one did in the logs.
- Rules. Your policy, turned into checks the code can run.
- User. A simulated user who knows only what the real one knew.
- Verifier. One per task. It looks at the final data and says pass or fail.
Every value points back to the line in your trace it came from. Nothing is invented.
What has been measured so far.
Measured on Sierra's public retail traces (tau2), where the real tools and database exist to compare against. Seen: runs used for the build. Held out: runs the build never saw.
| check | seen | held out |
|---|---|---|
| tool signatures | 0/15 | |
| starting rows | 0/252 | |
| writes | 0/37 | 0/20 |
| reads | 0/125 | 0/65 |
| errors | 0/6 | 0/4 |
Runs where a model had to stand in for a missing tool are shown in the report, never counted as passes.
All of it is public.
The code, the design, a decision log with every choice and the alternative it beat, 29 research reports, and the numbers measured so far. Apache-2.0. Pull requests welcome; the maintainer reviews every one.
One claim to earn.
A 2B model, post-trained in this.
Not claimed yet. Next: the same rebuild on domains it has never seen, then a 2B parameter model post-trained on trajectories the verifier passed, measured on the real held-out tasks. The numbers get published either way.