1
The gap report, failures included
The plan says what should happen; the traces, what did. A verifier that uses no AI lists, criterion by criterion, what was met and what was not. Nothing is hidden.
3 failures in viewreal · run v1.5
AI agent planner · demo A: medical prior authorizations
I planned, I built, and I measured the gap.
planlang writes an agent's plan as a contract that code can verify, builds the agent to that contract, runs it on cases with a known answer, and publishes the gap between what was planned and what happened, failures in plain sight.
How it works
Read as
Reading as leader. You see what planlang does and what its demos gave, in plain words.
Reading as expert. You also get how each claim holds up —an AI-free core, own traces with fingerprints, a cross-check between two languages— and the hook’s full source.
Goal
That an AI agent project is planned before it is built, and that the plan can be checked against what the agent did, case by case.
1I planned
The plan is written as a contract: every criterion with its measurement rule, every threshold with its signal. A validator rejects it if it is incomplete.
- DDecisions6
- RRisks10
- SAssumptions3
- CCriteria10
- UThresholds4
demo A planreal2I built
The LangGraph agent is built to that contract and runs on cases with a known answer and seeded adversaries.
enrutadorextractoraclaracionverificador_coberturadecisionpausa_humanaredactorguardia_salidathe sprint 3 agent · 8 of 8 piecesreal3I measured the gap
A verifier with no AI compares plan and traces and publishes the gap, with what failed in plain sight.
10 of 10 criteria met · 0 of 10 risks occurred
- S2 failed: two clarifications are enough
- S3 failed: several agents, slower than one
- The unforeseen failed: 3 failures only an evaluator saw
run v1.5 · 200 cases × 1real
Capacity measured with demo A
200 × 1cases per run, plus a single-agent baselinereal · run v1.5
1124 · 0decisions redone in another language · differencesreal · RF-09.2
10 of 10 · 0 of 10criteria met · risks occurredreal · report
0model calls when visiting this showcasedeclared
Expert
How each claim holds up
- AI-free core
- planner, verifier and playground call no model and read no clock or randomness (canonical JSON RFC 8785 + SHA-256); the playground, which runs in your browser, gives the same bytes in Node, Chromium, Firefox and WebKit
- Own traces
- planlang-trace/v1, each with its fingerprint; LangSmith is only an observability mirror
- Cross-check
- RF-09.2: the graph (Python) and the playground (TypeScript) evaluate the same plan rule; 0 differences in 1124 decisions across 2 runs
- Model
- sonnet through the author’s Claude Code subscription, in batches of 20 outside CI, with a switch to a keyed API
- Human decisions
- simulated in batch: the reviewer —the auditor in A, the officer in B— follows each case’s known truth (DA-04), and the showcase says so on every screen
- Stack
- LangGraph 1.2.12 · LangChain 1.4.2 · Python 3.12 · Claude Code 2.1.282 · Next.js static export · strict TypeScript
- Hook source
- LangChain, State of Agent Engineering (2025): 1,340 responses, Nov–Dec 2025; 89% have observability and 52% run evaluations. The survey does not ask whether what to evaluate was planned.
What no tool shows
All three are visible in both demos, on real runs.
2
Move a threshold and see what changes
Move one of the plan’s thresholds —in demo A, Texas mode too— and see, on the real runs, how many cases change path and how many go to a person; in demo A, also the minutes of review it costs.
3
The plan is the code of the edge
Every agent decision is declared in the plan as signal · operator · value, and a two-language test confirms the graph does exactly that.
The demos
Nothing that did not run is simulated here.
A · Medical prior authorizations
A router, an extractor, a rule-based coverage checker and a writer decide to approve, approve in part, deny with a stated ground, or escalate to a human auditor. No full denial goes out without a person.
Meets with warnings10 of 10 criteria · 0 of 10 risks · 3 failuresreal · verifier’s report
plan v1.5 · 200 cases × 1 + baseline · 2026-10-04real · sprint 3
B · Customer onboarding with due diligence
A router, an extractor, a rule-based watchlist checker, a context investigator for namesakes and a risk score decide to approve, reject or escalate to the compliance officer. No rejection goes out without a person.
Meets with warnings6 of 6 criteria · 0 of 4 risks · 1 failurereal · verifier’s report
plan v1.1 · run with v1.0 · 20 cases × 1 + baseline · 2026-10-04real · sprint 3
Also under construction
From the roadmap. None of it is simulated here.
- Under constructionConformal calibration and joint coverage
- Under constructionAnimated walk of one case through the graph
The 2026 interview question
How do you know your evaluation is trustworthy?
With a deterministic verifier over cases with known truth and seeded adversaries: no language model decides whether the agent got it right. And with the gap published even when it is unfavorable.