Caylent · DevBench
Development agents write the code. Judge agents decide whether it merges.
DevBench is an open-source LLM-as-Judge orchestration system that drives a structured backlog (Epic, Feature, Story, Task) from claim to merged pull request. Every task is implemented test-first, reviewed by five judges, and gated on evidence written to disk.
All review decisions come from Claude LLM evaluation; there are no hard-coded pass/fail rules.DevBench README.md:3
The per-work-unit loop
next·claim: the orchestrator takes the next actionable Taskensure-branch: one branch per work unit by defaultexecutorruns strict TDD: RED → GREEN → REFACTOR, each phase loggedreview-supervisor: four review judges run in parallelsecurity-reviewer: sequential security gategit-ops: commit → push → PR → CI → mergemark-done: roll up to the parent, then loop
The done-gate refuses mark-done unless the most recent review round carries all five REVIEW_PASS verdicts, including security_review.
Choose your path
Five ways into the same system
The same architecture, told at the altitude you need it. Green means it exists in the open-source repository today; purple means it is a proposal.
Executive summary
Two minutes, no jargon: what the controls are in business terms, what cannot happen, and where the honest limits sit.
Read the summary → TodayTechnical overview
For technical leadership: what each gate enforces and how, where you keep the final call, and how a team widens autonomy as evidence accumulates.
Read the overview → TodayEngineering view
The complete technical reference: every agent, hook, guard, skill, status, configuration knob and CLI command, cited to source.
Read the reference → TodayArchitecture diagram
The interactive spec-to-merge map. Click any node or edge for what it is, what triggers it, what it produces, and where it lives in the repo.
Open the diagram → VisionEnterprise vision
What this proven core would look like as a multi-tenant internal SaaS: a service, a portal, an API. Clearly framed as proposal, not product.
See the proposal →The problem
Autonomy without evidence is just sprawl
AI coding assistants can generate change faster than a team can meaningfully review it. The failure mode is familiar: diffs nobody scoped, tests written to pass rather than to prove, and a merge decision made on vibes. The problem is not the model. It is the missing contract around it.
Nothing to review against
Without a declared scope for a change, "did it do the right thing?" has no answer. Reviewers are left comparing a diff to their memory of the request.
No durable audit trail
Chat transcripts are not evidence. When the question comes twelve months later (who approved this, on what basis), a scrollback is not an answer a regulated organization can give.
Nowhere to put a human
Supervision tends to be all-or-nothing: approve every single step, or trust the entire run. Without a dial in between, teams either drown in prompts or hand over the keys.
How DevBench answers it
One deterministic pipeline, from spec to merged PR
Five skills across two Claude Code plugins take an operator from a blank page to a running orchestrator. After that, the loop is file-driven and exit-code-driven; the backlog on disk is the only source of truth.
- create-spec
- spec-to-backlog
- configure-devbench
- bootstrap-environment
- make start
The operator's setup chain, as stated at docs/onboarding.md:7
-
Author the spec
The
devbench-authoring:create-speccreate-specskill turns discovery material into a rigorous engineering specification at the depth the decomposer requires: 16 canonical sections, or the operator's own exemplar. -
Decompose the backlog
Epic → Feature → Story → Taskspec-to-backlogwrites one Markdown file per work unit across four levels plus theBACKLOG.mdindex.validate-backlogruns at the top of every orchestrator loop iteration and aborts on any error. -
Claim and implement
The orchestrator claims the next actionable Task, ensures its branch, and hands it to the executor for a strict TDD cycle. Every phase is logged to the work unit through the CLI.
RED → GREEN → REFACTOR -
Face the judges
A read-only supervisor fans out four review judges in parallel (code, tests, docs, and declared-versus-actual changes), then the security judge runs sequentially. All five must pass.
4 parallel + 1 sequential -
GitOps to merge
Commit, push, open the PR, watch CI, merge under the configured strategy, mark done, roll up to the parent, loop. Nothing runs past merge: there is no deploy or release stage in the codebase.
commit → PR → CI → merge
Verified by design
Three independent layers stand between an agent and your main branch
Judges evaluate the work. Guards constrain what any agent can physically do. Gates refuse to advance the state machine until the evidence exists on disk.
Judges, not rules
- Four review judges (
code_review,test_review,doc_review,changes_manifest) run concurrently on every task. - A sequential
security_reviewjudge evaluates against SOC 2, PCI DSS, FINRA, SEC, GDPR, CCPA and SOX, and fails on any critical or high finding. - A security failure is not retried like a code failure. The security judge runs exactly once per work unit, and a failure blocks the work unit for an operator rather than re-entering the executor retry loop.
- Every verdict lands as a timestamped audit comment on the work-unit file itself.
Guards, not trust
- 21 hook command registrations across 10 Claude Code event types wrap the whole session.
- Seven
PreToolUseguards can block a tool call outright before it runs: destructive shell and git commands, staging files outside the declared Changes Manifest, direct writes to work-unit files, malformed verdicts, control-language in audit comments, and reviewer mutations. - A
PostToolUsehook blocks when a matched test command exits non-zero, so a failing suite cannot be narrated as a pass. - The review supervisor is enforced read-only, and may only spawn agents on the review-team allowlist.
Evidence, not prose
mark-doneis refused (with an error, not a warning) unless the latest review round carries all five passing verdicts.- Judges take their scope from
devbench get-diff, the authoritative source, never from a hand-rolledgit diff. - Rejections are emitted as structured JSON validated against a schema, and the executor must clear each one before the unit can complete.
- Subagent prose is never control flow. The loop branches on files and exit codes, so an agent cannot talk its way past a gate.
Humans in the loop
Autonomy is a dial, not a switch
DevBench is designed to run a backlog without human intervention between tasks, and to let the operator decide exactly where that stops being true. Every checkpoint below is configuration, not a code change.
Nine work-unit statuses drive the lifecycle. Three of them exist purely to hold a human decision (draft, proposed and hold), and a fourth, in-review, becomes a human checkpoint when pause_before_merge is on.
- The merge gate
- Auto-merge is off by default, so in batch mode a human merges.
pause_before_mergepushes the PR, waits for CI to go green, then parks the unit inin-reviewand moves on to the next actionable unit; a later sweep reconciles the merge viacheck-merge. - The intake gate
- Set new work units to
draftand they stay invisible to the orchestrator until someone promotes them. - The proposal gate
- When the recovery cascade generates new work,
auto_accept_proposals: falseroutes every draft throughpromote-proposalorreject-proposal. - Manual controls
holdandunholdto park a unit under debate,declinefor a permanent won't-do,set-statusto override, andscopeto fence off what the orchestrator may touch at all.- Escalation
- Blocked work is classified into exactly one of seven paths. Three of them always require an operator. The other four can clear themselves through a dependency cascade, a proposal cascade, a bounded recovery window, or an automated orchestrator restart.
- Notification and observation
- Sixteen notification event types (Slack today, all off by default), plus read-only surfaces:
reportfor progress and cost,watchfor right now,hook-tailfor the raw event stream.
Open source, and boringly durable
DevBench is licensed under the Apache License 2.0 and copyright Caylent. The backlog is plain Markdown files on a filesystem with a Markdown index: no database server, no queue broker in the data path.
Because state lives on disk rather than in memory, the orchestrator resumes from where it stopped after a restart, and every agent action is already written down.
What this core could become
The pipeline you can read in the repo today runs as a process against one workspace, on a machine you control. The enterprise vision asks a different question: what would it take to run this as a governed, multi-tenant internal service, with an orchestrator behind an API, a backlog with a portal, queued work across teams, and the same evidence trail under SSO and audit?
Everything on that page is written as a proposal, in proposal language, and is visually marked as such throughout.
Read the enterprise visionThe repository documents this system in pieces, and in a few places its docs and its code disagree. This site assembles the whole component map in one place for the first time, and the engineering view lists those discrepancies rather than papering over them.
Verified counts
Every number on this site is a count of something in the repository
No estimates, no projections, no benchmark claims. Each tile below was produced by enumerating files, schema keys, or registry entries in the open-source repository.
Want the citations? The engineering view carries the source paths, and the architecture diagram attaches one to every component.