In October we ran a study. We want to move our support operation off Jira Service Management and onto a portal of our own, and I wanted to know whether a team of agents could build that portal end to end, with one person supervising and nobody else typing. So we built it: Next.js and Supabase, English and Portuguese, an MCP server so agents can read tickets, an outbox that feeds email, Slack and webhooks. The orchestrator ran for 124 hours, spread over seven working days, and produced 612 commits and 97 merged tasks. I did not write a single line of code. I was the only person on the project.
I want to be careful with that paragraph, because it sounds like a vibe-coding story and it is the opposite of one. The portal was built by a handful of Claude Code sessions running under an orchestrator built on the first day, with a QA agent that rejected more than a third of what the dev agents handed in. This post is about that method. If you lead engineering, you will care about the orchestrator and the permission model. If you sign the contracts, you will care about the audit trail and the section on what I did not let the agents decide.
Summary
-
I did not write the plan. I was interviewed for it. Claude asked me questions with screenshots of our Jira open, until no decision was left to guess. The output is a 500-line document with every decision in a table, the data model with row-level security, and a graph of 54 tasks with owner, size and dependencies, grouped into waves of what could run in parallel.
-
The orchestrator is plain code, with no model in the control loop. It creates a git worktree per task, runs a dev session, then a QA session, and reopens the dev session with the QA flags when the verdict is fail.
-
The agent that builds a feature never decides whether it is done. QA runs as a separate process with different permissions. One blocker or major flag fails the task. Three fails in a row and I get paged.
-
QA used more agent time than development. 99 hours against 53, and that includes QA running the end-to-end suite on every task and on main after every third merge. That is the line item that lets me stand behind the merged code.
-
Every session and every verdict is logged, with its cost, attributable per task, per role and per day. Nothing merged without an independent check, and nothing shipped that I could not trace back to an acceptance criterion.
Step one: get interviewed instead of writing the plan
My instinct on a greenfield project is to open an editor and start describing what I want. I had tried that before and the result is always a document that reads well and leaves half the decisions to whoever builds it. This time I gave Claude the screenshots of our Jira setup and asked it to interview me until it had nothing left to ask.
The result is PLAN.md, and it is still the single source of truth for the product. Five parts of it carry the method:
-
Closed decisions, in a table. Ticket lifecycle, roles, how permissions work, what is in the MVP and what is not. One row per decision. If a question is not answered in the table, the dev agent picks the simplest option that satisfies the acceptance criteria, writes the choice down and keeps going. It does not stop to ask, because nobody is there at 3 a.m. to answer when the agent is coding.
-
The data model with security decided up front. Every table has row-level security. Every write to a ticket goes through a SQL function. Because this was fixed before any code existed, no agent ever had the chance to treat it as a later concern.
-
A task graph. Fifty-four tasks, each with a goal, acceptance criteria, a size, a lane, the files it owns, the shared files it may touch, and the tasks it depends on. This graph is what the orchestrator runs.
-
Waves. From the dependencies, the plan lays out which tasks can run side by side and which have to wait. Wave one is the scaffold. Wave two is the database schema and the design system, in parallel, because neither needs the other. Wave three is the ticket domain and authentication. By wave five there are seven tasks with no dependency between them, and the only limit is how many slots the machine has. The plan also names the critical path, scaffold to schema to ticket domain to ticket detail to internationalization to the final smoke, so the scheduler knows which task to start first when several are ready. The waves are a forecast, not a schedule. The orchestrator decides at runtime from what has actually merged.
-
Acceptance criteria that apply to every task. Typecheck, lint, unit tests and end-to-end tests tagged with the task id, all green. No interface text outside the translation layer. No table without row-level security. Every server action validated and role-checked. Every new screen rendering at 390 pixels without horizontal overflow. No file edited outside what the task owns.
For the people who approve projects rather than build them: the interview is the part of the method I would ask any vendor about. It is where the scope stops being a conversation and becomes a document that an engineer, or an agent, can be held to. The plan grew through five versions during the build, each one appending tasks for a new scope (theming and onboarding, internal finance requests, invitations, AI triage). The interview was repeated every time. The method never changed.
Step two: build the orchestrator before the product
Day one was not product code. Day one was the orchestrator, built in a normal Claude Code session following the plan, and tested with a dummy task where QA was told to fail on purpose. I wanted to watch a rejection and a fix go through before I trusted the loop with anything real.
The orchestrator is a Node process. It reads the task graph and keeps its state in SQLite. It picks the next ready task by critical path, so the task with the longest chain of dependents goes first. It creates a git worktree and a branch for the task, assigns a port, and starts a dev session with claude -p, streaming JSON so that cost and events land in the database as they happen. When the dev session ends, it starts a QA session in the same worktree. On fail, it reopens the dev session with the QA flags. On pass, it merges into main under a global lock, one merge at a time. Every three merges it runs the full end-to-end suite on main. Any session silent for fifteen minutes, or past its cost ceiling, gets killed and its task marked blocked.
It never edits product code. When it needs a judgment call, say to turn QA flags into a fix prompt, to decide whether a task that failed three times should be split, or to look at a merge conflict, it calls a read-only agent with Read, Grep, Glob and read-only git, and expects validated JSON back. If the JSON does not validate, there is a dumb fallback. That agent does not write code either.
I kept the scheduler free of any model on purpose. When a task ran at 2 a.m. instead of 4 a.m., I wanted to be able to say exactly why, from a log, without asking a model to explain itself.
Three roles, three sets of permissions
The piece that made the difference is not the prompts. The prompts are fine, and you could write worse ones and still get most of the result. What mattered is that the three roles run as separate processes that are allowed to do different things.
Dev agents have persistent identities by lane: data and backend, ticket experience, operations, and later platform and admin. Each task is a fresh session in its own worktree. A dev can edit, write and run the project’s commands. It cannot merge, cannot switch branches, and the orchestrator directory is on its deny list. When QA fails a task, the fix reopens the dev’s original session, so the reasoning behind the first attempt is still in context.
QA agents run one session per verification. QA may write in exactly two places: its own report, and the end-to-end test folder, in case the dev left an acceptance scenario uncovered. Using that second permission is itself recorded as a flag. QA runs the mechanical checks in order, including the Playwright suite for the task against a live dev server, then compares the diff against the task’s ownership lists, then looks for evidence of every acceptance criterion in code or tests. A criterion with no evidence is a major flag. The report is a JSON file with a verdict, the output of every command, the evidence per criterion, and flags with file, line, detail and a suggested fix.
The orchestrator agent reads and reasons. Nothing else.
Dev sessions do run the test suite before handing in, and most hand in green. QA still failed 55 of them. The failures were real: an acceptance criterion met in the interface but not in the server action, a translation key missing in one language, a shared component edited that the task did not own, a screen that overflowed on a phone. A single agent checking its own work would have shipped every one of those.
Separation of duties is not new; your finance team has had it for decades. What is new is that it is cheap to apply to software at the level of every single change. A prompt that says “review strictly” is a suggestion. A process that cannot write product code is a control.
What the fix loop looks like in practice
ready → in_dev → in_qa → merging → done
▲ │
└── needs_fix ◄── QA flags (3 cycles without a pass → blocked, a human decides)
Most tasks took one or two QA rounds. A few did not. The worst was the first real data task, T04, which went through eleven fix cycles before it merged. Somewhere around the fifth rejection it was obvious that the task was underspecified, not the agent. The cost ceiling is what stopped it from running all night. That task is why every task now carries a cap by size, and why the third fail pages a human instead of trying again.
The fix sessions are where a lot of the agent time went: 40 hours, against 53 for first attempts. I think of it as the price of not merging work that looks done. On a human team that cost exists too. It just does not show up as a line in a database.
The list of things that broke
The orchestrator’s list of problems is about as long as the list of reasons we want to leave Jira. Each one became a rule, and I would rather show the list than pretend the run was smooth.
-
One shared local database across all worktrees. A branch with an older schema would knock every lane over at once. The fix was migration-drift detection that sends a task to rebase before QA runs.
-
Infrastructure failures counted as code failures. A busy port or a lock contention would burn one of a task’s three fix cycles. Now QA stops at the first infrastructure error and reports
infra_error, which requeues the task instead of penalizing it. Forty-three of the 207 QA runs ended this way. A third of all rejections were my environment, not the agents’ code. -
Cost running away on underspecified tasks. A ceiling per task size, a watchdog, and
blockedinstead of infinite retry. -
Two agents editing the same file. Every task now declares what it owns and what it may touch. One translation file per feature. Generated types only from the data lane. The ticket detail page, the board card and the notification router are each a list where a task adds one line, and only the owning task edits the component itself.
-
Flaky end-to-end tests with the machine pinned at 100%. A sandbox per spec, and nothing writes to the seed data. The last hotfix of the build was exactly this.
-
The deploy platform only accepts commits from the account owner. The orchestrator now commits and merges with the repository’s own git identity.
None of these were visible before the first run. All of them would have stayed invisible without the orchestrator logging every session, every verdict and every dollar.
What I did not let the agents decide
This is the part I get asked about most by people who are not engineers, so it deserves its own section.
The agents never decided what the product should do. Every product decision was mine, made in the interview, written in the table. The agents never decided whether their own work was acceptable. The acceptance criteria were mine, and QA ran as a separate process. The agents never merged, never deployed, and never talked to a client. When a task got stuck, the orchestrator stopped and waited for me.
What the agents did decide was how to implement a criterion I had already written, and how to fix what QA found. That is a lot of autonomy, and it is also exactly the autonomy I would give a senior contractor with a clear brief.
So when someone asks who is accountable when an agent ships a bug, the answer is the same as before: the person who owned the acceptance criteria and the person who reviewed the verdict. The method does not remove that responsibility. It makes it legible, because every change points back to a task, a criterion, and a QA report with a file and a line.
What I would tell a team trying this
Write the harness before the code. The acceptance criteria and the ownership lists are what let five agents work in parallel without stepping on each other.
Lay out the waves before you start. Knowing which tasks can run side by side is what turns a graph into a schedule you can reason about, and the critical path tells the scheduler what to start first.
Keep the orchestrator dumb. Scheduler, worktrees, locks and merge queue are plain code. Save the model for judgment calls, give it read-only tools, and validate what it returns.
Separate the grader from the builder with permissions, not with prompts.
Tell infrastructure failures apart from code failures on day one. Otherwise your retry budget goes to your own environment.
Cap cost per task, and page a human on the third fail. The overnight run is only safe because there is a ceiling.
Do not try this with a product you have not decided yet. The method assumes the decisions are closed. If they are not, the interview is the step, and the agents can wait.
Closing thought
Douglas da Silva, our CTO, wrote in May that the model is not the bottleneck, the harness is. I believed it in the abstract. I believed it for real around the fifth rejection of T04, watching a capable model fail the same task over and over because I had written a bad criterion.
The portal exists because the harness held. Models have been able to write 71 thousand lines of TypeScript for a while. What I had not seen before was 97 tasks merging into one branch, in parallel, with one person supervising and nobody typing code, and the result deploying and passing its own test suite.
The study is done and the portal is in internal testing. Jira is still where our support runs today. The next step is the one the portal was designed for: a long-running agent that reads tickets through its MCP server, triages them with evidence from the client’s codebase, and leaves the decision to a person.
If you are weighing whether to let agents build something that matters to your business, and you want to compare notes on the harness rather than the demo, I would like to talk.