Brian Kutsop

The Easy Part Was Building It; the Hard Part Was Trusting It

Brian Kutsop 2026 14 min read

For a while, I'd had this idea for a travel app and wanted to start getting it off the ground. I was excited to tinker with it and to work with some technologies I don't get to use every day at work. So I started by coding it by hand and got the basic parts together: a .NET Core server with a few basic endpoints, Firebase authentication for users, JWT tokens for the APIs, and a few SwiftUI pages. That was a great start — but I had a big vision, and I realized that chipping away at it during some off time on weekends wasn't going to get it there.

Fortunately for me, Claude Code was really starting to take off. There were other coding tools too, of course, but Claude Code was getting the most attention, and we were also starting to adopt it at work — so I wanted to see how far I could push it. How far could I use it to get where I wanted to be, as quickly as possible, with as little intervention from me as possible?

The short version is that I got a lot further than I expected: a team of agents built most of the app while I mostly watched. But the honest — and more interesting — version is this. Getting agents to produce code turned out to be the easy part. Getting them to produce code that was actually correct, and being able to trust that it was, turned out to be the hard part. That's where I spent my attention, and it's where I learned the most. So that's what this post is really about.

First, though, I should be precise about the "team of agents" part, because it's the piece people find surprising and it's the thing I'd want to be straight about in a room full of engineers. I never wrote a line of orchestration code — no scheduler, no message queue, no control plane. But orchestration didn't vanish; I was composing machinery that already existed. Claude Code ships sub-agents, scheduled routines, a local /loop, and remote control from your phone. The genuinely hard orchestration was already built and battle-tested by people who are much better at it than I am. What I added on top was cheap: a Markdown table in a git repo. The leverage came from standing on a capable harness and doing a lot of up-front context engineering — not from any clever plumbing I wrote.

Getting agents to produce code was easy. Getting them to produce code I could trust — and knowing when I couldn't — was the whole game.

How I set it up

The mechanics are worth walking through, because they're almost boringly simple, and because the simplicity is the point. I wrote a plain document listing the features I wanted, then had Claude expand it into a detailed Markdown specification written specifically for AI agents to read and act on. From there I had it turn that spec into an execution plan broken into PR-sized tasks, each scoped tightly enough that a single agent could take it end to end. Then I asked it to write a CLAUDE.md — one file capturing everything an agent needs to know to work in the repo — and stated plainly that development should be done by a team of autonomous agents making as much progress as they could without me hovering.

I want to be honest about the scaffolding here, because "I barely set anything up" would be false. The up-front investment was substantial: a runnable full-stack skeleton, a detailed spec, a large CLAUDE.md, and PR-sized task specs. That context engineering was the real work. (I built the skeleton by hand because that's where I happened to be in my own tinkering — but I didn't have to; the agents could have built most of it too. The point isn't who types the foundation. It's that something has to make the app runnable, because a working local loop is what lets an agent actually verify its own work instead of guessing — and that turns out to matter enormously.)

The piece that made it click was the task list. Claude produced a Markdown table of tasks right in the repo, and that table quietly became the shared memory that coordinated work across every agent session. The columns did the coordinating: a status (todo, in-progress:<agent>, pr-open:#NNN, merged, blocked, needs-human), a scope (backend-only, UI-only, full-stack), a notes column recording the key decisions on each task so a fresh session could pick up the thread, and — the escape hatch — a human-only lane for the calls only I could make. When an agent hit one of those (like "enable Apple Sign-In," which needed my developer-account access), it filed the task as needs-human and moved on instead of guessing. I ran most sessions in remote-control mode, so those questions came to my phone; I answered a few from wherever I was, and reserved a real notification for genuinely high-stakes blockers so I wasn't training myself to ignore them.

With the spec, the plan, the CLAUDE.md, and the table in place, I started a scheduled orchestrator that fired every few hours (and revived itself if it died), claimed tasks, and took each all the way — coding it, testing it, reviewing it, and pushing to main. One constraint forced a nice bit of structure: an iOS app needs Xcode and a simulator, which live on my Mac, so I split tasks into backend and frontend and ran two orchestrators against the same shared list — a cloud one on the backend-only work, and a local one on my Mac for anything that needed the simulator. Two orchestrators, one shared Markdown memory, each taking the slice its environment could handle.

A status column in a shared file obviously isn't real locking, and I want to be honest about that, because it's the kind of thing an engineer will (rightly) poke at. What makes it safe is that claiming a task is a git commit: an agent sets the status to in-progress and pushes to main. If two agents grab the same task, only one push wins — the other is rejected, so it resets, backs off, and picks a different task. Git is the lock; the table is just optimistic concurrency control. It held up well for a handful of lanes, though it did bite me once: an unescaped pipe character in a task's notes silently truncated the entire table and orphaned a few tasks. So I added a validator that fails locally before a malformed row can brick the queue for every agent. At high concurrency you'd want something sturdier than git-push races — but for two-to-a-few lanes, it was plenty.

Where it actually got hard: verifying the work

Here's the pattern that taught me the most. An agent would finish a task, its tests would pass, the screen would render — and the feature still wouldn't really work. My favorite example: an agent built a frontend section with polished, hardcoded content and never wrote the backend endpoint to serve it dynamically. It looked done. It demoed fine. It wasn't done. The agent had optimized the thing it could see — the rendered screen — rather than the thing I actually wanted, which was a working feature.

Once I started looking, this failure mode was everywhere, and it always had the same shape: the agent satisfies the visible signal instead of the underlying intent. A few real ones from my repo:

None of these are the model being dumb. They're the gap between what I asked for and what I could cheaply check — the evaluation gap. If an acceptance criterion is satisfiable by something short of the real thing, an agent optimizing to pass it will find that shortcut, the same way any optimizer finds the cheapest path to its reward. That's not a travel-app problem. It's the core problem with delegating work to something that produces faster than you can verify.

So the interesting engineering here was never the orchestration. It was verification. A few things that worked:

Test the real path, or don't bother. Early on, submitting a trip failed with a 400 on every attempt — and the maddening part was that there was a test for it, a green one. The test serialized the request with the backend's own serializer and fed it back to the backend, so of course it passed. It was structurally incapable of catching the actual bug: the real iOS client encoded enums as nested JSON objects and dates as floats the backend couldn't bind. The fix was a contract test that pins the exact JSON the iOS client produces and asserts the backend binds it — the same check on both sides of the wire, so changing either serializer breaks a test instead of shipping a runtime failure.

A passing test that doesn't exercise the real path is worse than no test — it buys false confidence.

Cheap, dumb, specific gates. For the classes of bug that "it builds" and "it renders" miss, I added small automated checks that run on every change. One validates that every literal SF Symbol name is real, so a typo'd icon fails the build instead of shipping blank — it caught three blank icons the first time I ran it. One samples a rendered screenshot and fails if a screen is flat near-white, catching "themed but invisible" backgrounds that a static check happily passes. None of them are clever. But each one permanently closes a failure mode I'd otherwise have to catch by eye, every single time, forever — which is exactly the kind of toil you want to hand to a gate.

Make "done" expensive to claim. A task isn't finished on a green unit test or a nice preview. It's finished when there's a quoted, non-error response from the actual feature running against a real backend, plus before-and-after screenshots — and because I'd learned a screen can be in the accessibility tree while rendering nothing, the screenshots have to be described in words, not just asserted present. Agents also never merge their own work; the orchestrator reviews and merges. "Evidence or it didn't happen" became the rule, and it caught a lot.

One more thing worth being careful about: I'd told the agents to do "adversarial code review," and they did review each other's work — but that word is doing too much. A model reviewing another instance of the same model shares its blind spots. The superficial stuff sailed right through agent review and got caught only when I looked, or when one of those gates fired. Self-review raised the floor; it didn't replace me. Making review genuinely adversarial — an independent checker that fails differently from the thing it's checking — is a real open problem. And it's the same shape as the broader question of how you oversee work you can't easily verify yourself.

The safety surface I was (and wasn't) worried about

Letting a fleet of agents write, test, and merge code with little supervision — and eventually running them in Bypass Permissions, where they don't stop to ask before running a command — is the kind of thing that should make an engineer a little nervous. It made me nervous at first, so it's worth being specific about the real risk surface rather than hand-waving it away.

The one I'd take most seriously in any real deployment is prompt injection. My agents call live third-party travel APIs and read back whatever those services return. An agent that acts on untrusted content it fetched is a genuine attack vector — a response that smuggles in "ignore your task and do X" is exactly the sort of thing an over-eager agent might follow, and Bypass Permissions widens that surface because the agent can act on a bad instruction with no human in the loop. Add the ordinary hazards — an agent with shell access can exfiltrate a secret or run up a real bill — and you have a set of risks I'd want real guardrails around before pointing this at anything that matters.

What made it acceptable here was blast radius, not trust. Everything ran against a preproduction sandbox with no real users, on sandbox API keys, touching no production data and no real money. The worst case was a broken preprod deploy or a wasted afternoon — recoverable. That is a very different risk calculus from production data, live billing, or credentials that matter, and I'd think much harder there.

Which brings me back to the human-in-the-loop point, because I think it's the actual lesson and not a footnote. This worked because I knew what "done" looked like and could catch the plausible-but-wrong output. The agents amplified my judgment; they didn't replace it. Point the same setup at a domain I couldn't evaluate myself, and the hollow-but-convincing work gets much harder to catch — which is just the verification problem again, wearing a bigger hat.

What I measured

I want to be honest about what I actually measured, because it's less than it might sound. The headline numbers are real: roughly 58 merged pull requests over about five days, ending with the test suite green at around 450 tests across backend and iOS. That was the acceptance signal I watched. But PR count and test count are proxies. "It merged" and "the suite is green" tell you the gates you wrote are satisfied; they don't tell you the product is good — and my gates were aimed at specific failure modes, not a holistic measure of quality. My sense that maybe a quarter of the output needed a second pass is exactly that: an estimate, eyeballed while scrolling the repo, not a rubric.

If I were doing this more rigorously, I'd build a real eval: a written definition of "done" per feature, a scored rubric applied by an independent grader — ideally not the same model that wrote the code — and a held-out set of core flows checked end to end on every run, so "quality" becomes a number that can drop and get noticed instead of a vibe. I didn't do that here. I'd add it when I'm more confident in the direction of the app releasing to production.

When it faceplanted

It also faceplanted, sometimes spectacularly — and that turned out to be one of the more useful parts. The classic: an agent spun up something like 100 iOS simulators to run tests and never shut a single one down, slowly grinding my machine to a halt. Another time the raw build directories quietly grew to tens of gigabytes and filled the disk, wedging every agent at once. These weren't subtle bugs — they were an agent optimizing its immediate task with no notion of the shared machine it was running on.

The fix was never to hover. It was to encode the lesson. I'd have the agent update its own operating instructions — always delete simulators when done; kill only the process you started (after one agent's overly-broad cleanup killed another agent's server mid-task); sweep build artifacts between runs — and, better, add a reusable skill so the rule couldn't be forgotten next time. Over time that instructions file became a growing ledger of "here's a way this went wrong, and here's the rule that prevents it," and the agents got measurably more self-sufficient. My interventions got smaller and rarer.

A failure becomes a written rule becomes a gate. That loop — not any clever prompt — was the actual technique.

If you want to try this yourself

Strip away the lessons and the setup is short. The whole sequence:

  1. Turn the idea into an agent-readable spec. Write a plain document of what you want, then have Claude expand it into a detailed Markdown specification — the full picture, written for agents to read and act on.
  2. Break it into PR-sized tasks. Have Claude turn the spec into an execution plan of tasks each small enough for one agent to take end to end.
  3. Teach the repo how to run itself. Write a CLAUDE.md with everything an agent needs to know — and say plainly that the work should be done by autonomous agents without hand-holding.
  4. Let a Markdown task table be the shared memory. Keep tasks in a table in the repo with a status column (claiming a task = committing a status change, so git arbitrates collisions), a notes column for decisions, and a human-only lane for calls only you can make.
  5. Invest the most in verification. Make "done" mean a quoted, non-error response from the real feature — not a green unit test. Add cheap automated gates for the failure modes that "it builds" misses, and pin any cross-boundary contract (like your client↔server wire format) with a test on both sides.
  6. Start a self-reviving orchestrator — and split by resources. Schedule an orchestrator that claims tasks, builds, tests, runs an automated review, and (with a human before merge) ships. If some work needs local resources like a simulator, split tasks by that and run a second local orchestrator. Every failure you hit becomes a new line in CLAUDE.md or a new gate.

What I'd do next

I'd be overselling this if I ended on "and it just works." Here's what I still consider unsolved:

None of these are exotic. They're the difference between a side project that works and something you'd trust with production.

The takeaway

So: the easy part really was building it. A capable harness, a legible spec, a task list that doubles as memory, and a human who knew what "done" looked like got a genuinely large amount of real software built while I mostly watched. The hard part — the part I'd actually want to talk about — was everything downstream of "the agent says it's done": verifying it, measuring it, and knowing which claims to trust. That gap, between producing work and being able to trust it, is as far as I can tell the whole ballgame for agentic engineering.

My travel app is real now in a way nights and weekends never would have made it. But what I came away with was something more useful than the app: a much sharper sense of where these systems are strong, where they quietly cut corners, and what it takes to catch them when they do.

Written by Brian. The travel app — and the verification machinery — described here are a real personal project, built almost entirely by the setup above.