ReplicantEsper: What Building with AI Agents Has Actually Taught Me

This is probably the first part of an ongoing series. Esper is still evolving, I’m still changing my mind about things, and several of the lessons in this post are corrections to things I’d have written with complete confidence a month ago. These aren’t best practices, and they definitely aren’t instructions. They’re observations from building real features, watching the workflow fail in interesting ways, and then trying to understand why.

Most writing about multi-agent AI systems focuses on collaboration.

The idea is simple enough: give agents different roles, let them share information, point them at a problem and watch what happens. The diagrams usually have lots of boxes and arrows. Quite often there are arrows between arrows.

I’ve spent a good chunk of this year building exactly that sort of system.

It’s called Replicant Esper, and it’s the workflow I now use to build features for both ReplicantCore and Everblossom, my storytelling app.

What surprised me is that after using it on real features, almost none of the interesting problems turned out to be about collaboration.

Getting agents to talk to each other turned out to be relatively straightforward.

The harder problems were deciding when they shouldn’t talk to each other, how to stop them modifying things they shouldn’t, how to prove a check genuinely passed, and how to tell when a safety mechanism has quietly stopped working.

In other words, the difficult part wasn’t collaboration.

It was governance.

What Esper Actually Is

The short version is that Esper sits above the code rather than inside it.

A feature starts with a request. The lead agent turns that request into a plan, breaks the work into blocks, routes implementation to builders and then sends the results through reviewers before anything can be considered complete.

At a very high level, the workflow looks something like this:

flowchart LR
    A[Lead plans a block] --> B[Builder implements]
    B --> C[Reviewer audits]
    C -->|changes requested| B
    C -->|approved| D[Lead runs checks]
    D --> E[Update records and tasks]
    E --> F[Commit or generate manifest]
    F --> A

That’s obviously a simplification. There are a few more moving parts in reality, but it’s close enough to understand the shape of the system.

One decision I’ve become increasingly happy with is that Esper lives outside the product repositories.

The workflow has its own workspace. It doesn’t install instructions into Everblossom, doesn’t commit agent configuration files and doesn’t leave process artefacts behind. As far as the repository is concerned, the workflow was never there.

I didn’t realise how important that would become when I started, but it means Esper adapts itself to the codebase rather than forcing the codebase to adapt to Esper.

Oh, and it’s also vendor agnostic. Want your devs using Claude and PM using Codex? no problem.

Everblossom - The Ideas Shelf

The feature that taught me the most wasn’t especially large or complicated.

Everblossom already has a guided story creation flow. One step asks users what they want the story to be about.

Some people immediately know.

Some people stare at the box for thirty seconds and then decide they’ll come back later.

If you’ve ever looked at a blank Word document and suddenly forgotten every thought you’ve had in your entire life, you’ll know the feeling.

The app already knows plenty about a family’s stories. It knows their characters, previous adventures, family memories and recently read books. It seemed odd to ask people for inspiration while ignoring all of that.

The result was something I called the Ideas Shelf.

Underneath the story prompt box, a parent can:

• Generate a random story starter
• Continue a previous story
• Build a story around existing characters
• Draw inspiration from family memories
• Add extra details such as dragons, pirates or underwater adventures

Every option still writes into the same editable text box. The parent remains in control. The shelf is there to reduce “blank page syndrome”, not replace creativity.

The feature ended up touching eight files and generating somewhere around 1,300 lines of code.

Most of it worked first time.

The interesting part was everything it taught me about the workflow.

Spoiler alert - It needs a bit of a clean, but this is the built and working feature in the app that Esper orchestrated.

Lesson 1: Independent Review Is More Important Than Collaboration

One of the earliest mistakes I made was assuming reviewers and builders should talk to each other directly.

It seems sensible. Real teams discuss work all the time. Why wouldn’t AI agents?

The problem is that explanations are persuasive.

Imagine a reviewer spots something questionable and asks why it was implemented a certain way.

The builder replies:

I used a local mapping here because changing the server contract felt out of scope.

The reviewer thinks:

Fair enough.

And suddenly they’re no longer reviewing the code. They’re reviewing the explanation.

Once I noticed that failure mode, I started treating the builder-reviewer relationship differently.

Builders and reviewers still exchange information, but they do it through records rather than conversation. Findings get written down. Responses get written down. The reviewer performs their first review without inheriting the reasoning that produced the implementation.

That small constraint improved review quality far more than I expected.

It also led to an interesting side effect.

Several times during the Ideas Shelf feature, agents disagreed about how something should behave. Both proposed solutions sounded reasonable. Both turned out to be flawed.

The eventual answer wasn’t either proposal.

It emerged from comparing the two.

That’s happened often enough now that I’ve started treating disagreement differently. When two agents disagree, the disagreement itself is often the useful output.

Lesson 2: Safety Systems Fail More Quietly Than You Think

The most useful thing I’ve discovered in Esper wasn’t really an AI problem at all.

Builders aren’t supposed to commit code.

They’re not supposed to leave their assigned repositories.

They’re not supposed to rewrite the requirements they’re being measured against.

I’d written protection rules for all of this.

I’d written tests.

Everything passed.

Then I installed the workflow on a clean machine.

On a default Windows setup, the hook script responsible for enforcing those boundaries wasn’t loading because of the local execution policy.

Nothing crashed.

Nothing complained.

Nothing warned me that the protection layer wasn’t running.

The workflow simply carried on.

If you’ve ever had that sinking feeling where you realise something has been broken for weeks rather than minutes, you’ll know what happened next.

A week later I found a second version of exactly the same problem on a different platform. This time a missing dependency meant another safety check couldn’t evaluate its input. Instead of denying access, it quietly allowed everything.

Different bug.

Same lesson.

A guard that can’t evaluate its input should deny access by default.

Anything else creates a situation where failure becomes indistinguishable from permission.

Since then I’ve become deeply suspicious of safety systems that haven’t failed in front of me at least once.

Lesson 3: Autonomy Should Remove Questions, Not Brakes

One of the first real features I ran through Esper came back with a ridiculous number of questions.

The workflow had successfully decomposed my request into dozens of smaller decisions and then handed them all straight back to me.

Technically it was asking sensible questions.

Practically it wasn’t doing much useful work.

The solution was simple: if a decision is reversible, within scope and can reasonably be inferred from the available information, the workflow should make the decision and record the reasoning.

That reduced interruptions dramatically.

Interestingly, the genuinely important questions didn’t disappear.

On the Ideas Shelf feature there was one decision that couldn’t be resolved technically: should emotionally negative memories appear as story inspiration for young children?

That’s not an implementation question.

It’s not even really a design question.

It’s a product judgement.

The code can’t answer it.

No amount of agent discussion can answer it.

Somebody needs to decide what sort of experience they want the user to have.

I’ve gradually become less interested in removing humans from the process and more interested in making sure they’re only asked to weigh in on the things that actually require judgement.

The goal isn’t maximum autonomy.

The goal is reducing unnecessary interruptions without removing the important brakes.

Lesson 4: Records Matter More Than Memory

One of the smallest changes I made to Esper ended up being one of the most valuable.

It now preserves my original request exactly as written.

Not the proposal, the design. or the requirements document - The original messy request.

At first that seemed slightly pointless because the structured version is always clearer.

Then I started looking back through completed changes.

Interpretations drift, requirements evolve, assumptions creep in.

Without the original wording it’s surprisingly difficult to identify when the plan has gradually wandered away from what was actually requested.

The same lesson appeared somewhere else too.

I have also started building a small desktop application to visualise Esper’s records. Nothing clever. A task board, a log viewer and a few status screens.

TODO//ESPER DESKTOP SCREENSHOTS

Within a single afternoon it found multiple problems in the format.

Not because the tool was particularly smart.

Because it was the first thing that had ever tried to read the output.

That experience produced one of my favourite notes from this whole project:

A format you only ever write is untested.

Lesson 5: You Still Need Humans

The biggest failure in the Ideas Shelf feature wasn’t found by a reviewer, a tester or the workflow.

I found my me on my phone during a device check.

One section of the Ideas Shelf works with story characters The problem was that Everblossom already has a previous step where users choose which characters will appear in the story. The feature had been designed against an older version of the flow.

The code was correct, The reviewer approved it, The requirements matched the implementation, All the checks passed - but the feature was still wrong.

The problem wasn’t code, it was context

A design decision that made perfect sense in one flow no longer made sense in another, and nobody in the workflow had any reason to challenge it because every document agreed.

A human (i.e Me) spotted it almost immediately

What I’m Actually Learning

When I started building Esper, I thought I was experimenting with multi-agent collaboration and thought it would be cool to get a bunch of agents to work together.

The longer I work on it, the more it feels like an experiment in actual software engineering (remember that?)

The difficult problems aren’t particularly new. They’re questions about verification, accountability, safety mechanisms, review processes and audit trails.

Teams have been dealing with versions of these problems for decades - the agents just make them visible faster.

The other thing I’ve noticed is that the list of things I know are untested seems to get longer rather than shorter.

Every time I become confident about one part of the workflow, I discover three assumptions hiding underneath it.

Multiple builders working in parallel, cross-vendor implementations, larger review structures.

Sections of the workflow that exist on paper but haven’t yet seen enough real use to earn my confidence.

That’s probably a good sign.

The bugs that worry me most now are rarely the obvious ones. They’re the assumptions I’m not questioning because they currently appear to be working.

If Esper has taught me anything so far, it’s that AI doesn’t remove the need for engineering discipline - If anything, it makes it more valuable.

Once a system is capable of producing large amounts of work automatically, questions of evidence, accountability and trust become harder rather than easier.

That’s turned out to be far more interesting than getting a handful of agents to talk to one another.

It’s still cool though 😎