Two weeks ago I wrote that my harness was bad and that we were making progress. Still true on both counts. The progress cost me, and there is still some bad in there, but less, and I know where it is now.
For anyone who did not read the last one: I’ve done what a lot of people are doing right now and built a small open-source thing, posse, that runs a team of AI agents against a shared to-do list.
Each agent has a persona file — a name, a job, a list of things it may never do, its own memory. A loop hands out work, gives each agent its own copy of the code, and folds the result back in when the item closes. One agent is the chief of staff and keeps the others moving. I watch from one screen. Inside their lines the agents get real freedom. Money, publishing, anything live, and the persona files themselves stay with me.
It went public on August 23. Since then the team has landed something like 1,750 changes, and for most of two weeks it ran itself while I handled deploys, permission files, and the handful of calls only I could make. That part works. So what did we learn?
The toe-stubs
#1 please dont kill the laptop you run on.
An agent finishing a test run wanted to stop a stray process and did what I would have done: killed everything whose command line looked like its own. On a laptop where seven agents are running the same command out of seven folders, that is everybody. We went back through every session on the machine. It had happened 138 times, and eleven of those took out another agent’s test run. One afternoon two agents killed each other’s runs three minutes apart, which I would find funnier if I hadn’t paid for both. Asking nicely in the prompt did nothing. Putting “never kill by pattern” in every persona file did.
Related, and my fault as much as anyone’s. To get an urgent fix moving, the chief of staff started an agent in my own working folder instead of handing it a copy. The fix was fine. But the tool files an agent’s memory by folder, so that agent’s notes to itself went into my notes. When I finally looked, my project memory had 114 entries from about a hundred sessions and almost none of them were mine. Every agent gets its own copy now, always, even when that is slower.
#2 monitoring the spend
The loop has a spend guard. It reads how much of my plan I’ve used and stops hiring near the ceiling. Twice the reading failed — once an expired login, once a rate limit — and both times the loop took “no reading” to mean “no limit” and kept hiring past a ceiling it had seen a few minutes earlier. The second time it hired four agents in one pass. The first time I only caught it because I happened to look at my own usage page. The rule is written down now: the last good reading stands until a fresh one arrives, and blind near the ceiling means stop. The code change is on the list. In the meantime the chief of staff reminds me when we need a refresh, which is not the same thing, and I know it.
#3 testing out of control
Every time the QA agent found something it wrote a test to keep it fixed, which is what I asked for. Then the tests started checking the wording of comments, the text of build scripts, and other tests. One was written around a live bug, so fixing the bug broke the test. By September 9 there were 1,120 of these, and 328 of the 818 files added in two weeks were that kind. Both outside reviewers put this at the top of their lists without talking to each other. One of them: “A census that cannot fail except by editing the comment it holds is not a test.” Same reviewer, on what was happening: “the immune system treating every verify finding as a new organ.” I made two rulings. A finding whose fix would not change what the software does is a note, not a work item. And a new test of this kind has to answer one question before it lands — what behaves differently if you delete it. That put a stop to most of it.
#4 watch the secrets in your pushes.
A push to the public repo got blocked because a made-up test credential looked too real. The chief of staff retried every few minutes to see if I had cleared it. When I did, the retry went straight through and landed seventeen changes, twelve of which had not been through the scan we run before anything leaves the building. They were clean. That was luck, and the rule says so now: scan the whole batch before every retry, or wait until I say go.
What the outsiders said
The crew runs almost entirely on Claude, Fable and Opus. On September 9 I gave the same review job to two agents on Grok and Astra — no team, no memory, code read-only. It took an afternoon. They did not see each other’s work. Some of what came back I would rather not quote, so here it is.
The first one: “Posse has become substantially better at protecting its own operations, and substantially larger than its stated job.” Its one-line verdict was “better and busier, with busyness outrunning demonstrated product benefit.” It also found two real bugs at the edge of the work queue — one where an agent could read the wrong project’s list, and one where closing a mistyped item closes a different one. Both are on the list.
The second: “This is not a thin harness.” The intro docs still describe a small loop over two off-the-shelf parts; on disk the loop alone is 5,500 lines, and since the last release the product has doubled and the tests more than doubled. “The last fourteen days bought a thicker wall, not a simpler product.” On the docs: “NOTES.md is not a map; it is a memoir.” And: “I would not ship current main as a release to anyone who is not already this shop.”
Hard to argue with any of it. Two things I noticed, though. They agreed on what to keep, which I’ll get to. And both of them hit the same rule about what can be written where, and stopped, instead of working around it — neither had seen the rule before. A model I don’t run, reading the code cold, obeyed a fence it found there. I’ll take it.
So what worked?
Every agent works in its own copy of the code and hands the result back through the loop. A “never” in a persona file beats an “allowed” anywhere else. Both reviewers put that first on the keep list; one called it the load-bearing decision and also the reason the thing is as big as it is, because keeping agents apart has a long tail. Both true.
What I kept for myself held. No agent spent money. No agent published a word under my name. Nothing live changed without my say-so on that specific change. Every stumble above was a line the team didn’t know it had crossed, not a bad call it made on purpose, and each one stopped when the line became something the software enforces instead of a sentence in a prompt.
The team can delete when it measures. We have a good idea now of what can and should go, and the direction to tighten is clear.
And the review itself was worth doing. If you run a team of agents and haven’t had an outside model read it cold, do it. Cheapest audit you’ll get.
What I would do differently
Run the outside review on day seven, not day seventeen.
In my defense watching the agents build and run is fun and addicting. but not useful.
Next
Lot of work to get to a shapeable, portable release. I’m motivated, because I have several ideas I want to launch a swarm at, with different models. Better tuning, better default personas. Quite a bit less aggressive on the tests.
I’m on a very old version of beads. That needs addressing soon.
Move the whole harness off the laptop and give it real testing infrastructure that scales. Not having that cost me half a weekend.
I’m still keeping an eye on superlogical.com as a target, but herdr.dev has some seed capital now, so maybe the presentation layer is fine where it is.