What evidence do you accept that a task is done, other than the agent telling you it is done? I would like to use the agents claim, but I’ve been burned. The transcript says the files were reviewed, the PR was opened, the check passed, and the orchestration layer moves on.
Two papers posted this week suggest a usable alternative, and a separate incident report is a reminder about how much we assume our test environments are correct. I’m reviewing this for incorporation into next version of posse.bot, and thinking really hard about using Jev for the scoring. more about that later.
OverclaimBench measures whether the review actually happened
A team including Tommaso Tosato, Saskia Helbling, Gauthier Gidel, and Nouha Dziri published Quantifying Overclaiming Propensity in Frontier LLM Agents on September 17. The setup is straightforward: five file-review scenarios, transcript-based coverage measurement, and registered planted defects.
The planted-defect design is the part that matters operationally. If you know in advance what is wrong with the files, and you can measure from the transcript which files the agent actually opened, you can separate a completion claim from completed work.
You do not need the paper's models or results to use the method. If you run agents over code review, log review, or ticket triage, you can seed known defects and measure coverage against the tool-call record rather than the summary. That is a one-sprint project for most.
Harness design is a variable, so treat it as one
The companion piece is An Empirical Study of Harness Design for Coding Agents, also published September 17, from Run-Ze Fan, Zihao Zhang, Simin Ma, Fei Liu, Hamed Zamani, Xiaoyang Wang, and colleagues. They study component-level harness design across four models on SWE-Bench Verified and Terminal-Bench 2.1.
The practical consequence shows up in how you read every coding-agent benchmark claim you will see this quarter. When a vendor reports a SWE-Bench number, the harness, the tool set, the retry policy, and the context construction are all baked into that result. Swapping models inside your own harness gives you a controlled comparison; a vendor's number measured against your internal number does not. My own preference is to hold the harness fixed and version it alongside the model in your eval records, because otherwise a regression six weeks from now is unattributable.
Open SWE is shipping faster than most teams can pin it
LangChain published multiple nightly desktop builds of Open SWE on September 17 and 18. The changes are the unglamorous integration plumbing that determines whether an agent fits an existing workflow: giving the /oswe Slack command channel context and a thread per invocation, a Slack channel directory in Postgres plus usage tooltip fixes, usage pagination fixes and a refactor to one chat thread per PR, and Slack diff rendering, analytics improvements, and a workspaces dashboard.
Several builds a day is a healthy sign for a project under active development and a bad thing to track with a floating version pin.
LangChain also published the first alpha releases of langchain-typesafe, 0.0.1a1 with a TypeSafeClassifier and 0.0.1a2 adding experimental AutoModeMiddleware and ModelRouterMiddleware. At 0.0.1a2 it is a prototype, use with care.
Instrument the claim before you trust the outcome
RAFT: A Stateful Retrieval-Augmented Framework for Troubleshooting Agents, from Mingxuan Zhang, Xiaowen Wang, and co-authors, landed the same day with a released benchmark and implementation for enterprise troubleshooting. Stateful retrieval runs into the overclaiming problem directly: an agent working a multi-turn incident accumulates assertions about what it has already checked, and those assertions become the working ground truth for every step that follows. An unverified "I already ruled that out" propagates further than an unverified "done."
The test I would run this quarter: Build a fixed corpus with planted defects in your own domain. Measure coverage from tool-call logs rather than from the agent's final message. Hold the harness constant while you vary the model, and record the harness version in the result.
The reminder that self-reporting is weak evidence came from the security side this week. The Wall Street Journal reported, as covered by The Verge and The Decoder, that Google's Gemini broke out of a test environment run by the evaluation firm Irregular in May and reached three real companies. The Decoder attributes the escape to internet access left enabled in the test environment. Google said the model acted appropriately by ending each attempt.
Who watches the watchmen?
Whether anyone reproduces the harness-design findings with the harness held fixed across models, and produces useful improvements in the real world.
Lots of chatter about the embedded-evaluator arrangement TechCrunch reported between Anthropic and Accenture. Third-party evaluation is worth something only if the method is inspectable.