Measuring my own agent fleet

The problem

Cheap flash models burned their budget on prompts of about 90,000 tokens. Subagents stopped at the iteration cap with nothing written, and long sessions had no pass condition.

The cost ledger over-counted the real bill 17 times, and the Mac ran out of memory before I could see it coming.

What I built

Wrote a fail-closed model-lock proxy with an allow-list, daily USD caps per model and bucket, and cache-aware pricing.

Tuned the agent config with a measured reason for each key, and wrote five patches against the agent source.

Built a read-only measurement harness over the ledger, the proxy log and the state database, scored against twelve criteria.

Built Headroom, a native Swift menu bar app that counts running agents and shows how many more fit before the Mac swaps.

Wrote a preregistered benchmark for a managed agents API against a custom harness, with CI contract tests. It reports no winner yet, because no live paired trial has run.

How the work ran

Agents wrote much of this code under my direction. I set the measurements, reviewed each change and kept the numbers that held.

Result

The cost ledger went from 17 times too high to within 1.1 percent of the provider's own figure, over 265 requests.

Median prompt size fell from 89,965 to 32,707 tokens.

Budget-cap stops in the middle of a task went from several a day to zero.

My MCP catalog ships 56 servers plus an aggregator.