kayba-ai/autoharness

Contribute to kayba-ai/autoharness development by creating an account on GitHub.

Summary

autoharness is a Python control plane that autonomously improves an existing agent harness by proposing and testing changes against a benchmark. It edits prompts, config, middleware, and source in a target repo, runs evals, and keeps or discards candidates based on the scores. State, proposals, and champion runs are persisted under .autoharness/.

What it is useful for

autoharness is useful for engineers who maintain an agent harness and want a reproducible way to optimize it without hand-tuning. Instead of guessing whether a prompt rewrite or middleware tweak actually helps, you define a benchmark command and let autoharness iterate: it generates candidate changes, runs the benchmark, and promotes the winner.

It also helps standardize optimization runs. The README documents six adapters (generic_command, pytest, harbor, tau2_bench, hal, car_bench) so the same tool can drive a shell command, a pytest suite, or a domain benchmark. Proposal generators range from manual and failure_summary to model-backed ones (openai_responses, codex_cli, claude_code), with a Python plugin model for adding more under .autoharness/plugins/ or AUTOHARNESS_PLUGIN_PATHS.

A documented limitation: the README explicitly notes that "some intervention combinations can regress" and that results depend on the benchmark, harness, and evaluation setup. On a fresh install, if neither codex, claude, nor OpenAI credentials are available, autoharness falls back to the non-LLM failure_summary generator, which narrows what kinds of proposals it can produce.

How engineers can use it

The README documents a straightforward install via pipx:

pipx install "git+https://github.com/kayba-ai/autoharness.git"
cd your-harness-repo
autoharness guide

Inside a TTY, guide asks a few setup questions and writes autoharness.yaml plus a benchmark config. In CI or scripts, the same step accepts flags like --non-interactive, --benchmark-command, --generator, and --autonomy. The README also shows an assistant-assisted path: autoharness guide --assistant codex --print-next-prompt (or --assistant claude) produces an autoharness.codex.md onboarding brief and a ready-to-paste prompt for the assistant.

Once guide finishes its doctor pass, the common workflow is:

autoharness run-benchmark
autoharness generate-proposal
autoharness optimize
autoharness report

optimize runs a resumable search loop over candidate proposals; report summarizes iterations and champions. If you switch the generator to openai_responses, export OPENAI_API_KEY first. For flags, retention, plugin hooks, and the campaign worker model, the README points to docs/quickstart.md and docs/usage.md.