An autonomous experiment loop for AI skills and codebases. It changes one thing, measures it against a held-out split, and keeps the change only if the difference is larger than what the eval set can actually resolve. Everything else gets rolled back. Works with Claude Cowork and OpenClaw, fully autonomous or step by step.
Optimize SKILL.md files or any codebase. Run fully autonomous or step through each decision.
Improve a Skill's SKILL.md by running evals, grading outputs, and scoring with a gate score. Works with Claude Cowork and OpenClaw skills.
Improve any file against any shell command that returns a number: test coverage, bundle size, Lighthouse, Docker image size, lint errors.
Fully autonomous loop. Start it, go to sleep, wake up to a morning report with measurable improvements.
Interactive loop with 5 checkpoints where you review, adjust, and approve each experiment before it runs.
In guided mode, you stay in control while the agents do the heavy lifting.
See generated test cases, adjust assertions, change weights, add your own edge cases.
The agent proposes what to change and why. Accept, modify, or provide your own direction.
See exactly what will change before it's applied. Edit, approve, or skip.
See the score before and after. Override the automatic recommendation if you know better.
See progress so far and decide: another round, N more rounds, or generate the final report.
Each experiment cycle is driven by five specialized agents. Four run every round; the archivist every fifth.
Analyzes failures, consults the coverage matrix, and produces a testable hypothesis targeting the highest-impact root cause.
Applies one minimal, focused change. Works on SKILL.md instructions or code files. Documents every change with category tags.
LLM-as-Judge evaluating output quality across three dimensions. In Generic Mode, the shell command provides the score directly.
Assembles context for every agent call, picks the exploration/exploitation phase, collects near-miss hypotheses, and keeps history, coverage matrix and checkpoint in sync.
Every five experiments, distils what kind of edit actually works for this particular skill. Writes to the optimizer's own notes, never into the target skill. Each bullet needs an experiment id, each previous bullet gets a verdict.
Every mutation is measured against a gate score. No vibes, no gut feeling — just numbers.
The gate score is the assertion pass rate, so it moves in steps of 1/N. With nine assertions a single flip moves 0.111; a fixed threshold of 0.02 sits below that and turns every flip into a KEEP. The keep threshold is therefore at least two flips wide, and the loop tells you the number before it starts: “resolution 0.065, changes below this are not measurable with this eval set.”
NEUTRAL rolls the mutation back. A loop that keeps the new version on a zero round drifts away from its starting point without a single measurement to justify it. Deltas just below the keep threshold carry a near_miss flag, so the next round can vary the same hypothesis instead of dropping it. Efficiency (tokens, wall clock) is still measured and reported in the morning report, but it no longer gates anything.
Three runs under the v2 decision rules. Read the caveat below the numbers.
These runs predate v3. They used the old gate, where efficiency carried 20 percent of the
score, and the old decision cascade. The repository contains no evals.json,
history.json or snapshot for any of them, so none of it is reproducible from
here. The 0.74 to 0.90 figure is a composite score under the old weighting; the 87 and 93
percent figures are raw assertion pass rates. Results from v3 onwards ship with their
history.json and evals.json, or they do not get published.
Safeguards for a loop that edits your files unattended, all night, with nobody watching.
The optimal mutation for flake8 src/ | wc -l is to delete src/. Before every generic-mode experiment the loop pins the metric config, the test files, the file count and the byte size of what it measures. Without a passing check it gets INVALID instead of a number.
Put anything between FORGE_KEEP markers and the loop will never change it. A second region holds notes that bypass the gate. Both are checked byte-wise in Python, not by asking the agent whether it behaved.
Every experiment snapshots the state before it, with a manifest. A rollback restores the recorded files and removes the ones the mutation added, from any working directory.
The gate score is the assertion pass rate, so it moves in steps of 1/N. The keep threshold is at least two flips wide. A fixed 0.02 would sit below the resolution of any real eval set and turn every single flip into a KEEP.
train feeds hypotheses, val decides keep or revert, test is touched exactly twice: once for the baseline, once for the report. Assigned by a stable hash, so deleting an eval does not reshuffle the rest.
Five new hits against three new failures is a plus two in the aggregate and looks like progress. The report pairs the same evals across both versions and lists regressions first.
Before looking for a cause, every failure is classified: was there already a rule that would have prevented this? If yes, that produces a note, not a rewrite. Otherwise one subagent slip costs a correct rule.
A token budget spans the skill plus everything the loop wrote beside it. Over budget, the next round is forced into pruning.
Three consecutive crashes pause the loop. If the process dies between mutation and decision, the checkpoint knows and rolls back before continuing.
Five independent reviewers went through the code, and a mutation test over 69 targeted changes hunted for tests that would not notice a real bug. Every finding is pinned in the suite.
Skill Forge uses the standard SKILL.md format and runs wherever AI skills are supported.