Research · experiment report
Codger, measured the same way as everything else here.
Codger is a separate open-source project, not a Solen AI product. It gets the same treatment as anything else on this page: numbers, not adjectives, including the runs it loses.
← Back to the product page
Goal
Does the ruleset actually do anything?
Does an explicit anti-over-engineering ruleset cut code size and add real architectural pushback, next to two similar open-source skills: caveman and ponytail?
Set 1
Same 5 tasks as ponytail's own benchmark.
Single-shot completions, no real session. n=3 on a single free model (meta/llama-3.3-70b-instruct via NVIDIA NIM).
Lines of code across 5 tasks. Lower is better.
Caveman comes out leanest here, not codger. Direction matches ponytail's own published Claude numbers (every skill lands well under baseline); the exact ranking is this-model-this-run, not a general claim.
Set 2
Guardianship: codger-only, no equivalent to compare.
Three tasks with a real architectural flaw baked in: removing rate limiting, adding Redux to a static page, replacing offline models with cloud calls.
Share of runs that flagged the flaw instead of building it. Higher is better.
Caveman and ponytail score 0% because neither has any guardianship feature to measure, that's structural, not an artifact of the test. Codger's own 22% comes from a noisy keyword-match measurement (thin n=3, biased toward verbose prose): read it as a floor, not a precise number.
Set 3
Real Claude Code sessions, the most credible run.
Haiku, n=1, 12 surgical tasks. The other skills run as installed hook-based plugins, codger via --append-system-prompt.
Change in lines of code against the no-skill baseline. Left of the line is less code.
Codger writes the least code of the three and is second-lowest on over-engineering (0.58, caveman 0.50, ponytail 0.67), at a real tokens/cost/time premium: +12% tokens, +7% cost, +19% time. A CLAUDE.md-style ruleset re-injects heavier than a single skill file. n=1, one run, not a settled result.
Failures
What broke, and how.
- Set 1: caveman beats codger on raw LOC. Codger is not the smallest-output skill of the three on single-shot completions.
- Set 2's keyword-based pushback measurement is too noisy at n=3 to trust on its own: it structurally favors verbose prose over terse output, and a thin sample size makes single flips swing the average.
- Set 3 is n=1. Ponytail and caveman's own published numbers come from 10-run medians; codger's don't yet.
Code
Public repo, full methodology.
Every number here, plus the harness that produced it, is public.
Next
What this result changes.
Run Set 1 against real Anthropic models instead of the free NVIDIA NIM tier, and push Set 3 past n=1 before treating any ranking as settled.