Research
Claims are cheap. Evidence isn't.
An agent isn't useful because it looks intelligent for five minutes. It's useful when it completes a task, handles errors, communicates its limits, and produces a result you can verify. That's what we measure, and we publish what we find, including the parts that don't flatter us.
Current tracks
What's being measured.
This page is young, like everything here. Tracks fill in with public results as experiments complete.
Agent benchmarks
in developmentTask suites that measure what matters for an operational assistant: finishing multi-step work, recovering from errors, knowing when to stop and ask.
Solen Model evaluation
in developmentOur fine-tuned model measured against its open-source base and against hosted alternatives, on the tasks Jarvis actually runs, published win or lose.
Tool-use reliability
plannedHow often does an agent pick the right tool, use it correctly, and interpret the output honestly? We want numbers, per tool, per model.
Memory and retrieval
researchWhen does long-term memory help and when does it mislead? Testing retrieval quality against a growing Astro over months, not minutes.
How experiments are reported
Every experiment, same skeleton.
Each published experiment follows one template, so results stay comparable and failure cases can't quietly disappear.
First published experiment
Codger, evaluated the same way.
A separate open-source project, not a Solen AI product, held to the same reporting skeleton as everything else on this page.
Second published report
Astro, measured against itself.
Not a benchmark against Obsidian or Logseq, that head-to-head hasn't been run yet, but a real before-and-after: what v0.1.0 got wrong on a 364-note vault, and what v0.1.1 actually fixed, counted rather than described.
Notes that appeared in the app, out of the vault's 364, before and after v0.1.1 keyed notes by path instead of title.
Reported, not benchmarked
The two Astro skills, with a description where a number would go.
Some things have no comparable project to be measured against, and inventing a benchmark for them would be worse than admitting it. These follow the same skeleton, with the result section replaced by what the skill actually does and a full account of where it is judgement rather than measurement.
Want to run experiments with us?
The research program is one of the main things Solen Lab exists for. If benchmarks and honest evaluation are your kind of fun, come help design them.
Visit the Lab