Independent research · Pre-registered
verityResearch runs small, rigorous experiments on what fine-tuning and tool access actually buy an AI model. Arm definitions, allowed interpretations and pass/fail thresholds are fixed and frozen first. The results are then read against criteria that already existed.
Tool-call rate
Held-out set, n = 106. Share of prompts answered with a working tool call.
0 → 98% Same tools available to both arms.
Method
A result is only as good as the moment its criteria were fixed. Every study published here follows the same sequence, and the order is the whole point.
Each arm is specified in full — which model, which tools, which prompts — before a single evaluation is run. What counts as a valid interpretation of a response is written down at the same time, so it cannot be widened later to accommodate an answer.
The thresholds that decide the verdict are committed alongside the arm definitions. From this point the study can fail. That possibility is what makes the eventual claim worth anything.
Only now is a result file opened. The verdict is whatever the frozen criteria say it is. This is a claim about method, not about outcomes — it does not promise that findings will be favorable, only that they were not decided after the fact.
Verdict
Fine-tuning earns its claim beyond tool access alone, on both evaluation sets.
| Arm | Tool-call rate | Answer rate |
|---|---|---|
| Held-out set — n = 106 | ||
| Base model | 0% | 48% |
| Fine-tuned — v20 | 98% | 96% |
| Adversarial probe — n = 43 | ||
| Base model | 2% | 13% |
| Fine-tuned — v20 | 83% | 86% |
Criteria frozen before any result file was read
Sharpest finding
The base model essentially never emits a working tool call at all, regardless of phrasing. Tool access alone does not confer tool-use behavior at this model size.
About
Narrow, answerable questions about what fine-tuning and tool access actually buy a model — run end to end, with the criteria set in advance and the code open. Studies appear here when their pre-registered criteria have been resolved, whichever way they resolve.
verityResearch is run by Tony Houston, working with AI collaborators.