Our tests
How accurate is Rubicon compared with plain Claude?
We gave the same model the same eleven evaluation questions twice, once with the Rubicon plugin and once without it, and had every answer checked against its documents. Rubicon’s answers had about a third as many errors as plain Claude’s, and fewer on every one of the eleven questions. Plain Claude’s answers were just over a third of the length, and took a fraction of the time and cost to make. We ran the test on 26 October 2026, on the version of the plugin current that day.
What we compared
We gave the same model, Claude Opus 5.5 running in Claude Code, the same task twice: once with the Rubicon plugin and once without it (“plain Claude”). Each time it had the same one-line instruction (the evaluator’s question, then “Answer the question, and write your answer to answer.md”), the same documents, the same table of each document’s attributes, the same tools, and the same note telling it to make its own reasonable choices wherever it would otherwise stop to ask the evaluator. The plugin was the only difference.
We asked eleven questions from our own test bank, each of a different kind, each on a small set of public documents:
| Kind of question | Documents |
|---|---|
| A count of who says something from their own experience | 4 interviews about using AI at work |
| A proportion compared between two groups | 4 interviews about using AI at work |
| Competing explanations weighed against each other | 4 interviews about using AI at work |
| A ranking | 4 interviews about using AI at work |
| Causal chains as each person tells them | 4 interviews about using AI at work |
| A typology of whole cases | 6 interviews about using AI at work |
| An open question over written evidence, organisations against individuals | 4 written submissions to a UK parliamentary committee |
| Reports read for results against their aims | 2 annual reviews of a UK-funded social protection programme in Zambia |
| A verdict against the evaluator’s stated standard | 18 household interviews from a published QuIP study |
| Negative and unplanned effects | 5 household interviews from the same study |
| The programme’s contribution against other influences | the same 5 household interviews |
The interviews about AI at work are published under a CC BY licence. The household interviews come from an anonymised QuIP study that is already public. The committee submissions are published under the Open Parliament Licence, and the annual reviews by the UK government under the Open Government Licence.
How we counted errors
Each answer was checked twice, independently, by a separate instance of Claude that read every sentence and quotation against the documents. Neither reviewer was told which set-up had written the answer, and neither saw the other’s review. A third instance, also not told, then merged the two reviews so that an error either reviewer found is counted once.
We count root errors: one misread passage, one wrong definition or one wrong statement counts once, even where it makes several later sentences wrong. Each reviewer also marked the errors it judged substantive, meaning they matter to what the answer concludes; an error counts as substantive here if either reviewer marked it so. Rates are errors per 100 sentences of the answer’s text, leaving out tables and headings.
What we found
Errors per 100 sentences, from the two merged reviews, with 95% confidence intervals in brackets:
| All errors | Substantive errors | |
|---|---|---|
| Rubicon | 5.6 (3.7 to 8.0) | 2.1 (1.1 to 3.8) |
| Plain Claude | 18.8 (14.8 to 23.5) | 6.9 (4.6 to 10.0) |
Rubicon’s answers had fewer errors per sentence than plain Claude’s on every one of the eleven questions, and about a third as many overall. A difference this large across eleven questions is very unlikely to be chance (p < 0.001 for all errors, p = 0.001 for substantive ones). Counting each sentence the reviewers tested, tables included, gives the same picture (6.1 against 13.7 errors per 100), though by that count plain Claude’s rate is the lower on two of the eleven questions.
Plain Claude’s errors were mostly statements that reach further than the documents: a quotation cut so that it says more than the speaker did, a wish read as current practice, a rule said to hold in every case when several of the cases cited break it, and in one question a second interview in two of the files that was never read. Rubicon made the same kinds of error, less often. Its commonest substantive error was a claim about a whole group of people that one member of the group contradicts.
Where plain Claude does better
- Length. Rubicon’s answers were 2.7 times as long. We also asked two models from other companies (GPT-6 Sol and Gemini 3.1 Pro) to compare each pair of answers, in both orders, and counted a verdict only where a judge gave it in both orders. On concision they preferred plain Claude 11 times and Rubicon never.
- Time and cost. A Rubicon run took a median of 14 minutes, where plain Claude took under a minute. Priced at Anthropic’s list prices, a Rubicon run used about seventeen times as much.
- Overall preference. The same judges did not prefer either answer overall (Rubicon 9, plain Claude 5, out of 22 verdicts, which could be chance). They preferred Rubicon on writing (12 to 0) and on how well its claims are supported by evidence (11 to 1).
- Coverage. The judges leaned towards Rubicon on whether the answer covers what the question asks (8 to 2), which could be chance. Our reviewers find wrong statements more reliably than they notice what an answer leaves out, so the error counts above miss most omissions.
The limits of the test
- One answer from each set-up per question.
- Short document sets: 2 to 18 documents a question, all but one question under 10,000 words in all.
- The questions and documents are our own test bank.
- The reviewers and judges are AI models.
- The reviewers were not told which set-up wrote an answer, but Rubicon’s answers look different: they are longer and cite a passage for most sentences.
This page was drafted by the Causal Map team with the assistance of Claude, an AI model from Anthropic; the test design, the figures and the conclusions are our own.