Our tests
How large a folder can Rubicon handle?
We gave Claude, with the Rubicon plugin, the same two questions about a folder of 18 interviews and about a folder of 100, about 300 pages. On 100 interviews Claude shared the coding out among ten helpers, other copies of Claude, each reading ten interviews in full. The 18 interviews were coded at least as completely inside the folder of 100 as on their own, and on one question more completely. A question over 100 interviews took 31 to 43 minutes, and used between 2.3 and 3.6 times as much of a Claude plan as the same question over 18, measured as the list-price value of what each run used. We ran the tests on 5 November 2026, on a development build of the plugin.
What we tested
We gave Claude Opus 5.5, running in Claude Code with the Rubicon plugin, two questions about interviews on using AI at work, which are published under a CC BY licence:
- “Is people’s reluctance to use AI at work better explained by worries about their jobs, or by doubts about the quality of what it produces?”
- “What work activities do the interviewees describe doing, and in which of them do they use AI? List each activity with how many people describe it.”
The first question turns on a few passages in each interview. The second asks about something every interviewee describes at length, about nine activities an interview, which is a stricter test of whether the coding misses things.
We asked each question twice: about a folder of 18 interviews (156,000 characters, about 50 pages), and about a folder of 100 interviews (907,000 characters, about 300 pages) that holds the same 18. The 100 are 34 creative workers, 33 scientists and 33 people from the general workforce. Each run had the question, the documents, a table of each document’s attributes, and a note telling it to make its own reasonable choices wherever it would otherwise stop to ask the evaluator.
How Claude handled 100 interviews
- On both questions about 100 interviews, Claude read one interview, wrote one set of definitions for the coding, divided the 100 interviews into ten batches of ten, and gave each batch to a helper with the same instructions. Each helper read its ten interviews in full. Claude then merged their coding, the second reading checked the answer against the documents, and fixed rules recounted every number and created the report.
- On the activities question the second reading shared itself out as well, with one helper for each batch.
- On 18 interviews Claude did not share the work out: it read and coded all 18 itself.
- Each of the four reports was recreated from its zip by fixed rules and matched it exactly, so every number in it was recounted from the coded passages and every quotation was found in its document.
Did the coding suffer on the larger folder?
We compared each run’s coding of the same interviews, in the small folder and inside the large one.
For the first question we set each run’s coding of the 18 interviews beside a reference coding made earlier for our test bank. The reference marks 10 of the 18 interviews as raising worry about the speaker’s own job, whether to voice it or to deny it, and 16 as raising doubts about the quality of what AI produces. Of the interviews the reference marks, the number each run also marked:
| 18 interviews | the same 18 among 100 | |
|---|---|---|
| Worry about one’s own job, of 10 | 7 | 7 |
| Doubts about the quality of what AI produces, of 16 | 16 | 16 |
The run on 100 also marked one interview as showing job worry that the reference does not, a borderline case. Both runs reached the same answer: doubts about quality, rather than worries about jobs, explain why people hold back.
For the second question we wrote a key before either run started: 53 activities that six of the 18 interviewees describe, each with the passages it rests on. A separate instance of Claude, not told which run had written which coding, judged which of the key’s activities each run had found. Activities in the key that each run found, out of the 53 in those six interviews:
| 18 interviews | the same six among 100 | |
|---|---|---|
| Activities found | 38 (72%) | 48 (91%) |
The difference seems to come from how the coding was shared out rather than from the size of the folder. The helpers on the large folder, with ten interviews each, coded 8.6 activities an interview on average; Claude, coding all 18 itself, coded 6.1. We have not tested whether asking Claude to share out the coding of a small folder gives the same gain.
Time, and how much of a Claude plan a run uses
For each run, the minutes it took and the list-price value of what it used, which is what the same work would cost at Anthropic’s API prices, as Claude Code reports it:
| Question | Folder | Minutes | List-price value |
|---|---|---|---|
| Job worries or quality doubts | 18 interviews | 18 | $5.90 |
| Job worries or quality doubts | 100 interviews | 31 | $13.79 |
| Work activities | 18 interviews | 20 | $5.50 |
| Work activities | 100 interviews | 43 | $19.83 |
On a Claude plan you pay your subscription rather than these amounts. A run uses part of the plan’s allowance, and these values show how large one run is against another. Our tests ran on a Max plan. We have not measured what share of a Pro or Max plan’s allowance a run of this size uses.
Most of the extra on 100 interviews was the helpers. Their coding came to 61 and 66 cents of list-price value for each 100,000 characters, about 33 pages. On the activities question the second reading’s helpers added $6.91, more than the coding.
A causal map of every claim
A causal map asks for more: every causal claim in every interview, coded to a codebook built from a sample of the interviews. We tested that on 4 November 2026 on ten interviews from the same collection (88,000 characters, about 30 pages), with version 0.1.7 of the plugin, published at the time. We asked as an evaluator would: “Make a causal map of these interviews: every causal claim the people in them make, what they say led to what, with the quotation behind each. Build a codebook of causal factors from a representative sample of the interviews first, then code all of them to that codebook.”
The run took 25 minutes, with a list-price value of $7.14. Claude built a codebook of 27 items from five of the ten interviews, coded all ten interviews to it, 121 causal links in all; its report matched its zip.
Minutes and list-price value of each stage of the run, out of its 24.7 minutes and $7.14 (rounded):
| Stage | Minutes | List-price value |
|---|---|---|
| Reading the interviews and choosing the sample | 1.3 | $0.97 |
| Building the codebook from five interviews | 5.8 | $1.20 |
| Coding all ten interviews to the codebook | 1.7 | $0.53 |
| Writing the answer | 2.6 | $1.00 |
| The second reading | 11.5 | $3.03 |
| Writing the report | 1.8 | $0.40 |
Coding the interviews to the codebook was the smallest part, because Claude had read all ten at the start and coded them from what it had read. The second reading, a fresh helper rereading all ten interviews and every coded link, was the largest; it found 8 links the first coding had missed.
Compared with Causal Map’s one-click coding
Causal Map, the causal-mapping app from the same team, has a one-click AI coding route: it codes each document with a fast, low-cost model (Gemini Flash), suggests a codebook from all the links it coded, relabels each of those links to the codebook from its cause, effect and quote, and then adds links that join separate parts of the map (“connect islands”). We gave it the same ten interviews and the same request, as a new user would, leaving every setting at its default.
So the two products used the codebook differently. One-click relabelled the links it had already coded, and a link end the relabelling did not move kept the coder’s own label. Rubicon coded the interviews again against the codebook (hard recoding), which the Causal Map app also offers, though one-click does not use it by default.
The two runs on the same ten interviews, each cost by its own measure:
| Rubicon | One-click | |
|---|---|---|
| Minutes | 25 | 6 |
| Cost | $7.14, the list-price value of what the run used of a Claude plan | $0.38 of model calls, as logged in the app |
| Causal links | 121 | 175, of which 29 from connecting islands |
| Links with both ends on the codebook | 110 of 121, on 27 items | 69 of 175, on 20 labels |
To see which claims each coded, two of the ten interviews, drawn at random, were read claim by claim against both codings. Causal claims in those two interviews, out of the 35 that either coded:
| Claims | |
|---|---|
| Coded by both | 22 |
| Coded by both from the same passage, but as different links | 2 |
| Coded by one-click only | 9 |
| Coded by Rubicon only | 2 |
One-click’s coding covered more of what people said: 7 of its 9 extra claims are about using AI, mostly why the person reached for it and what came of it. Rubicon’s codebook stayed closer to what they meant. One-click’s relabelling filed two of a copyeditor’s statements that AI cannot replace her, among them “AI will never completely take over copyediting”, under “Job disruption”, and its connect-islands step added 3 links in those two interviews that join its own links rather than restate anything the interviewees said.
Sharing the coding out
Rubicon’s lower count looked like the cost of Claude coding all ten interviews itself, as it coded the 18 interviews above. So we tried an experimental version of the plugin, never published, that shares the coding of a causal map out among helpers three interviews at a time. Helpers reading three interviews each coded 211 links, against 116 for the same version with Claude reading all ten itself. On one interview read claim by claim, most of the extra links were claims the single reader had passed over, three of them claims only one-click had coded.
The cost doubled: $14.39 of list-price value against $6.61, and 47 minutes against 23. Most of the extra went on the second reading, since each helper that checked a batch read the answer and its tables again before its own interviews. A second version with a lighter check went past its limit of $12.73 before it finished. We did not adopt the change. Batches of three interviews are much smaller than a real folder would use, so these runs overstate the overhead on a large folder by an amount we have not measured.
We have not yet tested a causal map of a larger folder. On a folder of about 1,000 pages the interviews alone would fill Claude’s whole context window, the amount of text it can work with at once, so Claude could not read the folder itself as it read these ten.
What we have not tested
- More than one run of each question on each folder.
- A folder of more than 100 interviews with the plugin. In October 2026 Claude without the plugin, told to keep a record of its work, answered single questions from a folder of 114 documents, about 600 pages, sharing every document out among helpers by itself; that test had none of the plugin’s checks and created no report of this kind.
- Long or varied documents. These interviews are about three pages each.
- How accurate each answer is, sentence by sentence, as the accuracy page measures it. Rubicon’s own checks find every quotation and recount every number, but they do not catch a passage that was misread.
- The published plugin on these folders: the tests above, apart from the first causal map, ran on a development build.
- More than two interviews compared claim by claim with Causal Map’s coding, or more than one run of either product on them.
The key for the activities question was written by Claude before either run started, and the judge that scored both runs against it is also a Claude model. The interviews compared claim by claim with Causal Map’s coding were read by a Claude session too.
This page was drafted by the Causal Map team with the assistance of Claude, an AI model from Anthropic; the test design, the numbers and the conclusions are our own.