Checkable AI for evaluators

Example report: chains of events, gathered into factors

An evaluator used the Rubicon plugin for Claude to answer one evaluation question from a set of interviews. Click any number, source ID or quotation in the report below to see the interview passages behind it. This report follows each interviewee's account of what happened and what they then did, keeps only the chains one person tells whole, gathers the steps into a short list of factors, and draws a causal map of the links inside those chains.

The question
What leads interviewees to trust or distrust what AI produces for them? Trace the steps they describe, from what happens to what they then do, keeping only chains one person tells whole.
The interviews
Four professionals (a scientist, a creative writer and two general workers), from the Anthropic Interviewer dataset, published under CC BY.

The original report file, byte for byte (this page is the same report, with its scripts moved to a file for the site) · Word copy · About the Rubicon plugin for Claude

Rubicon report

What leads interviewees to trust or distrust AI's output

Question The evaluator asked: "What leads interviewees to trust or distrust what AI produces for them?

Trace the steps they describe, from what happens to what they then do, keeping only chains one person tells whole." How it will be answered. The four interviews in corpus/ (one scientist, one creative writer, two general-workforce workers; each interview is one person and one case) are read in full, and every passage coded in which the interviewee tells a causal chain that runs from something that happens (a quality of AI output, an experience of it being right or wrong, a feature of the task) through their trust or distrust in what AI produces to what they then do (check it, accept it, rewrite it, keep a task to themselves, use AI more or less). Each link of such a chain is one row, with a cause and an effect in the interviewee's own words, and the rows of one chain carry the same sequence label and their order in it. Only chains one interviewee tells as one, in one passage or across consecutive answers about the same episode or task, are kept and reported; links from different people, or from unrelated parts of one interview, are never joined into a chain. A chain counts only if it reaches trust or distrust (stated, or plainly implied by checking, refusing or relying on the output) and goes on to what the person does. The cause and effect labels are then gathered into one short list of factors drawn from the interviews (a soft recode: each label is given its factor), and the told chains are listed by factor, each with the interviewee who told it. What counts: the interviewee's own account of their own use (first-hand). General opinions about AI that do not pass through what AI produced for them and what they did are not chains. The interviewer's words never count. The comparison: the answer describes the kinds of chain told, which lead to trust and which to distrust, and what each person does; counts are of interviewees out of the four, and are descriptive of these four people only. Distinctions kept apart: chains ending in trust (accept, rely, delegate more) against chains ending in distrust (check, refuse, keep the task); a told chain against links merely joined. Counting decisions that could have gone the other way: one interviewee can tell several chains, and each is listed; a chain told across two consecutive answers about the same task counts as one told chain. Second reading: a check by a fresh reader of the answer against its cited passages and the documents (the default; no blind recode). Every passage is coded systematically. No separate section for material outside the frame.

In these four interviews, the chains that run through distrust of what AI produces are of two kinds, and they end in different actions: errors in the output lead to checking it while going on using AI, and output that lacks human depth or originality leads to setting it aside for that part of the work. Accepting a particular output follows a test the person applies to it; handing AI whole tasks or a bigger role rests instead on the kind of task, or on trust expected to grow. A third reason for holding back is not a fault in the output but the person's need to understand or control the work themselves. Every one of the four interviewees told at least one whole chain ending in checking AI's output () and at least one ending in not using it (); chains ending in relying on AI, by handing it whole tasks or a bigger role, came from interviewees. These are the accounts of four people in short interviews, so the answer describes the kinds of chain they told, not how common each is.

4 interviews in 1 groups · 26 passages coded · 20 October 2026 · Run 2026-10-20-x69q

  • What was automatically checked in this reportChecked when this report was created
  • 21 numbers, every one computed directly from the coded passages rather than written by the AI. Click any number to see who it counts.
  • 26 quotations, every one found word for word in its interview. Click an interview's ID beside a quotation to read it in place.
  • 5 key claims tagged as numbers are, each with the passages for it and against it and why they bear it out. Click a claim's tag to see them.
  • 43 sentences of the answer were read against the interviews by a second AI reader, separate from the one that wrote them.
  • Not checked: whether the passages cited are all the documents hold on each point, or whether each is labelled correctly.

The chains each interviewee told whole, written in the factors their steps were gathered into:

Chain as toldInterviewees
Output lacks human depth → Does not use the output
AI produces output → Checks the output → Output lacks human depth → Does not use the output
Output has errors → Checks the output
Output has errors → Checks the output → Keeps AI under direction
Task does not suit AI → Output has errors → Keeps AI under direction
AI produces output → Checks the output → Output passes a check → Accepts or delegates
AI produces output → Checks the output → Output passes a check → Checks the output
Checks the output → AI produces output → Checks the output
Task suits AI → Accepts or delegates
Trusts output more → Accepts or delegates
Own understanding needed → Does not use the output
Own control needed → Does not use the output

Each number is out of 4 documents.

The map draws the links inside these told chains, each arrow counted by the interviewees who told it.

Drawing the map with Graphviz.

Each arrow is a causal link; its number is how many sources mention it. Click an arrow or a box to read its passages.
Show as tableHide table
Checks the outputDoes not use the outputAI produces outputKeeps AI under directionAccepts or delegatesOutput has errors
AI produces output00000
Output has errors0000
Output lacks human depth00000
Checks the output0000
Output passes a check0000
Own control needed00000
Own understanding needed00000
Task does not suit AI00000
Task suits AI00000
Trusts output more00000

The links within the told chains, between the factors they were gathered into, each counted by the interviewees who told it, out of the four interviewed. Interviewees in each group. Each number is out of 4 interviewees. Click a count for its passages.

holds the documents’ full text

Errors lead to checking, not to stopping

Where interviewees found or expected mistakes in AI's output, they went on using it but checked it or kept it on a short leash (). The investor-relations worker tells the chain in three steps: "The output created by AI still must be checked. It does make errors at time that must be caught." and so "I view AI currently as a tool rather than a total replacement." (). The writer, who lets AI give only feedback on their own writing, and only on blog articles, says it sometimes tells them they have made "mistakes which aren't really mistakes", such as removing sentence fragments that writers use for effect, and so takes "everything with a pinch of salt" (). The software developer says that on existing projects, with "legacy code, AI may not fit in well and can be less efficient and make thing worse", and so collaborates "with AI throughout tasks" there rather than handing them over (). Errors lead straight to checking in the chains of interviewees, and to closer direction of AI in the developer's. No interviewee told a chain in which errors led them to stop using AI for a task ().

Shallow or generic output leads to not using it

Where the output was not wrong but lacked human understanding, nuance or originality, the interviewees set it aside for that part of the work (); told such a chain. The writer tells two. In one, the writer weighs AI's suggestions of perspectives to add: "I pay attention to feedback about perspectives I might have missed in my articles and consider if I need to add them." () But "Since they can't think like humans, often the feedback is not something I would ever need to add to an article", and "Most of the time, I would not add them as I've noticed LLMs are just trying to provide any content." (). In the other, "AI is only capable of remixing what's already out there", so the writer does not "really want to use it for idea generation" (). The investor-relations worker prefers to do their "own final draft writing" because "it is still too easy to spot when AI has written a document and AI still isn't good enough to capture nuanced language" (), while still working with AI on the drafts before it: "I collaborate back and forth during the drafting process." (). In these chains the output is weighed and then mostly left unused for that part of the work, where in the chains through errors it is checked and then used () § Errors lead to checking, not to stopping.

Accepting an output follows a test of it

Where interviewees accept or act on a particular output, it is after a test they apply to it (). The developer describes the step: "I then review the proposed fix and accept if it makes sense." (). The scientist runs AI's code "to see if it produces an error", and if it does not, still goes "through the new code, line-by-line, to check if it is logical and if it is actually doing the thing I had originally intended" (): passing the first test leads to a second check rather than straight to trust. The investor-relations worker asks the assistant for its own analysis to test "how intelligent the assistant is", and "Once the assistant calls out a trend I dig deeper into the data to confirm that trend." (). A test applied to AI's output appears in interviewees' chains, the writer's being the weighing of suggested perspectives above.

The two chains that end in handing AI more of the work do not run through a test of one output, and go against this (). Only the investor-relations worker speaks of trust as such, and as an expectation for the future: "The more you trust the output the bigger role it will have in my work place." (). On new projects, where "AI is better positioned to help", the developer tends "to give AI complete tasks", though "I always review AI's work and decide accordingly" (). For the developer, the kind of project sets how much is handed over: whole tasks on a new project, step-by-step collaboration on an existing one () § Errors lead to checking, not to stopping.

Some holding back is not about the output's quality

Two interviewees tell chains in which they keep work from AI for reasons that lie in their own needs rather than in anything wrong with the output (). The scientist, once back in practice with coding, "needed to know my own code like the back of my hand so I could make small adjustments", and "Getting an AI to write this code would not be useful for me, as I would not understand it as well as if I had written it myself." (). For theoretical work the scientist holds back because AI "does the hard work then gives you an answer that you simply just read", and instead needs to "spend the time looking for the information and trying to understand it from multiple sources" (). The developer prefers "to work by myself when dealing data" because the requirements are driven by data that is "often business related" and "I feel I have better control over the data." (). These chains end in not using AI, as the chains through shallow output do, but their reasons are the person's own understanding or control, not errors or shallowness in what AI produced () § Shallow or generic output leads to not using it.

Limitations

  • There are four interviews of about ten minutes each, led by an AI interviewer, with one scientist, one creative writer and two people from the general workforce. The kinds of chain found are what these four people told; other people, or longer interviews, would likely add others.
  • Several chains were told in answer to a direct question from the interviewer, such as the scientist's account of checking code, which answers the interviewer's "how do you verify or validate what it produces?" (). They were told when asked, not raised unprompted.
  • Some accounts were left out because the person did not tell them whole, from what happened to what they then did: the developer's Java upgrade, where "AI failed to detect" a needed library upgrade but no response is described (), and the same developer's satisfaction with AI-written property-based tests, which "sped up the process" with no stated effect on trust (). The investor-relations worker's concern that "anything written on an AI platform becomes open access" () is about trust in the platform with confidential data, not in what AI produces, and was also left out. Counting these as chains would add to the errors and trust sides.
  • The factors were drawn from the interviewees' own words by one reader and checked by a second reader against the passages; they were not coded twice independently.
  • The count of interviewees whose chains end in relying on AI () rests on coding the developer's new-project chain as relying (handing over complete tasks) rather than checking ("I always review AI's work and decide accordingly", ); coded as checking, only the investor-relations worker would remain, and then only as an expectation.
  • That some holding back is not about the output's quality rests, for the developer, on reading "I feel I have better control over the data" () as a need for control rather than distrust of what AI would do with the data (); read the other way, that finding would rest on the scientist alone ().

Annexes

How this report was written

What was read. Rubicon read all 4 documents in full. The counts below are of interviewees, one document each.

Coding. It then restated each causal claim the documents make as a link from a cause to an effect, also coded stance (what the chain this link belongs to ends in: what the interviewee does with what AI produces). It grouped the answers of cause and effect into themes, which became a codebook, and placed each passage under one of them.

Checking. A separate reader then read the answer against the passages it cites and the documents, and checked every sentence and quotation, so the coding and the answer in this report are the corrected ones, counted again after the check.

Counting. Interviewees were then counted in 3 tables: the order in which events are told; by stance; a causal map of the links kept by the exclude self loops filter (each factor and its opposite drawn as one).

Claims. The writer tagged 5 key claims that rest on no count, as each number is tagged: each names the passages for it and against it, how they would be found again and why they bear it out, and fixed rules checked that every passage named is one the coding holds.

Writing and checks. Rubicon wrote the answer from these counts, citing for each finding the passages it rests on. Every number in the answer, 21 in all, was computed directly from the coded passages rather than written by the AI, and all 26 coded passages were found word for word in their documents; every quotation in the answer, 36 in all, was found in the documents.

The workflow x-ray beside Step by step, below, draws the same procedure, a box for each stage and what it made.

Step by step

The analysis was written down as a workflow before it was counted: what to look for, how to code it, what to count. It was then counted directly from the coded passages rather than by the AI, so anyone can check a number, and another coder can follow the same definitions and code the interviews afresh.

What was read

All 4 documents, read in full: 4 anthropic/anthropicinterviewer.

Coding: Chains

one row for each passage that fits; 26 rows.

The instruction a second coder would follow
Read the whole interview. Code only the interviewee's own words, never the interviewer's. Find every chain the interviewee tells as one, in one answer or across consecutive answers about the same task or episode, that runs from something that happens (a quality of what AI produced for them, an experience of it being right or wrong, a feature of the task) through their trust or distrust in what AI produces to what they then do with it (check it, accept it, keep the task to themselves, discount it, give AI more or less of the work). Code each link of the chain as one row: the cause and the effect in the interviewee's own words, each one specific thing with its direction, and the passage quoted exactly. Give every row of one chain the same sequence label and its place in the chain as step (1, 2, 3). Do not join links from different parts of the interview that the interviewee does not tie together, and do not code a chain that stops before what the person does (such as an error described with no response to it) or that is about AI in general rather than what AI produced for this person. Where AI is held back from a task because of what its output would be (not understood, not under the person's control, lacking depth), code the chain: refusing the output is acting on distrust of it.
Cause

What leads to the next step of the chain, in the interviewee's own words: one specific thing, with its direction.

Effect

What the cause leads to, in the interviewee's own words: one specific thing, with its direction. The last effect of a chain is what the person does.

Stance

What the chain this link belongs to ends in: what the interviewee does with what AI produces. Every link of one chain takes the same value.

CodeMeansCountsDoes not count
The person accepts AI's output or gives AI more of the work.giving AI complete tasks; expecting AI to take a bigger role as trust growsaccepting only after a check of each output, which is checks
The person uses AI's output but tests, reviews, confirms or discounts it before acting on it, or keeps AI under close direction.running and reading AI code; reviewing a fix and accepting it if it makes sense; confirming a trend in the data; taking feedback with a pinch of saltnot using AI's output for the task at all, which is holds back
The person does not use AI's output for the task, or does not act on it.doing the final draft oneself; not getting AI to write code one must understand; not adding AI's suggested perspectives; not using AI for ideasusing the output after checking it, which is checks
The coded passages 26 rows
Relies 3 passages
  • For new projects, I tend to give AI complete tasks, but I always review AI's work and decide accordingly.
  • For new projects, since we are starting from scratch, AI is better positioned to help with architecture, coding with latest technologies and patterns.
  • Using AI is partly about your trust in the output it creates. The more you trust the output the bigger role it will have in my work place.
Checks 11 passages
  • I take everything with a pinch of salt. Sometimes, it will tell me I've made mistakes which aren't really mistakes. For example, with grammar. LLMs love to remove sentence fragments I've noticed.
  • I verify what it produces by running the code to see if it produces an error or doesn't.
  • If it doesn't produce an error, I go through the new code, line-by-line, to check if it is logical and if it is actually doing the thing I had originally intended.
  • AI would give me suggestions for fixing the issue.
  • I then review the proposed fix and accept if it makes sense.
  • For existing projects, I collaborate with AI throughout tasks.
  • Existing projects have existing structure, including legacy code, AI may not fit in well and can be less efficient and make thing worse.
  • I am also looking for the assistant to provide its own input. I am basically testing how intelligent the assistant is.
  • Once the assistant calls out a trend I dig deeper into the data to confirm that trend.
  • The output created by AI still must be checked. It does make errors at time that must be caught.
  • I view AI currently as a tool rather than a total replacement.
Holds back 12 passages
  • I pay attention to feedback about perspectives I might have missed in my articles and consider if I need to add them.
  • Most of the time, I would not add them as I've noticed LLMs are just trying to provide any content.
  • Since they can't think like humans, often the feedback is not something I would ever need to add to an article.
  • I don't even really want to use it for idea generation, because I don't want to write things that don't add anything to the world.
  • Right now, AI is only capable of remixing what's already out there, and I think I'm better off learning to generate ideas on my own.
  • I needed to know my own code like the back of my hand so I could make small adjustments.
  • Getting an AI to write this code would not be useful for me, as I would not understand it as well as if I had written it myself.
  • the "interacting and playing" with ideas part is complete by the AI model. You ask a question and the AI model does the hard work then gives you an answer that you simply just read.
  • I need to spend the time looking for the information and trying to understand it from multiple sources to really understand and be able to use these ideas in my research.
  • I prefer to work by myself when dealing data say retrieving data via Rest API.
  • The requirements are driven by data, which is often business related. Say what we retrieve from other systems and what we publish. I feel I have better control over the data.
  • I prefer to do my own final draft writing. I feel like it is still too easy to spot when AI has written a document and AI still isn't good enough to capture nuanced language.

holds the documents’ full text

Grouping: Factors

The cause and effect columns of Coding: Chains: 45 distinct labels on 26 passages, drawn into 13 items, and each passage given the item its label is listed under.

ItemMeansCountsDoes not countOpen labels it gathers
AI produces output
AI produces something for the person: code, a fix, a trend, an analysis
The point in a chain where AI hands the person something it has made or found.AI writing or fixing code; AI suggesting a fix; AI feedback pointing to perspectives that might be missing; the assistant calling out a trend; the person asking the assistant for its own inputa judgement of how good the output is (goes to Output has errors)AI feedback points to perspectives that might have been missed, AI writes or fixes code, AI suggests a fix for the issue, asks the assistant for its own input on trends, assistant calls out a trend
Output passes a check
The output passes the person's test
AI's output survives a test the person applies: it runs without an error, or it makes sense on review.code running without an error; a proposed fix that makes sensethe test itself (goes to Checks the output)code runs without an error, proposed fix makes sense
Output has errors
AI's output is wrong or could make things worse
The person has found, or expects, mistakes in what AI produces.errors that must be caught; feedback flagging mistakes that are not mistakes; AI changes that may make existing code worseoutput that is correct but shallow or generic (goes to Output lacks human depth)AI flags mistakes that aren't really mistakes, AI may not fit in and can make things worse, AI makes errors at times
Output lacks human depth
AI's output lacks human understanding, nuance or originality
What AI produces is not wrong as such but generic, unoriginal or without the nuance a human would bring.AI not thinking like humans; feedback that is just any content; ideas that only remix what exists; writing that is easy to spot and not nuancedfactual or technical mistakes (goes to Output has errors)AI feedback often not something to add, LLMs can't think like humans, AI ideas would not add anything to the world, AI only remixes what's already out there, AI writing easy to spot and not nuanced enough
Own understanding needed
The person needs to understand the work themselves
The task requires the person's own grasp of the material, which taking AI's output would not give them.needing to know one's own code to adjust it; not understanding AI-written code as well; AI doing the thinking and leaving an answer only to readwanting control over business data (goes to Own control needed)needs to make small adjustments to own code, needs to know own code like the back of the hand, would not understand AI-written code as well as own, AI does the interacting and playing with ideas, AI gives an answer you simply just read
Own control needed
The person wants direct control over business-critical work
The task turns on business requirements the person wants to keep in their own hands.requirements driven by business data; feeling of better control working aloneneeding to understand the material (goes to Own understanding needed)has better control over the data working alone, requirements driven by data that is often business related
Task suits AI
The task is one AI is well placed to help with
Something about the task makes AI's output more likely to fit.a new project starting from scratch, where AI can bring the latest patternsa task with existing structure AI may not fit (goes to Task does not suit AI)AI better positioned to help with architecture and latest technologies, new project starts from scratch
Task does not suit AI
The task is one AI's output may not fit
Something about the task makes AI's output less likely to fit.an existing project with legacy codea new project from scratch (goes to Task suits AI)existing project has legacy code
Trusts output more
The person trusts AI's output more
Greater trust in what AI produces, stated as such.more trust in the outputaccepting one output after checking it (goes to Accepts or delegates)more trust in AI's output
Checks the output
The person tests, reviews, confirms or discounts AI's output
The person does not take the output as given but tests it, reads it through, confirms it against the data or weighs it with suspicion.running and reading code; reviewing a fix; digging into the data to confirm a trend; output that must be checked; taking feedback with a pinch of salt; considering whether to add a suggestion; testing how intelligent the assistant isnot using the output at all (goes to Does not use the output)takes everything with a pinch of salt, considers whether to add them, runs the code to see if it errors, checks the new code line by line for logic and intent, reviews the proposed fix, wants to test how intelligent the assistant is, digs deeper into the data to confirm the trend, AI output must be checked
Keeps AI under direction
The person keeps AI as a helper under their own direction
The person works with AI step by step, or treats it as a tool rather than handing work over.collaborating with AI throughout a task; viewing AI as a tool, not a replacementhanding AI complete tasks (goes to Accepts or delegates)collaborates with AI throughout the task, currently views AI as a tool, not a replacement
Accepts or delegates
The person accepts AI's output or gives AI more of the work
The output is taken up, or AI is given whole tasks or a bigger role.accepting a fix; giving AI complete tasks; AI taking a bigger role in workworking with AI step by step under close direction (goes to Keeps AI under direction)accepts the fix, tends to give AI complete tasks but always reviews its work, AI expected to have a bigger role in work
Does not use the output
The person does the task without AI, or does not act on AI's output
AI's output is not used for the task, or its suggestions are not taken up.doing the final draft oneself; not getting AI to write code; researching from several sources oneself; mostly not adding AI's suggested perspectives; not wanting to use AI for ideas; working alone on data retrievalusing the output after testing it (goes to Checks the output)most of the time does not add the suggested perspectives, does not really want to use AI for idea generation, does not get AI to write this code, spends the time looking for information from multiple sources, works alone on data retrieval, does own final draft writing
What the grouping merged, split or left alone

Factors were drawn from the cause and effect labels together, so a factor that is only ever an effect is still found. The task's fit with AI is kept in two items, one for each side. Understanding and control were kept apart because they are different reasons for holding back, each told by a different interviewee. 'wants to test how intelligent the assistant is' was placed under Checks the output because the speaker gives it as the purpose of asking for the assistant's own analysis, which is then confirmed against the data.

Counting: Factors by cause item and effect item

Interviewees by cause item and effect item

Each chain an interviewee told as one, from what happened to what they then did, written in the factors its steps were gathered into, with the interviewees who told it, out of the four interviewed.

The table 12 cells
ChainInterviewees
Output lacks human depth → Does not use the output of 4
AI produces output → Checks the output → Output lacks human depth → Does not use the output of 4
AI produces output → Checks the output → Output passes a check → Accepts or delegates of 4
AI produces output → Checks the output → Output passes a check → Checks the output of 4
Checks the output → AI produces output → Checks the output of 4
Output has errors → Checks the output of 4
Output has errors → Checks the output → Keeps AI under direction of 4
Own control needed → Does not use the output of 4
Own understanding needed → Does not use the output of 4
Task does not suit AI → Output has errors → Keeps AI under direction of 4
Task suits AI → Accepts or delegates of 4
Trusts output more → Accepts or delegates of 4

holds the documents’ full text

Counting: Chains by stance

Interviewees by stance

Interviewees who told at least one whole chain ending in each kind of response to what AI produced, out of the four interviewed.

The table 3 cells
StanceInterviewees
of 4
of 4
of 4

holds the documents’ full text

Counting: Factors by cause item and effect item

Interviewees by cause item and effect item

The links within the told chains, between the factors they were gathered into, each counted by the interviewees who told it, out of the four interviewed.

The table 13 cells
Cause itemEffect itemInterviewees
AI produces outputChecks the output of 4
Output has errorsChecks the output of 4
Output lacks human depthDoes not use the output of 4
Checks the outputAI produces output of 4
Checks the outputKeeps AI under direction of 4
Output has errorsKeeps AI under direction of 4
Output passes a checkAccepts or delegates of 4
Output passes a checkChecks the output of 4
Own control neededDoes not use the output of 4
Own understanding neededDoes not use the output of 4
Task does not suit AIOutput has errors of 4
Task suits AIAccepts or delegates of 4
Trusts output moreAccepts or delegates of 4

holds the documents’ full text

Key claims

  • Where interviewees found or expected errors in AI's output, they went on using AI but checked its output or kept it under close direction, rather than stopping.
    Each chain that runs through errors ends in checking, discounting or step-by-step collaboration, and none ends in giving up AI for the task. Rests on: . Found by: rows whose cause_item is Output has errors or Task does not suit AI, read with the rest of their chain.
  • Where AI's output lacked human depth, nuance or originality, the interviewees set it aside for that part of the work: most of the time not adding its suggestions, not really wanting it for ideas, or writing the final draft themselves.
    Every chain through this factor ends in the person not adding AI's suggestions most of the time, not wanting it for ideas or writing the final draft themselves; the writer first weighs the suggested perspectives, and the investor-relations worker still collaborates with AI on earlier drafts. Rests on: . Found by: rows whose cause_item is Output lacks human depth, read with the rest of their chain.
  • Where interviewees accept or act on a particular output of AI, it is after the output passes a test they apply to it.
    The fix is accepted only if it makes sense on review, code that runs is still read line by line, and a trend is confirmed in the data. The two chains that end in handing AI more of the work do not run through a test of one output: the developer's new projects, where whole tasks are handed over because of the task (though still reviewed), and the investor-relations worker's expected link from more trust to a bigger role. Rests on: . Against: . Found by: chains ending in Accepts or delegates or running through Checks the output and Output passes a check.
  • Some holding back turns not on the quality of AI's output but on the person's need to understand the work or to control business-related data.
    These chains end in the person not using AI for the task, and their reasons are about the person's own grasp or control, not about errors or shallowness in what AI produced. Rests on: . Found by: rows whose cause_item is Own understanding needed or Own control needed.
  • For the software developer, the kind of project decides how much is handed to AI: on a new project the developer tends to give whole tasks, on an existing one with legacy code they collaborate step by step.
    The developer tells the two chains side by side, across two consecutive answers, as two sides of one contrast. Rests on: . Found by: rows whose cause_item is Task suits AI or Task does not suit AI.

Writing: Answer

The instruction the answer was written to

What the evaluator was asked to agree

The four documents (default taken; evaluator not available) (2026-10-20T14:34:40+01:00). Put to the evaluator in the chat; their answer is as the AI that asked recorded it. They answered: Use these four interviews as the documents? (Evaluator not available in this session: the recommended default was taken, not chosen by the evaluator.) Yes, all four (recommended).

  • The documents, as shown under What was read.

Each was set out to them only by the AI that asked, so that it was shown is the AI's word.

The settled question (default taken; evaluator not available) (2026-10-20T14:34:41+01:00). Put to the evaluator in the chat; their answer is as the AI that asked recorded it. They answered: Answer the question as settled in the run? (Evaluator not available in this session: the recommended default was taken, not chosen by the evaluator.) Accept (recommended); Second reading before the draft is shown? Check by a fresh reader (recommended); Route to the factor list Soft recode: one reading, labels gathered into factors (recommended); Separate section for material outside the frame? No (recommended).

  • The question, as shown at the head of this report.

Each was set out to them only by the AI that asked, so that it was shown is the AI's word.

The checks that were run

  • 26 of 26 coded passages found word for word in their document.
  • 36 of 36 quotations in the answer found word for word in the documents (a quotation of under twelve characters, such as one word, is not checked).
  • 21 numbers in the answer, every one computed directly from the coded passages rather than written by the AI; 0 written by the AI without a count behind them.
  • 0 citations to passages that do not exist; 0 quotations not in the passage they cite.

The check

A separate reader read the answer against the passages it cites and the documents, and checked every sentence and quotation of the answer against the documents, correcting the coding and the answer where they were wrong. The answer above is the corrected one, recounted after the check.

The check's record check.md
Counts
  • Disagreements ruled: 0 for the rows, 0 for the second coder, 0 unsettled. No second coder coded these documents (there is no recode/disagreements.md), so the ruling on the coding was left out, as the brief says.
  • Rows changed: 8 in coded/chains.json (1 added, 7 corrected). judged/factors.json updated to match (labels moved with their rows, one new label placed under each of two items, "counts" text updated); judged/k.json updated (the new row added to depth; earned narrowed and given the rows that go against it; depth and task wording corrected).
  • Sentences changed in answer.md: 27 (listed below), plus one row added to the chains table.
  • Recount (`recount.py . --answer answer.md`) runs clean: no bare numbers, no citations to missing rows, every quotation found in its document and in what it cites, and no audit findings.
Omissions found
  • factors by cause item and effect item[AI produces output,Checks the output] (before the change it covered science_0001, work_0195, work_0860). I searched creativity_0104 and found that the writer weighs AI's suggested perspectives before deciding: "I pay attention to feedback about perspectives I might have missed in my articles and consider if I need to add them." (creativity_0104, in reply to the interviewer's question on how they respond to AI's feedback). This is the first link of the writer's perspectives chain, which both readings missed. Added as row chains.s2ers (perspectives, step 1, holds back), and the steps after it were renumbered. The cell now covers all four interviewees. The sentence that cites this count was updated, and the depth section's claim that "no check can rescue the output" was corrected, because the writer does weigh each suggestion.
  • factors by cause item and effect item[Output has errors,Checks the output] (creativity_0104, work_0860). I searched science_0001 and work_0195. The scientist's "if my code gave an error ... I would copy and paste this into an AI model" is about errors in the scientist's own code, not in AI's output. The developer's Java upgrade ("AI failed to detect that") has no response told. Nothing added.
  • factors by cause item and effect item[Output lacks human depth,Does not use the output] and the matching factors by cause item and effect item cell (creativity_0104, work_0860). I searched science_0001 and work_0195 and found nothing that fits.
  • chains by stance[relies] (work_0195, work_0860). I searched science_0001 and creativity_0104. The scientist uses AI when it "would solve a problem quicker", and the writer says AI has "a place as more of a writing assistant". Neither passes through trust in an output to what the person does, so nothing was added.
  • "No interviewee told a chain in which errors led them to stop using AI": searched all four, and it holds.
  • "Only the investor-relations worker speaks of trust as such": I searched all four for trust/rely/confidence. The writer's "unconfident" is about their own article, so the claim holds.
  • "Two interviewees keep work from AI for reasons in their own needs": the writer also keeps research to themselves because "I enjoy doing that myself too much" (creativity_0104). That does not pass through AI's output, so under the definition it is not a chain. The sentence was narrowed to "tell chains in which".
Rows changed
  • chains.s2ers (creativity_0104): added, see Omissions.
  • chains.42bez (creativity_0104): the effect "AI feedback is just any content, never needed" overstated "often the feedback is not something I would ever need to add" (how often) and borrowed words from another sentence. It is now "AI feedback often not something to add", and the step is now 2.
  • chains.3ugj8 (creativity_0104): the effect "does not add the suggested perspectives" dropped "Most of the time". It is now "most of the time does not add the suggested perspectives", the cause matches the corrected link before it, and the step is now 3.
  • chains.4ut8a (creativity_0104): "does not use AI for idea generation" became "does not really want to use AI for idea generation", because the passage gives a hedged wish ("I don't even really want to use it").
  • chains.8f26y (work_0195): "gives AI complete tasks" became "tends to give AI complete tasks" ("I tend to give AI complete tasks").
  • chains.h2jtd (work_0195): "requirements driven by business data" became "requirements driven by data that is often business related" ("which is often business related").
  • chains.vuuh4 (work_0860): "views AI as a tool" became "currently views AI as a tool" ("I view AI currently as a tool").
  • chains.59vy5 (work_0860): "AI plays a bigger role in work" became "AI expected to have a bigger role in work". The passage looks ahead, in answer to "how do you envision your relationship with AI at work evolving?", and says "will have".
Sentences changed

1-4. Opening. "distrust ... comes from two kinds of experience" became the two kinds of chain that run through distrust, and "leads them not to use it" became "setting it aside for that part of the work". The ideas chain is a hedged wish, not an experience, and the drafting worker still uses AI on earlier drafts. "Trust ... is earned one output at a time ... rather than given in advance" was narrowed to what the section shows: accepting a particular output follows a test, while handing over whole tasks or a bigger role does not (the new-project chain and the trust-role chain go against the old wording). "not distrust of the output at all" became "not a fault in the output", because the coding definition treats refusing output one would not understand or control as acting on distrust, so the old wording contradicted the codebook. "accepting output or handing AI more work" became "relying on AI, by handing it whole tasks or a bigger role", because under the stance definition, accepting after a check counts as checks, not relies. 5. The second paragraph's description of method was cut to a one-line lead-in to the table. 6. Table: added the writer's newly coded chain, AI produces output → Checks the output → Output lacks human depth → Does not use the output. 7. Errors section, writer. "who asks AI only for feedback on blog articles" overstated: the writer also uses AI for admin, taxes and rewording editing feedback (creativity_0104). It now reads "lets AI give only feedback on their own writing, and only on blog articles", and "sometimes" was added ("Sometimes, it will tell me..."). 8. Errors section, developer. "expects AI to go wrong" became "says that ... AI may not fit in well", which keeps the hedge. 9. Depth section, first sentence. "did not use it" became "set it aside for that part of the work". 10. Depth section, perspectives. Now quotes the writer weighing the suggestions [chains.s2ers] and keeps "often" and "Most of the time". 11. Depth section, investor-relations worker. Now "prefers to do", and adds their own "I collaborate back and forth during the drafting process." (work_0860), because only the final draft is kept from AI. 12. Depth section, last sentence. "no check can rescue the output, so the person keeps the work" was corrected, because the writer weighs each suggestion and the worker uses AI on earlier drafts. It now reads: weighed and then mostly left unused, against checked and then used. 13-14. "Trust is earned output by output" became the heading "Accepting an output follows a test of it", and the first sentence was narrowed the same way. "The chains that end in accepting AI's output pass through a test" was wrong for the new-project and trust-role chains. 15. "A test ... appears in" now reads 4 of 4 after the added row and names the writer's case. 16. Added a sentence naming the accounts that go against the finding (chains.ck669, chains.8f26y, chains.59vy5). 17. "as a link to the future" became "as an expectation for the future". 18. "The nearest thing to trust given in advance ... gives 'AI complete tasks'" became "tends 'to give AI complete tasks'". 19. "Two interviewees keep work from AI" became "Two interviewees tell chains in which they keep work from AI". 20. Scientist, "needs" became "needed" (past: "at this point, I needed"). 21. Scientist, "spends the time" became "needs to 'spend the time'" ("I need to spend the time"). 22. Developer, "requirements are 'business related'" became "driven by data that is 'often business related'". 23. "These chains end where distrust would ... but they would not change if the output were better" was cut. The counterfactual is in no document, and the codebook counts these as acting on distrust. The sentence now says only that their reasons are understanding or control. 24. Cut "That matters for anyone reading ..." as padding. 25. Limitations, second item. It now says the question was the interviewer's, and "show what the person does when asked" (which overstated) became "told when asked, not raised unprompted". 26. Limitations: removed the old item on understanding and control "as distrust". After sentences 3 and 23 the finding no longer claims these are not distrust, and the item's claim that holding back would become "more common" did not follow from a count that is already 4 of 4. It is replaced by the narrow-judgement line below. 27. Limitations: added two narrow-judgement lines (below).

Narrow judgements
  • Opening and section on accepting an output: "accepting a particular output follows a test; relying on AI rests on the task or on trust expected to grow". This rests on the stance given to each chain and on chains by stance[relies] (2 of 4). The developer's new-project chain (chains.8f26y, "I tend to give AI complete tasks, but I always review AI's work and decide accordingly") is a fair call either way: relies (handing over complete tasks) or checks (each piece of work reviewed). Coded checks, relies would be the investor-relations worker alone, and only as an expectation. Added to Limitations. The investor-relations worker's trust-role chain (chains.59vy5) is general and only expected, but the definition of relies explicitly counts "expecting AI to take a bigger role as trust grows", so no line.
  • "Some holding back is not about the output's quality". This rests on the factor reading of two chains: Own understanding needed (science_0001) and Own control needed (work_0195), each 1 of 4 in factors by cause item and effect item. The scientist's reason ("I would not understand it as well") cannot fairly be read as a fault in the output. The developer's "I feel I have better control over the data" could fairly be read as distrust of what AI would do with business data; read that way, the finding rests on the scientist alone. Added to Limitations.
  • "Errors lead to checking, not to stopping". This rests on placing three chains under Output has errors (creativity_0104, work_0195, work_0860) and on their ending in checks. Read again, the writer's "mistakes which aren't really mistakes" and the developer's "may ... make thing worse" are fairly errors, and no error chain ends in holding back. The finding stands either way, so no line.
  • "Shallow output leads to not using it". This rests on factors by cause item and effect item[Output lacks human depth,Does not use the output] (2 of 4) and on coding the writer's perspectives chain as holds back. That chain could fairly be checks (the writer considers each suggestion), but the ideas chain and the investor-relations worker's final-draft chain keep the finding at 2 of 4. It would stand, so no line.
  • No judge step or threshold applies (recount/tables.md gives no margins), so the lines name the cells they rest on.

The record

This is run 2026-10-20-x69q. Its ID stays the same through every revision, and the zip's fingerprint, below, names this version of it. This report was created directly from the run's zip, the-evaluator-asked-what-leads-interviewees-to-2026-10-20-21aa674b.zip. The zip's fingerprint is 21aa674b3ce9939c678e8bc09bfbfe116a04466406ad2be2abc9e43a73db58b1. It is the SHA-256 of the zip's SHA256SUMS file, which lists the SHA-256 of every other file in the zip, so changing any file breaks the match. To check that the zip is the one this report was created from, unzip it, run sha256sum -c SHA256SUMS, then sha256sum SHA256SUMS, and compare the result with the fingerprint above. Rubicon's own fixed rules check the rest: python verify.py <report> <zip> recounts every number from the coded passages in the zip, creates the report again from the zip and says whether it matches this one exactly.

Every interview at a glance

Each row is one interview, each column a reason as the coding defines it. Click a mark to read the passages.

Anthropic/AnthropicInterviewer
🔥 Carbon cost of this report: Between 0.0022 and 0.17 km driven in an average UK family car

Making this report used between 2.9 and 222 watt-hours of electricity, which is the carbon of driving an average UK family car for that distance. A ballpark only: Anthropic publishes no energy figures, and the run, made in the evaluator's own Claude, records no token counts, so the estimate is made from the run's size. The 4 documents were read in full once by each of the 1 coding steps, by the second coder and by the check, and the writer read the tables; everything the run wrote counts as output. That comes to about 46,680 tokens read and 20,327 written, at 4 characters a token.

  • High: 0.39 Wh per thousand tokens read and 1.95 per thousand written, with what was read counted 10 times over for the agent's own overhead. Simon P. Couch, 'Electricity use of AI coding agents', 20 January 2026, https://simonpcouch.com/blog/2026-01-20-cc-impact/: about 390 Wh per million input tokens and 1,950 per million output tokens for Claude Opus, back-solved from Epoch AI's estimate for a 100,000-token prompt with output at five times the input rate, as Anthropic's prices are. The author calls them napkin maths. Our assumption, not a measurement: an agent re-sends its instructions, earlier turns and tool output with every request, so it reads far more than the documents themselves. The low case counts only what was read and written; the high case counts what was read ten times over.
  • Low: the same rates divided by 20, what was read counted once. Oviedo et al., 'Energy use of AI inference, efficiency pathways, and test-time scaling', arXiv:2509.20241: a median 0.31 Wh per query for frontier models, and widely cited estimates overstated by 4 to 20 times. The low case takes the high case's rates divided by the top of that range.
  • 0.124 kg CO2e per kWh. DESNZ greenhouse gas reporting conversion factors, UK grid electricity, 2024.
  • 0.16601 kg CO2e per km. DESNZ greenhouse gas reporting conversion factors, average car of unknown fuel, 2024.

Made with Rubicon, free for Claude: rubicon.causalmap.app