Checkable AI for evaluators

You used AI for your evaluation analysis.
You used a principled, transparent workflow.
You hope.

Can you prove you did? To your client? AI always produces confident-sounding results, even if you just dump your docs into some AI and cross your fingers.

If only there was a way to help you use AI systematically. And to prove you did.

“Everyone thinks they can write a good evaluation report with AI now.
But they can’t.”

“There’s still a role for evaluators if we can find the right tools.”

“I’d already agreed all the analytical steps with the client.
In writing, sort of.”

“Now it looks like I have to rework the whole thing, and nobody’s paying for that.”

“Write the analysis steps down before we’ve read the interviews?
How?”

“So we didn’t. And now nobody can say quite what we did.”

“How did we get from the interviews to the findings?
Good question.”

“A lot was assumed and not much written down. I couldn’t check it now if I tried.”

“I asked the AI for the main themes.
It found some.”

“I just can’t tell the client how it found exactly those.”

You vouch for how the analysis was done, for choosing a defensible workflow. Rubicon helps you do that, and prove you did it.

What is Rubicon?

A free, open-source plugin for your own Claude, for evaluators who want to use AI to systematically analyse text like reports and interviews. With it, you build an explicit, defensible workflow to answer each question, in as many steps as the question needs: splitting, coding, combining, comparing and judging.

  1. Bring

    Your evaluation questions, and the evidence you gathered: interviews, reports, documents.

  2. Agree

    You agree the questions with Claude before any document is read, as an experienced evaluator would with a client: how they fit together, what analysis they can share, what each answer is for, which groups to compare.

  3. Run

    You start the run, and Claude carries out each step of the workflow: coding the passages on each part of the question, say, then comparing groups and judging the counts against your rubric. Every count is made by fixed rules, out of a stated base. Every quote is, of course, checked against the source.

  4. Hand over

    For each question, a report answering it that you can click through, a Word copy, and the whole run, written in an open format that anyone can check without special software.

Rubicon helps you design the steps.
It never decides what counts as good.

See what a report looks like, and how to get the plugin

Why is Rubicon special?

AI models such as Claude, ChatGPT and Gemini are improving faster than any tool built for qualitative analysis can follow. So Rubicon does not try to out-analyse them: it uses the model you have, and improves as that model does.

But a model reporting on its own work is not something you can put your name to. Rubicon adds an open trust framework around the findings.

Three AI models, each better than the last. Below them, separately, what Rubicon adds, which stays the same whichever model is used: counted by code, every quote found, a blind second reading, a record to rerun, and an open format for any model. THE MODEL better with every release today's the next the one after RUBICON: WHAT STAYS Counted by code. Every number in the report is computed by code from the coded passages, not written by the AI. Click a number in a report to see the passages it counts.7/20Countedby code Every quote found. Each quotation is found, word for word, in its document. Click one in a report to read it in place.“ ”Every quotefound Blind second reading. A second coder codes the documents again without seeing the first coding, and a fresh reader rules on every disagreement.Blind secondreading A record to rerun. Beside each report is a zip of the whole run: the documents, the coded passages and the workflow. Somebody else can check every number from it, or do the work again.A recordto rerun Open format, any model. The record is written in an open format that anyone can read without special software, whichever company's model did the work.Open format,any model

The principle in full

Rubicon features

Agree first

  • The questionAgreed before anything is read, with ways to answer it drawn from process tracing, contribution analysis and more.
  • RubricsNot just “is it good” but “is it good enough”, agreed with your client in advance. Example: the sustainability of a training programme
  • StakeholdersPauses for them to take part, and the draft open for sensemaking.

Counted and checked

  • QuotationsEvery one double-coded, adjudicated and checked word for word against its document.
  • An x-ray of the workflowA clickable report. Open any finding to check each step behind it, from the counts to the coded passages to the documents. Word copy too.
  • RevisionsRevise or expand it later. Changes are recounted by fixed rules, every revision dated.

Seen at a glance

  • Causal maps and loopsMaps of what your sources say, where the question calls for one (example), and CLDs of the feedback loops they describe (example).
  • Tables and gridsCounts compared across groups. Every document at a glance, grouped as you ask.
  • Combined reportThe answers from one documents folder in one report: a cover, the questions, each report exactly as sealed, the annexes at the end. Word copy too. (Example)

Yours to keep

  • Free, on your own ClaudeCosts nothing beyond your own Claude subscription. Your documents go nowhere else.
  • A reconstruct-me zipThe whole run in one file, in an open format, so somebody else can rebuild it.
  • Open sourceMIT licence: file issues, suggest features, fork it and adapt it. GitHub

Work with us

  • Your workflowWe can help adapt Rubicon to it.
  • A live evaluationWe can help you use Rubicon.
  • WorkshopsFor evaluation teams, on using AI defensibly.

Want to try Rubicon?

The plugin is free. You install it from inside your Claude app. It is open source under the MIT licence, with its code on GitHub. We also run workshops and help teams apply it to live evaluations: write to hello@causalmap.app. A Rubicon app with optional all-EU/UK processing will follow.

How to install it

About the plugin

Steve Powell, Causal Map Ltd