How Rubicon compares with what evaluators use now

What we compare on

Evaluation questions are usually fuzzy: “Was the programme effective?” The heart of Rubicon is turning a question like that into ones the evidence can answer, and then into evaluative judgements against rubrics. So that is the first thing we ask of every tool, and the one that matters most.

• Fuzzy question to answerable ones. Does the tool help break a broad question into smaller questions that a reader could answer from the evidence?
  • The breakdown can be extensive and creative. A workflow can sample documents, code them, count, compare groups and judge, in as many steps as the question needs. A two-step workflow is a perfectly good answer to a two-step question.
  • Rubrics can come in at several points: to decide whether a passage belongs in a category, whether a count is large enough to mean anything, and what overall verdict the pieces add up to.

Then three more questions, and one optional extra.

• Judgement against a rubric. Can the evaluator set levels or thresholds, at more than one stage, and does the tool judge the evidence against them?
• Quotes located in the source. Is every quotation checked to exist in the source text, and dropped if it does not?
  • Two different things go by “checked”. Linking a quotation to the passage it came from, so a reader can click through and look, is now a commodity: Anthropic, Google, AWS and Microsoft all ship it as a platform feature. Rejecting a quotation automatically, by default, when it does not match, so a plausible-sounding fabrication never reaches the reader, is what we did not find described on any of these tools’ own pages, ours excepted.
• Reconstructibility. Could someone given only what the evaluator hands over rebuild the workflow and run it again?
  • Reruns of an AI analysis will not match exactly, and Rubicon does not claim they will. Reproducibility of the numbers is a close cousin of what we claim. Reconstructibility concerns the method: a second person does recognisably the same thing to the same material, for the same stated reasons. See the open format.
• Optional: a plan fixed before the run. Can the steps and the standard be written down, dated and versioned before the evidence is read, if the evaluator or the commissioner wants that?
  • Many evaluators prefer to explore first, and a tool that supports this option is not better for it. We list it because some commissioners ask for it.

Where a page below says a feature is “not described”, we looked at the vendor’s public pages and did not find it. That is weaker than “does not have it”. Features change, and vendors document some of them where we did not look.

The sections headed “Compared with” set Rubicon beside one alternative each.

What Rubicon does, and what it has not yet done

Rubicon is in beta and has not yet been used on a live evaluation. This describes what it is built to do.

• Ruby, the assistant, does not answer a fuzzy question as put. She helps the evaluator break it into parts that can be answered, and says where the breakdown turns on a choice that only the evaluator or their stakeholders can make.
• The workflow can be as extensive as the question needs. Steps sample documents, code passages, count, compare and judge, and the evaluator and Ruby can be inventive about how to combine them.
• A rubric is written by a person, and the AI applies it. It can apply one at several stages: to a passage, to a count, and to an overall verdict.
• Optionally, the evaluator and the commissioner agree the steps and the rubric in advance and register them. Registering records what was decided before the evidence was read, and does not freeze the workflow: every revision to a step, rubric or threshold is dated and versioned, and earlier versions stay readable.
• Rubicon drops any quotation it cannot find in the source, and counts it.
• Every run keeps the record an open format bundle is made from: the sources, the workflow with every setting it ran with, what each step produced, and the rubrics.
• Ruby is an opinionated adviser. She proposes a workflow, an expert may disagree and change it, and the record shows what was changed.
• Rubicon does not compute effect sizes, differences in differences or regressions, and it declines questions whose answer sits only in a spreadsheet.

Compared with: a chat assistant, used carefully, with your own notes

ChatGPT, Claude, Copilot and NotebookLM are the tools that evaluator guidance names most often. Used with saved chats, copied prompts and a log in Word or Excel, this is a reasonable choice for a small evidence set, one evaluator and a client who accepts a hand-kept record.

• What the guidance asks for. The UK Evaluation Society’s good practice guidelines (November 2025) ask evaluators to document how AI was used so that a reader can scrutinise it, to make findings traceable to the evidence, and to verify AI-generated content. The means are left to the evaluator, and a careful log meets them for many jobs.
• What evaluators do in practice.
  • Itad describes keeping records of prompts, outputs and verification steps, and reviewing a sample by hand.
  • Julian King published the full transcripts of a rubric exercise with Perplexity and got different conclusions from a one-minute run and a one-hour run.
• What the vendors document as captured.
  • The ChatGPT export holds chat history and account data, and the Claude export holds conversation data.
  • Neither page describes the model version or a time for each message.
  • Neither describes a record of the project instructions or uploaded files as they stood at the time of the run.
  • EvalCommunity’s NotebookLM tutorial says prompts are not saved, and suggests an external log.
  • Microsoft’s audit log for Copilot is the fullest we found, listing the files each interaction accessed. A tenant administrator reads it, and the page is inconsistent about whether the model name is recorded.
• Quotes and rubrics.
  • OpenAI’s help page lists fabricated quotes among known failures, so checking quotes is the evaluator’s job.
  • OpenAI’s file upload FAQ lists applying a rubric from one document to another as a use. A rubric that is set, dated and versioned is not described.
• Where Rubicon differs. The plan is a dated, versioned object rather than a habit of the evaluator. Rubicon checks the quotes. The bundle lets a second person rebuild the workflow without having watched the chats.

Compared with: qualitative analysis software (NVivo, ATLAS.ti, MAXQDA, Dedoose)

These are the established tools, with mature manual coding, queries and mixed-methods support. They are built around the coded text: you bring the material and code it. Rubicon starts from the evaluation questions, and coding is one kind of step among several, a means to answering them. All four export projects in the open REFI-QDA exchange standard, which moves sources, codes and coded passages between tools.

• What the standard leaves behind. NVivo’s documentation lists queries and matrices as not transferred, and MAXQDA’s lists AI summaries and paraphrases. The coding travels. Neither page lists a rubric or a plan for the analysis among the items exported.
• ATLAS.ti is the closest of the four.
  • Its Intentional AI Coding takes a research question, proposes sub-questions and code categories, and lets the analyst edit them before it runs.
  • Saving that plan as a dated record is not described.
  • Its own page says results may differ each time for the same material.
  • Its MCP server (26.1, June 2026) lets an outside AI assistant read a project and write codes and memos back with the analyst’s permission. A record of the prompts and model used is not described.
• MAXQDA applies a code definition written beforehand in a code memo, and adds a comment explaining each coded segment (manual). The same page treats exact reproducibility with caution and points to transparency and an audit trail instead, which is close to our position.
• NVivo offers an AI Assistant for summaries, suggested child codes and sentiment coding. Its audit-trail claim is about coding actions in general, and prompts and models per AI run are not described.
• Dedoose describes keyword-based auto-coding and no language-model features.
• Across the four, a workflow that runs from an evaluation question through several kinds of step to a judgement, rubrics, and a check that AI quotations exist in the source are not described on their public pages. Criteria dated before the evidence is read are not described either.
• When to prefer them. When the analyst wants to code by hand with software they already hold, and the AI is a helper.

Compared with: AI-first qualitative tools (Evidano, Delve, Thematic, Dovetail)

These tools start from the text: you upload it and the AI proposes or applies codes and themes. Rubicon starts from the evaluation questions. Coding is a means, one step in a workflow that also samples, counts, compares and judges, and it is chosen because the question calls for it.

• Evidano (ailyze.com now redirects to it) is the most relevant to evaluators.
  • Its case studies name UNICEF, UNDP, UNESCO, WFP and the African Development Bank.
  • Users can supply their own codebook, and the codebook has version history.
  • Themes link to the source passage, and its quote traceability page tells the user to treat a quote as unverified if they cannot land on the passage. Checking is left to the user.
  • Prompts and models for each run, rubrics with levels, and an open handover format are not described.
• Delve is a coding workbench where AI applies your codebook and writes a memo giving its reasons for each code. Its advice is to double-check AI outputs.
• Thematic is customer-feedback analytics for large volumes of comments, and its own article on accuracy says 100% accuracy does not exist because analysis is subjective.
• Dovetail is a research repository for product teams. Its whole-project export is encrypted and cannot be opened outside Dovetail.
• When to prefer them. For a fast first pass over a large body of text, when the aim is to see what themes are in it rather than to answer set evaluation questions, or, in Thematic’s case, for a continuing stream of feedback.

Compared with: Sopact Sense

Sopact Sense is the closest to Rubicon by intended buyer: foundations, nonprofits and evaluation teams.

• It scores submissions against rubric criteria and links scores to evidence, and it collects the data in the same system.
• Its guidance pages advise finalising criteria before review, and recording who approved any change to a rubric and which entries were affected (guidance). A product feature that locks and dates the rubric is not described.
• Checking quotations against the source is not described. Its guidance says to verify quoted passages.
• The same guidance says a defined rubric does not guarantee identical AI outputs, which matches our position.
• When to prefer it. When collecting the data, running forms and following participants over time matter as much as analysing the text.

Compared with: tools for drafting rubrics and evaluation matrices

EvalCommunity Academy’s Rubric Design Studio uses AI to unpack an evaluation question into sub-questions, draft criteria across four performance levels and draft justifications.

• It works on pasted sample evidence, and its output is copied or saved as a PDF. A saved, versioned workflow run over the whole evidence set is not described.
• Its Evaluation Matrix Builder is a template that uses no AI.
• How it relates to Rubicon. These help to draft a rubric. With Rubicon, the evaluator runs the agreed rubric over the evidence and keeps the record.

Compared with: working without software

Scoring evidence against a rubric by hand in Word or Excel, or hiring an evaluator to do it, is fully auditable when done carefully. It is also the slowest option for a large body of text, and we do not have any cost figures to offer for it.

The objections, and where Rubicon stands

• Evaluative judgement should stay with people. Julian King argues that judgements about human affairs should remain human responsibilities. In Rubicon a person declares what counts as good, and the AI applies that standard to the evidence.
• Reruns differ. ATLAS.ti and MAXQDA say so on their own pages. Rubicon says so too, and records the steps so that a difference between runs can be seen and traced to where it arose.
• Confident output without a method behind it. The UK Evaluation Society’s guidelines and the UN Evaluation Group’s principles both ask for transparency and human control. Rubicon is one way to evidence them. A careful log is another.

A test to run on any tool, including ours

Before delivering a report, give a competent stranger, or an AI, only what you would hand to the client. Ask what was done, on which sources, against which standard, and in what order. Where they cannot say, the gap is in what was recorded. The test does not need any particular software, and the open format describes a bundle that passes it.