Did the Warm Homes Visits help households keep warm for less?
Question Did the Warm Homes Visits help households keep warm for less, well enough to justify funding the scheme for another year?
Answered against the evaluator's rubric, agreed on 7 October 2026 before any interview was read. Criterion A: the householder describes at least one change the visit brought about that was still in place or still in use at the interview (draught strips or reflector panels still fitted, still on the tariff they switched to, heating controls still used as advised, loft or cavity insulation installed through the referral, or a Warm Home Discount received after the adviser helped them apply); not met where no such change is described, where the change has since stopped, or where the householder puts it down to something other than the visit. Criterion B: the householder says the home has been warmer, or bills or top-ups lower, and links this to the visit or to something it brought about, in their own words or when asked; not met where neither is described, or where one is described but put down to something else, such as a mild winter, a fall in the price cap or other help, or not linked to the visit. Both criteria must be met in the same household. Good enough: more than half of the households interviewed meet both. Not good enough: half or fewer meet both. Cannot say: the interviews did not ask about changes since the visit.
Synthetic documents. All documents were written by Claude to test the method. They are not records of real people, and nothing in this report is evidence about anyone.
No: on the evaluator's rubric the scheme is not good enough to justify another year's funding. Of the households interviewed, () describes both a change the visit brought about that was still in place at the interview and lower bills linked to the visit. The rubric asks for more than half. In the other households (, , ), what the visit fitted did not stay in place, nothing could be fitted, or the warmth came from somewhere else. The verdict would stand if any one of those three were read the other way, since that would still be half, not more. Every interview asked what had changed since the visit, so the rubric's condition for giving no verdict does not arise. The account that goes against the verdict is 's: "It cost me nothing and it's saved me money." ()
4 interviews in 3 groups · 8 passages coded · 10 October 2026
Verification in this report:
4 numbers, every one computed directly from the coded passages rather than written by the AI. Click any number to see who it counts.
8 quotations, every one found word for word in its interview. Click an interview's ID beside a quotation to read it in place.
16 findings and 3 high-level findings, every one checked against the interviews after they were coded a second time.
Changes that lasted
Only households () describes a change from the visit still in place about six months later: the adviser switched its tariff during the visit, and the householder is still on it: "I am, yeah. It's going on for six months now." ()
The other households describe no lasting change, for different reasons:
: draught strips were fitted but stopped the back door closing: "My son pulled them off before Christmas." () The adviser gave her a number to ring about insulation, and she did not ring it because she "didn't want workmen tramping through." ()
: in a privately rented house the landlord refused any fittings: "The landlord wouldn't have anything fitted." () The householder decided against switching tariff because she was on a prepayment meter and unsure about it.
: reflector panels came down soon after the visit: "They fell down, though. Within a few weeks, behind the radiator, they just slid off. I never put them back." () The same householder says the heating controls advice was "stuff I already knew." ()
Warmer homes and lower bills
The household that switched tariff () links lower bills to the switch: "All that's different is the company, so it's got to be that." ()
The other households do not link a warmer home or lower bills to the visit:
remains as cold as before: "It's as cold as it ever was." () Her bills are "Higher, if anything." (), though she does not know whether from prices or from using more.
praises the adviser warmly ("I felt so much less on my own with it all." ()), but when asked whether the house has been any different she says: "If I'm honest, no. It's still cold." () Her praise is for the support, not for any change in the house.
describes a warmer flat and lower bills but puts them down to a new boiler and a mild winter: "The boiler, mainly. The housing association put a new one in, in November." () Asked whether the visit made a difference, the householder says: "Not really, no." ()
Limitations
The verdict rests on what the householders interviewed said about their own homes. It does not measure energy use, and it covers only the households interviewed, not every household visited. No householder's account of a warmer home or of lower bills was checked against bills or meter readings.
Annexes
How this report was written
What was read. Rubicon read all 4 documents in full. The counts below are of households, one document each.
The workflow x-ray beside Step by step, below, draws the same procedure, a box for each stage and what it made.
Step by step
The analysis was written down as a workflow before it was counted: what to look for, how to code it, what to count. It was then counted directly from the coded passages rather than by the AI, so anyone can check a number, and another coder can follow the same definitions and code the interviews afresh.
What was read
All 4 documents, read in full: 1 older person alone, 2 family with children, 1 working-age adult.
Coding: Lasting change
one row for each document, read whole; 4 rows.
The instruction a second coder would follow
This is an interview with a householder about six months after a Warm Homes Visit, in which an adviser checked their energy tariff and helped them switch or apply for the Warm Home Discount, fitted draught strips and radiator reflector panels, advised on heating controls, and referred them for grant-funded loft or cavity insulation. Read the whole interview and give one row for the household. Quote the passage that best shows whether a change the visit brought about was still in place or in use at the interview; where there is none, quote the passage that shows that. Code each column exactly as defined.
Asked about changes
Whether the interviewer asked the householder, in any words, what has changed since the visit or whether anything came of it. yes: such a question is asked anywhere in the interview. no: the interview never asks about changes since the visit, even if the householder mentions some unprompted.
Lasting change
Whether the householder describes at least one change the visit brought about that was still in place or still in use at the interview. yes: at least one of these, described as brought about by the visit (done by the adviser, or arranged, applied for or learned at the visit) and still in place or in use when interviewed: draught strips or radiator reflector panels still fitted; still on a tariff they switched to through the visit; heating controls (timer, thermostat, radiator valves) still used as the adviser showed; loft or cavity insulation installed through the visit's referral; or a Warm Home Discount received after the adviser helped them apply (this counts although it is paid once). no: no change brought about by the visit is described; or every such change has since stopped (strips or panels removed or fallen off and not replaced, back on the old tariff, controls set back as before, a referral not followed up); or the householder puts the change down to something other than the visit, such as the landlord or housing association, or says they did it before or without the visit. Advice the householder says they already followed before the visit is not a change. Tone does not decide it: a householder who is critical of the visit but describes such a change still in place is yes; one who praises the visit but names nothing still in place is no.
The coded passages 4 rows
My son pulled them off before Christmas.
The landlord wouldn't have anything fitted.
They fell down, though. Within a few weeks, behind the radiator, they just slid off. I never put them back.
I am, yeah. It's going on for six months now.
Coding: Warmer linked
one row for each document, read whole; 4 rows.
The instruction a second coder would follow
This is an interview with a householder about six months after a Warm Homes Visit, in which an adviser checked their energy tariff and helped them switch or apply for the Warm Home Discount, fitted draught strips and radiator reflector panels, advised on heating controls, and referred them for grant-funded loft or cavity insulation. Read the whole interview and give one row for the household. Quote the passage that best shows whether the householder links a warmer home or lower bills to the visit; where they do not, quote the passage that shows that. Code the column exactly as defined.
Warmer linked
Whether the householder says their home has been warmer, or their bills, direct debit or prepayment top-ups lower or lasting longer, and links this to the visit or to something it brought about. yes: they describe a warmer home or lower energy costs in their own case, and put it down, at least in part, to the visit or to a change it brought about (a measure fitted, a tariff switch, a Warm Home Discount, insulation arranged, controls used as shown), in their own words or by agreeing when asked. no: they describe neither a warmer home nor lower costs (feeling reassured, supported or less alone is not a warmer home); or they describe one but put it down only to something else, such as a new boiler, the landlord or housing association, a mild winter, a fall in the price cap or other help; or they do not link it to the visit when the interview gives them the chance. Tone does not decide it: grudging words that still credit the visit with lower bills are yes; warm praise for the adviser with the house still cold and bills unchanged is no.
The coded passages 4 rows
It's as cold as it ever was.
If I'm honest, no. It's still cold.
The boiler, mainly. The housing association put a new one in, in November.
All that's different is the company, so it's got to be that.
Counting: Change
Households by lasting change
The table 2 cells
Lasting change
Households
No
of 4
Yes
of 4
Counting: Warmth
Households by warmer linked
The table 2 cells
Warmer linked
Households
No
of 4
Yes
of 4
Counting: Both
Households by asked about changes and lasting change and warmer linked
The table 8 cells
Asked about changes
Lasting change
Warmer linked
Households
Yes
No
No
of 3
Yes
Yes
Yes
of 1
No
No
No
of 3
No
No
Yes
of 1
No
Yes
No
of 3
No
Yes
Yes
of 1
Yes
No
Yes
of 1
Yes
Yes
No
of 3
Judging: Verdict
The evaluator's rubric, agreed on 7 October 2026 before any interview was read.
How many of the households interviewed describe at least one lasting change made because of the visit and, in the same household, a warmer home or lower bills they link to the visit? · level met: Not good enough
Verdict: Not good enough, since no earlier rule holds (both_in_same_household is Not good enough), so Not good enough.
The verdicts the rule could reach
Good enough
More than half of the households interviewed describe at least one lasting change made because of the visit and, in the same household, a warmer home or lower bills they link to the visit.
Not good enough
Half or fewer of the households interviewed describe both.
Writing: Answer
The instruction the answer was written to
The checks that were run
8 of 8 quotations found word for word in their document.
4 numbers in the answer, every one computed directly from the coded passages rather than written by the AI; 0 written by the AI without a count behind them.
0 citations to passages that do not exist; 0 quotations not in the passage they cite.
Second coding
The same documents were coded again to the same definitions; the run does not record whether a separate coder did it, so it is not called independent here. The second coder's rows were not kept in this run's zip.
The check
A reader ruled on every place the two codings differed, and checked every sentence and quotation of the answer against the documents, correcting the coding and the answer where they were wrong. The answer above is the corrected one, recounted after the check.
The check's record check.md
Counts
Disagreements between the two coders: 0 (recode/disagreements.md: "The two coders placed every document and value alike"; recode/compare.json shows kappa 1.0 on both judged columns and 4 of 4 agreement on asked_about_changes).
Ruled for the rows: 0. Ruled for the second coder: 0. Unsettled: 0.
Rows changed: 0 of 8. Definitions changed in workflow.json: 0.
Sentences in answer.md changed, added or cut: 15 (11 changed, 3 added, the verdict section's 3 sentences cut with one moved to the top).
recount.py on the corrected answer: no faults (no bare numbers, no ids naming no count, no citations to no row, no quotation outside the documents or outside what it cites).
Rulings on the coding
There were no disagreements to rule on. All eight rows were nevertheless reread against the documents and the definitions, and each holds:
lasting_change, SYNTHETIC-B-01, no: strips removed ("My son pulled them off before Christmas."); insulation number not rung ("But I didn't ring."). Both are listed as stopped or not followed up.
lasting_change, SYNTHETIC-B-02, no: "The landlord wouldn't have anything fitted."; tariff switch not made ("So I left it."). Nothing was brought about.
lasting_change, SYNTHETIC-B-03, no: panels fell and were not replaced; controls advice was "stuff I already knew" (already followed, so not a change); "nobody has" been in touch about the referral.
lasting_change, SYNTHETIC-B-04, yes: tariff switched at the visit ("she phoned the new company from my kitchen") and still on it ("I am, yeah. It's going on for six months now.").
asked_about_changes, all four, yes: each interview opens by asking what has changed since the visit.
warmer_linked, SYNTHETIC-B-01, no: "It's as cold as it ever was."; bills "Higher, if anything."
warmer_linked, SYNTHETIC-B-02, no: "If I'm honest, no. It's still cold."; bills "About the same". Feeling "less on my own" is excluded by the definition.
warmer_linked, SYNTHETIC-B-03, no: warmer and cheaper, but put down to the boiler and a mild winter, and asked about the visit: "Not really, no."
warmer_linked, SYNTHETIC-B-04, yes: lower direct debit linked to the switch: "All that's different is the company, so it's got to be that."
Changes to answer.md
Top: the bold opening now answers the question as asked, including the funding decision ("No: ... not good enough to justify another year's funding"), rather than only naming the rubric verdict.
Top: "a warmer home or lower bills linked to the visit" narrowed to "lower bills" for the one household that meets both. SYNTHETIC-B-04 describes lower bills only ("The bills are lower."), and says nothing about a warmer home.
Top: documents named for the household that meets both (SYNTHETIC-B-04) and for those that do not (SYNTHETIC-B-01, -02, -03), per the rule that a finding names its documents.
Top, added: the verdict survives any one of the three non-meeting documents being read the other way (that would make half, and the rubric needs more than half).
Top: "Every interview asked what had changed since the visit, so the rubric's condition for giving no verdict does not arise." moved up from the cut verdict section, unchanged.
Top, added: names the account that goes against the verdict, SYNTHETIC-B-04: "It cost me nothing and it's saved me money."
Changes that lasted: "Only ... households describe" corrected to "describes", document named, and "a tariff switch the adviser made during the visit" kept in substance ("the adviser switched its tariff during the visit"), matching "she phoned the new company from my kitchen and got it sorted".
SYNTHETIC-B-01: "did not follow up the insulation referral because she did not want workmen in the house" corrected. The document shows the adviser gave her "a number to ring", not a referral made on her behalf; quotation now in her words with the id: "didn't want workmen tramping through."
SYNTHETIC-B-02: "unsure about doing so on a prepayment meter" reworded to "on a prepayment meter and unsure about it", matching "I'm on the meter, the key one that you top up, and I wasn't sure about it."
SYNTHETIC-B-03: quotation "stuff I already knew" given its document id, which it lacked.
Warmer homes, SYNTHETIC-B-04: document named.
SYNTHETIC-B-01: "with bills higher" was overstated. She says "Higher, if anything." and "I don't know whether that's the price or whether I've used more"; the sentence now says both, with the id.
SYNTHETIC-B-02: quotation "I felt so much less on my own with it all" given its id; "when asked about the house" made exact ("when asked whether the house has been any different", the interviewer's question being "Has the house been any different this winter?").
SYNTHETIC-B-03, added: the householder's answer when asked whether the visit made a difference, "Not really, no.", since the definition requires that the link to the visit be absent when the interview gives the chance.
Cut: "The verdict against the rubric" section. Its first and last sentences repeated the opening paragraph's count and verdict; its middle sentence was moved to the top.
Left as written: the heading, the three bullets' row quotations (all exact and in the cited rows), the B-02 praise sentence, and the Limits section (accurate: no bills or meter readings are in the documents).
Not added: reports of others (B-02 told her sister-in-law; B-04's neighbour "has got her name down"; B-01's guess that it "would for someone else"). None is counted, and none bears on whether these households kept warm for less.
The record
This report was drawn directly from the run's zip, did-the-warm-homes-visits-help-households-keep-warm-2026-10-10-revision-1-ea65cb3c.zip. The zip's fingerprint is ea65cb3ca0e580b64e97cea0307a5144846102de9f1e93b3c387521aca038f55. It is the SHA-256 of the zip's SHA256SUMS file, which lists the SHA-256 of every other file in the zip, so changing any file breaks the match. To check that the zip is the one this report was drawn from, unzip it, run sha256sum -c SHA256SUMS, then sha256sum SHA256SUMS, and compare the result with the fingerprint above. Rubicon's own code, which is public, checks the rest: python verify.py <report> <zip> recounts every number from the coded passages in the zip, draws the report again from the zip and says whether it matches this one exactly.
Revisions
Changes made after the first report, each recounted and checked before this version was drawn. Earlier versions keep their own files.
10 October 2026: Each table now says what it counts: every count names its unit, households, beside the number it is out of. Workflow: every tabulate step names its unit. Coding and answer unchanged. Recounted the whole workflow; no count moved. The recount also states how near each verdict is to moving (its margin), which the engine that made the earlier recount did not.
Every interview at a glance
Each row is one interview, each column a reason as the coding defines it. Click a mark to read the passages.
🔥 Carbon cost of this report: Between 0.00062 and 0.099 km driven in an average UK family car
Making this report used between 0.8 and 133 watt-hours of electricity, which is the carbon of driving an average UK family car for that distance. A ballpark only: Anthropic publishes no energy figures, and the run, made in the evaluator's own Claude, records no token counts, so the estimate is made from the run's size. The 4 documents were read in full once by each of the 2 coding steps, by the second coder and by the check, and the writer read the tables; everything the run wrote counts as output. That comes to about 15,600 tokens read and 2,800 written, at 4 characters a token. It covers the first run and its revision. The zip holds only the latest state, so each revision is counted at the low end as rereading the tables and writing the answer again, and at the high end as a whole run again.
High: 0.39 Wh per thousand tokens read and 1.95 per thousand written, with what was read counted 10 times over for the agent's own overhead. Simon P. Couch, 'Electricity use of AI coding agents', 20 January 2026, https://simonpcouch.com/blog/2026-01-20-cc-impact/: about 390 Wh per million input tokens and 1,950 per million output tokens for Claude Opus, back-solved from Epoch AI's estimate for a 100,000-token prompt with output at five times the input rate, as Anthropic's prices are. The author calls them napkin maths. Our assumption, not a measurement: an agent re-sends its instructions, earlier turns and tool output with every request, so it reads far more than the documents themselves. The low case counts only what was read and written; the high case counts what was read ten times over.
Low: the same rates divided by 20, what was read counted once. Oviedo et al., 'Energy use of AI inference, efficiency pathways, and test-time scaling', arXiv:2509.20241: a median 0.31 Wh per query for frontier models, and widely cited estimates overstated by 4 to 20 times. The low case takes the high case's rates divided by the top of that range.
0.124 kg CO2e per kWh. DESNZ greenhouse gas reporting conversion factors, UK grid electricity, 2024.
0.16601 kg CO2e per km. DESNZ greenhouse gas reporting conversion factors, average car of unknown fuel, 2024.