Grounding Inspector: Does Your LLM Know When It's Lying?

Grounding Inspector checks each claim in an AI-generated summary against its source document, and shows which claims are grounded, partially grounded or unsupported, claim by claim, not as a single document-wide verdict.

At the last few AI presentations I've been to, someone in Q&A always asks a version of the same question: you're told you can use AI, but you're still accountable for what it produces. How do you check that without re-reading the entire document yourself?

AI-generated summaries of long documents can state things that sound entirely plausible but aren't actually backed by the source. With no way to check individual claims, you either trust the summary blindly or manually re-verify everything, which defeats the point of using AI at all.

Architecture

Five stages. One document pair in (source document, AI output), claim-level verdicts out.

1. Decompose: split the AI output into individual, atomic claims
2. Verify: score each claim against the full source document using MiniCheck, an entailment model: does the source text entail the claim, contradict it or say nothing about it? No retrieval step narrows the search first, so no chunk is silently skipped
3. Label: aggregate the sub-claim verdicts into grounded, partial or unsupported for the displayed claim
4. Localise: map grounded and partial claims back to their source span and page number
5. Inspect: browse the result in a two-pane viewer, claims on the left, source on the right, click a claim to see its evidence

No custom retrieval index, no vector store. Every chunk of the document is scored against each claim, not just the one a retriever thinks is relevant.

Implementation

Decompose-then-verify over hierarchical RAG

Retrieval optimises for finding the most relevant chunk. Grounding evaluation has to check every claim against everything, so that we don't incorrectly flag a claim as "unsupported" just because the RAG didn't retrieve the evidence.

def verify_subclaim(subclaim: str, doc_chunks: list[str], scorer) -> tuple[bool, float]:
    """MiniCheck: score sub-claim against every chunk, max-pool. No retrieval gate."""
    if not doc_chunks:
        raise ValueError("doc_chunks must not be empty")
    labels, probs, *_ = scorer.score(docs=doc_chunks, claims=[subclaim] * len(doc_chunks))
    best = max(probs) if probs else 0.0
    return best >= 0.5, float(best)

Every chunk gets scored, the best match wins against MiniCheck's own default 0.5 threshold, not a value tuned for this project. That trades cost and latency at longer document lengths for exhaustive recall: validated on short-to-medium documents (PDS-length, a few pages). A 100+ page document would need a retrieval pre-filter, which isn't built or benchmarked here.

MiniCheck (local, free) vs Claude Haiku (cloud, paid)

The pipeline supports two verifier backends behind the same interface, an explicit recall/cost trade-off rather than a single "best" choice:

κ (Cohen's kappa), not raw accuracy, because it corrects for chance agreement with a human labeller. Both numbers come from RAGTruth, a published benchmark of labelled hallucinations, not a self-labelled test set (n=300, seed=0, drawn evenly across RAGTruth's summarisation, question-answering and data-to-text task types). Recall is the metric this tool actually optimises for: a missed hallucination costs more than a false alarm when someone downstream is accountable for the output.

The two rows aren't equally reproducible. MiniCheck's numbers are stamped into every fixture's scorecard with a pipeline commit hash, so they can be traced back to the exact code that produced them. Claude Haiku's numbers came from a real run of the project's benchmark script, but its output directory is gitignored by design (it holds API-derived data) and no longer exists locally, so that specific run isn't independently reproducible from what's in the repo today. The trade-off table above is accurate; the paper trail behind one row of it is thinner than the other.

def verify_subclaim_claude(subclaim: str, doc_chunks: list[str], client, model: str = "claude-haiku-4-5-20251001") -> bool:
    """Claude: send all chunks as one context, return True if supported."""
    context = "\n\n".join(doc_chunks)
    msg = client.messages.create(
        model=model,
        max_tokens=10,
        system=_VERIFY_SYSTEM,
        messages=[{"role": "user", "content": f"CLAIM: {subclaim}\n\nDOCUMENT CONTEXT:\n{context}"}],
    )
    return msg.content[0].text.strip().upper().startswith("SUPPORTED")

Same call signature as the MiniCheck path, swapped at runtime. Neither verifier changes the decompose, label or localise stages. The two aren't fully symmetric, though: MiniCheck scores each chunk independently and keeps the best match, while Claude Haiku sees every chunk concatenated into one prompt and answers once, trading exhaustive per-chunk scoring for a single call over the full context.

Numeric-consistency as a deterministic pre-pass

NLI models tolerate span substitution when sentence structure matches, so a transposed number can still score as entailed: MiniCheck can't do arithmetic, only pattern-match phrasing. A claim stating "$4,000" against evidence stating "$3,000" reads as structurally identical to the model. That gap doesn't get patched into the verifier; it gets caught by a separate, deterministic check that runs after labelling. That check is scoped to dollar figures specifically ($-prefixed numbers); percentages, dates and other counts aren't covered by this pre-pass.

def numeric_mismatch(claim_text: str, evidence_text: str) -> float | None:
    """Return the claim's asserted number if it does not appear anywhere in
    the evidence text's number set, else None.
 
    Conservative by design: only flags when the claim states exactly one
    number and that number is entirely absent from the evidence.
    """
    claim_nums = extract_numbers(claim_text)
    evidence_nums = extract_numbers(evidence_text)
    if len(claim_nums) != 1 or not evidence_nums:
        return None
    claim_value = claim_nums[0]
    if claim_value in evidence_nums:
        return None
    return claim_value

If a mismatch is found, the label is downgraded (grounded to unsupported, partial stays partial) and the claim gets an honest, factual rationale rather than a rationale that reads like the verifier reasoned about the number, because it didn't.

Results

The Budget Direct fixture scores 63/100: 2 claims grounded, 1 partial, 1 unsupported.

The unsupported claim states that COVID-19 cancellations are covered on the same basis as any other unforeseen event. The source PDS's pandemic-exclusion clause explicitly excludes exactly that. Click the claim and the inspector shows no matching span, because there isn't one to show, alongside a plain-language evidence note pointing at the clause that contradicts it.

That's the actual payoff: with recall 0.69 (CI 0.59-0.78) and κ=0.195 in hand, the reading burden shrinks from the whole document to the handful of flagged claims. That's a stronger position than "I read it and it seemed fine," which has no error rate at all, because nobody measured it.

What Was Hard

κ=0.195 is "slight agreement" on the Landis & Koch scale, not a number I'd want to present without the caveat above it. The Haiku alternative gets to 0.331 ("fair agreement") at roughly A$0.05 per document (Anthropic bills in USD; this is an approximate conversion), and that trade-off table exists precisely so nobody mistakes the free default for the accurate one.

MiniCheck also doesn't do arithmetic, which is why numeric-consistency exists as a bolt-on rather than something the verifier itself catches. And the RAGTruth numbers are a benchmark-distribution result: measured across summarisation, question-answering and data-to-text tasks, not on insurance, legal or regulatory text specifically. The travel-insurance PDS fixtures shown here are illustrative, not a validated domain.

Entity and citation substitution is the other known gap, and it's the same structural issue as the numeric one: an NLI verifier tolerates a substituted name or citation as long as the sentence shape matches. That's not unique to this build. It's a field-wide limitation with its own emerging reference architectures (Proof-Carrying Numbers, HalluGraph), named here rather than hidden.

A third gap sits above both of those: grounding only evaluates claims that are present in the output. It has no mechanism to flag what should have been said but wasn't. NOHARM, a Stanford/Harvard ARISE Network benchmark of 31 LLMs on medical consultation tasks, found errors of omission account for more than 80% of severe errors, not commission. Fact-checking a claim against a source is a different problem to noticing a claim's absence, and no amount of tuning the verifier closes that gap; it's structural to the paradigm.

On "why not just use an existing eval harness": Grounding Inspector ships as a custom UK AISI Inspect Scorer, extending Inspect's task runner rather than competing with it.

Where It Goes Next

The pipeline is a pattern, not just a travel-insurance tool. The same decompose-then-verify shape applies anywhere a long document and an AI-generated output need to be checked claim by claim.

Regulatory submissions (TGA AusPARs, similar public disclosure corpora) and legislative or policy text are the two most direct extensions: same accountability problem, different document type.

The legislative extension isn't hypothetical. The Dept of Finance's GovCMS DXP team is already building this whole-of-government: search.gov.au (alpha) ingests all in-force Australian legislation plus content from roughly 5,000 government websites into one semantically chunked, authority-graphed corpus. A verification stream they call "Scrutiny" checks AI-generated answers against that corpus using a hierarchical multi-agent review pipeline, the same decompose-then-verify shape as this tool, running at whole-of-government scale on GovAI. (Government content is AI food, APS Digital Profession Innovation Month, July 2026.)

Decomposition makes the verification unit small enough to audit, and MiniCheck's entailment judgement is interpretable: you can read why a claim was flagged. None of that was built to satisfy an external framework, but it happens to line up with one: the IMDA's Model AI Governance Framework for Agentic AI (January 2026) names explainability and auditability as requirements for responsible deployment. It's what defensible engineering looks like anyway.

How does this compare internationally?

Australia isn't alone in treating government content as AI's training and retrieval diet, but the shape of each country's response differs. The UK's GOV.UK Chat is the most shipped comparator: a Claude-based RAG chatbot live across the GOV.UK app, answering from a single 700,000-page domain, with accuracy climbing from 76% to 90% across its pilots. Singapore's Pair Search indexes legislation, court judgments and parliamentary debates for public officers, and GovText runs whole-of-government topic modelling. Estonia's Bürokratt is the most mature cross-agency service-delivery chatbot ecosystem, now extending to digital identities for AI agents. None of these, as far as public documentation shows, pairs an authority graph (which agency is the legal source of truth for a given claim) with claim-level verification the way search.gov.au's Find/Align/Verify streams do. GOV.UK Chat ships a live product; it doesn't appear to run a per-claim grounding check comparable to decompose-then-verify. That's the gap this tool, and search.gov.au's Scrutiny pipeline, both sit in.

AI Tools

Claude Code was used to build the pipeline and viewer, and Claude was used to draft this post.

Code and Further Reading

The full source is on GitHub (MIT).

The engine is Python with MiniCheck; the viewer is Hono and Vue 3.

The live demo is gated for now. Ping me on LinkedIn and I'll get you access.