How We Built a Citation Validator for AI-Generated Review Articles — and What It Taught Us

When a large language model writes a scholarly review article, it sounds authoritative. The prose is fluent, the structure is logical, and the citations look right. But are the citations right? Does each claim actually follow from the paper it references? Or is the model doing something more subtle — projecting what it "knows" onto whatever source happens to be nearby?

We built a system called the Citation Claim Validator (CCV) to answer that question rigorously, and then used its findings to make our AI writing agent measurably better. Here's how we got there.

The Problem: Fluent but Unfaithful

Our Review Writing Agent (RWA) generates long-form scholarly review articles from a set of references, an outline, and extracted "thoughts" from each source. Depending on what's available, those thoughts may come from a full-text chunk of the paper or just its abstract. The output reads well — but early testing revealed a pattern that's familiar to anyone working with LLMs in high-stakes domains: the model often writes claims that are plausible and topically relevant but not actually supported by the cited source.

Consider a concrete example. The RWA cited a 2025 review on digital twins in personalized medicine (Saratkar et al., "Digital twin for personalized medicine development," Front. Digit. Health, 2025) and produced the following sentence: "Blockchain-enabled audit trails and secure multi-party computation are envisioned as mechanisms to assure data provenance, consent management, and tamper-evident logging within FL-powered digital-twin ecosystems."

Sounds authoritative. But when we decomposed that sentence into individual claims and checked each one against the source, the picture fell apart. The paper's abstract — all that was available to the writing agent — mentions blockchain as a future enabler for digital twins. That part checks out. But "audit trails," "data provenance," "consent management," "tamper-evident logging," "secure multi-party computation," and "federated learning-powered ecosystems"? None of that appears in the abstract. And having checked the full paper, "secure multi-party computation" and "federated learning" aren't discussed anywhere in it — those concepts were imported wholesale from the model's general knowledge of privacy-preserving AI. The blockchain-specific terms ("audit trails," "tamper-evident logging") are at least adjacent to the paper's description of blockchain creating "unchangeable and permanent" records, but the specific technical vocabulary is the model's own elaboration. Three of the four atomic claims in that single sentence were unsupported — and not because the excerpt was too thin. The model took one grounded kernel and built a richly detailed but fabricated technical architecture around it.

This is what makes citation validation for LLM-generated text so tricky. The claims aren't random — they're plausible, topically coherent, and exactly the kind of detail a reader would expect. The model has strong priors about what a "complete" description should include, and it fills in the blanks from its training data without distinguishing between what the source says and what it knows independently.

To complicate things further, sometimes that latent knowledge is correct. In a separate manuscript, the RWA cited the Decagon framework for drug-drug interaction prediction and stated that it produces interaction scores "across 964 edge types." The source abstract describes Decagon's architecture but never mentions the number 964. A strict validation would flag this as unsupported — but the full paper (to which the agent did not have access) does report exactly 964 edge types. That number is too specific and arbitrary to be a hallucination. The agent was recalling a precise quantitative detail from the paper's results section, going beyond the abstract it was validated against.

So the challenge isn't simply "does the model make things up?" It's that the model freely blends faithful citation, legitimate latent recall, and confident confabulation — all in the same fluent prose, sometimes in the same sentence. You need a principled way to tell them apart.

Designing the CCV: Two Dimensions of Validation

The CCV decomposes every citation in a manuscript into atomic claims — individual assertions that can be independently verified. A single citation might contain one or more distinct claims, each of which gets its own assessment.

But not all citations make the same kind of assertion. Before decomposing claims, the CCV classifies each citation's intent: is it a specific claim (an empirical result, named method, or quantitative assertion that demands full scrutiny), background (general context where topical relevance is sufficient), or a review anchor (a reference to a review representing a body of work)? Citing a paper to say "GWAS are widely used" and citing it to say "this model achieved 94% accuracy on 20,000 drug pairs" are fundamentally different acts — the evidentiary bar should match the strength of the assertion. In our manuscript, specific claims made up nearly two-thirds of all citations and were consistently the lowest-scoring category.

Each claim is then evaluated along two independent dimensions:

Source Grounding: Does the cited source say this?

The first dimension, source_grounded, checks whether the claim is actually supported by the source material provided — either an abstract or a full-text chunk from the cited paper. This is the core citation integrity question: if you read only what the agent had access to, would you find support for this claim?

Judge Knowledge: Is this claim independently true?

The second dimension, judge_knowledge, asks a separate model to assess the claim from its own knowledge, without seeing the source. The judge can return one of three verdicts: confirmed, contradicted, or unknown.

This second dimension exists precisely because of cases like the Decagon example. When a claim is not source-grounded but judge-confirmed, we have a signal that the agent may be drawing on legitimate knowledge that simply wasn't in the excerpt. These claims get a score of 1.0 rather than being penalized — a deliberate design choice that avoids punishing correct recall. The judge isn't infallible, of course. It may confirm claims it shouldn't, miss claims it could verify, or return "unknown" for reasons that have more to do with its own training gaps than the actual knowability of the claim. But in practice, the two-dimensional approach produces meaningfully better signal than source grounding alone — it's a useful heuristic, not an oracle.

Combining the Dimensions

The two dimensions produce a natural matrix:

For claims that are not source-grounded and not judge-confirmed, the CCV classifies a failure mode — the type of error the agent made. This taxonomy turned out to be essential for understanding what to improve.

The Baseline: A Sobering Picture

We ran the CCV against a baseline manuscript — a review article on AI in personalized medicine generated by our RWA. The results were less than encouraging.

Each claim receives a score on a [−0.5, 1.0] scale: 1.0 for a claim that is source-grounded or independently confirmed by the judge, 0.0 for an ungrounded claim the judge can't verify, and −0.5 for the worst case — ungrounded and contradicted by independent knowledge. The asymmetry is deliberate: a confidently wrong citation should hurt more than an unverifiable one.

The mean citation score for the baseline was 0.614. Nearly half of all claims (49%) were not grounded in their cited source. The score for specific evidentiary claims — the ones that matter most for scholarly credibility — was even lower at 0.548.

The CCV also revealed a pronounced positional bias: citation quality degraded as the agent progressed through each section. Claims in the first quartile of a section scored 0.661; by the fourth quartile, that dropped to 0.468. The model appeared to use its strongest citation matches early and fill later paragraphs with weaker ones, or to lapse into aspirational, synthesizing prose at the end of sections.

The Failure Taxonomy: Not All Errors Are Equal

But the aggregate numbers only tell you how much the model fails, not how it fails. Through systematic review of the hundreds of failed claims in our baseline, we identified seven distinct failure modes — and understanding the taxonomy turned out to be the key to designing effective interventions.

Fabricated specifics was the dominant failure mode at 32% of baseline problems. The agent invents details that sound plausible — system components, architectural features, application domains — filling in gaps from its training data. The blockchain elaboration from our opening example is a textbook case.

Source mismatch (23%) represents cases where the cited paper simply doesn't address the claim's topic. An AI fairness review gets cited for clinical triage performance. A post-hoc explainability paper gets cited for inherently interpretable architectures. The agent appears to select citations by keyword proximity rather than by verifying actual claim support.

Overgeneralization (14%) converts cautious findings into definitive ones. Three papers that each say a technique "could be more powerful" get aggregated into "numerous studies demonstrate markedly higher accuracy."

Editorial embellishment (11%) adds interpretive framing — "cornerstone," "transformative," "critical" — that the source never states.

Speculative synthesis (9%) creates novel claims by combining separate points from a source in ways the source doesn't support.

Aspiration as achievement (6%) was small in count but consequentially important. Review articles describing "promise" and "potential" get cited as evidence that systems are "deployed" or "have begun to influence" clinical decisions. In a medical review, this error can mislead readers about the maturity of a field.

Insufficient source (6%) reflects cases where the source excerpt was simply too brief to verify the claim — a retrieval issue rather than a writing quality issue.

Closing the Loop: Targeted Prompt Interventions

With the CCV's diagnosis in hand, we designed targeted prompt modifications aimed at the six highest-priority failure modes. The key principle was specificity — rather than generic instructions to "cite accurately," we gave the writing agent concrete rules informed by the actual failure patterns.

To understand where the prompt changes go, it helps to know the RWA's basic architecture. The RWA operates downstream of two other agents: a ReferenceFinderAgent that identifies and retrieves relevant papers, and an OutlineAgent that produces a structured section-by-section outline. Given those inputs, the RWA pipeline works in stages: first, it fetches reference abstracts and extracts "thoughts" — key claims and findings — from each one, assigning them to the relevant sections of the outline. Those thoughts then drive targeted retrieval against full-text chunks, from which a second round of thought extraction captures more specific details. Finally, the writing agent composes each section using all the extracted thoughts as its source material. This means there are two natural intervention points: the thought extraction prompts (which shape what the writer sees) and the writing prompts themselves (which shape what the writer does with it). We modified both.

We added a Citation Discipline block to the writing prompt with rules that mapped directly to the top failure modes:

• Against fabricated specifics: Only attribute specific details — system components, metrics, architectural features — that appear in the source excerpt. If the source says "blockchain can enhance digital twins," write that; don't elaborate it into audit trails, consent management, and tamper-evident logging.
• Against source mismatch: Before citing a source for a claim, verify that the source actually addresses the claim's topic — not just a related keyword. A paper on explainability is not a citation for interpretability.
• Against hedging erosion: Preserve the source's epistemic stance. If the source says "may improve" or "shows promise," don't upgrade to "demonstrates" or "has begun to influence."

We added a separate Source Fidelity block to the thought extraction prompt, tightening how source material flows into the writing stage. The extraction step now explicitly instructs the model to capture the source's own hedging language and scope limitations, so the writing agent receives more faithful raw material to begin with.

Finally, we added guidance addressing the positional degradation pattern, explicitly instructing the agent to maintain consistent citation quality throughout each section rather than front-loading its strongest material.

Crucially, we did not tell the agent to cite less or be more conservative. The goal was better grounding, not less ambition.

The Results

We re-ran the RWA with modified prompts on the same outline and reference set, then re-ran the CCV with identical configuration. The only variable was the prompt changes described above.

The headline numbers told a clear story:

The mean claim score jumped from 0.614 to 0.817. Source-grounded claims rose from 51% to 76%. Problematic claims were cut by more than half, from 382 to 172. And the specific-claim score — the hardest category — improved by 0.260 points.

Failure Modes: Targeted Reductions

The failure mode comparison showed that the interventions hit their intended targets:

Fabricated specifics dropped 72%, from 123 to 35 instances. The agent stopped inventing system architectures, component lists, and application domains. Source mismatch fell 73%, from 86 to 23. Aspiration as achievement was nearly eliminated — from 24 instances to just 1. (The remaining 35 fabricated_specifics deserve a closer look — more on that below.)

Overgeneralization, editorial embellishment, and speculative synthesis also declined in absolute terms, though more modestly. These are subtler errors — upgrading "capability" to "demonstrated improvement," or adding "cornerstone" as framing — and proved more resistant to prompt-level correction. They now make up a larger share of the remaining failures, but the remaining failures are genuinely less severe than before.

Positional Bias: Gone

Perhaps the most satisfying result was the elimination of positional degradation. Where the baseline showed a steady decline from Q1 (0.661) to Q4 (0.468), the revised manuscript showed essentially flat quality across all quartiles — hovering around 0.81 throughout.

The explicit instruction to maintain consistent quality throughout each section appears to have worked. Concluding paragraphs now receive the same citation discipline as opening ones.

What Didn't Change

The revised manuscript actually had more citations than the baseline (380 vs 344), confirming the prompt changes didn't make the agent citation-averse.

Right Fact, Wrong Paper: The Missing Dimension

The 35 remaining fabricated_specifics in the revised manuscript sound alarming — but manual review reveals that many aren't fabrications at all. They expose a subtler problem that the CCV's current two-dimensional design can't fully resolve.

Consider this case from a revised manuscript (that is, one run after the changes we made to RWA). The writing agent cites a multi-omics review article and claims that DeepDRK — a drug repurposing framework discussed in that review — was trained on "more than 20,000 cell-line–drug pairs." The review describes DeepDRK's architecture but never mentions that number. The CCV correctly flags the claim as not source-grounded. The judge model can't independently confirm the figure, so it's classified as fabricated_specifics. But the original DeepDRK paper reports exactly that figure in its abstract. The claim is correct — it's just attributed to the wrong source.

This looks superficially like the Decagon example from earlier, but it's a fundamentally different problem. With Decagon, the agent cited the right paper and recalled a detail from its full text that wasn't in the abstract — a CCV limitation caused by validating against a thin excerpt. With DeepDRK, the agent cited a secondary source (a review) for a detail that belongs to the primary source (the original paper). That's a citation targeting problem upstream of the CCV — the reference finder selected a review article when the original paper would have been the correct citation.

The CCV's two dimensions — source grounding and judge knowledge — can distinguish "supported by this source" from "true in the world." But they can't distinguish "right paper, thin excerpt" from "wrong paper, correct fact recalled from elsewhere." Both show up as not-grounded, and the judge either knows or it doesn't. If the judge had confirmed the DeepDRK claim, the current design would have scored it 1.0 and moved on — silently passing a citation that would lead a reader to a paper that doesn't contain the claimed detail.

This points toward a possible third dimension of validation: citation appropriateness. Is this the right paper to cite for this claim, independent of whether the claim is true? That's a harder question — it requires reasoning about the relationship between a claim and the broader literature, not just checking one source. But it would let the CCV distinguish genuinely fabricated details from misattributed correct ones, and flag citation targeting problems that the current system either misclassifies or misses entirely.

For now, the practical implication is that the 35 remaining fabricated_specifics is a ceiling, not a floor. That bucket almost certainly overstates how many claims are truly invented. It rolls together at least three distinct situations — genuine fabrication, correct facts cited against the wrong paper (a targeting problem upstream of the CCV), and correct recall from elsewhere in the cited paper when neither the excerpt nor the judge could corroborate it — the Decagon pattern, but without the judge's safety net. Separating those automatically remains an open problem.

What We Learned

Building an AI system that generates scholarly review articles isn't just a language generation problem — it's a citation integrity problem. And you can't fix what you can't diagnose.

Diagnosis drives intervention. The CCV's failure taxonomy — not just the aggregate score — was the real driver of improvement. Fabricated specifics and source mismatch responded well to targeted prompt instructions. Overgeneralization and editorial embellishment proved subtler and more resistant. Insufficient source turned out to be a retrieval problem, not a writing problem. Without the taxonomy, we'd have written generic "be more accurate" prompts and seen generic results.

Positional effects are real and fixable. The finding that citation quality degrades toward the end of sections — and that a simple instruction eliminates this pattern — suggests that LLM attention and discipline are more shapeable than we expected.

The model isn't lying; it's over-completing. The most common failure isn't fabricating false information. It's projecting true-but-unsourced knowledge onto nearby citations, filling in plausible details the source doesn't provide, and upgrading hedged language to definitive claims. The model wants to write a "complete" scholarly review, and completeness often means going beyond what any individual source actually says.

For anyone building AI systems that cite sources — whether in scholarly writing, legal research, or technical documentation — the lesson is that evaluation granularity drives improvement granularity. The CCV's claim-by-claim decomposition enabled targeted interventions that cut problematic claims by 55%, nearly eliminated the worst failure modes, and produced a manuscript where citation quality is consistent from the first paragraph to the last. You can't fix what you can't diagnose.