VerusCite V1 Benchmark: Citation Verification Accuracy
Andrew Wheeler 2026-07-31
8 Reproducibility
The Problem
Fabricated citations in academic papers have grown sharply since the widespread adoption of generative AI writing tools. A Lancet audit of 2.5 million biomedical papers found hallucinated reference rates rising 12-fold between 2023 and early 2026 (Topaz et al. 2026). Separate estimates project roughly 147,000 hallucinated citations across arXiv, bioRxiv, SSRN, and PMC in 2025 alone (Zhao et al. 2026).
Existing peer review and editorial safeguards largely fail to catch these errors before publication. The problem spans disciplines: biomedical research, AI/ML conferences, legal filings, and government reports.
While AI use is driving a surge in publications (see A. Wheeler 2026 for a practical overview), the same large language models that create this problem can also be used to verify citations at scale. VerusCite (veruscite-data.com) is a tool to automatically extract and verify citations from uploaded documents. This report documents the accuracy of the V1 verification pipeline against a hand-labeled ground truth corpus.
This tool is meant to be a quick and cheap approach to scan a bibliography for a paper, and determine if a citation is correct, has minor errors, or appears to have more serious issues, such as an AI generated hallucination. This paper serves as reference to the overall architecture and design of the project, as well as a source of false positive and negative rates on a benchmarked corpus of real papers.
Approach
Design Philosophy
Even in documents with hallucinated citations, the majority of references are legitimate. A tool with a high false positive rate – flagging real papers as hallucinations – would waste a reviewer’s time chasing false alarms rather than finding actual problems. VerusCite prioritizes precision over recall: it is better to miss some hallucinations than to incorrectly flag verified citations.
The target false positive rate for hallucination flags is below 1%. Recall for detecting actual hallucinations is around 70-80%. This trade-off means a human reviewer spends most of their time on citations that genuinely need attention rather than dismissing false reports.
The application is designed for human-in-the-loop review. Each citation’s verification status can be manually overridden, and the interface surfaces the reasoning behind each determination so reviewers can make informed decisions quickly.
Similar to work in Pangram, I believe that it is necessary to have the tool have as few of false positives as possible (Emi 2024). While this report will show that those false positives are not zero (any realistic tool should report a certain error rate), I believe
Pipeline Architecture
VerusCite processes documents in two stages:
Citation Extraction – An LLM reads the document text and identifies individual references, parsing out structured fields (title, authors, year, journal, DOI).
Citation Verification – Each extracted citation is checked against external sources to determine whether it exists as described.
For citation extraction, the tool uses text searching across the document (if a PDF or word document is uploaded) to attempt to determine the location of the reference section. This tool currently works with reference sections combined in an article, it does not work with footnote style citations (as is common in law articles).
If uploading a bibliography file, such as a Bibtext .bib file, a .ris file, or an Endnote .enw file, the tool uses text tools to automatically extract out the reference information.
Once the reference section is identified, the reference section is then chunked into smaller portions, and uses a served LLM model (see later benchmark results) to extract out the citations into a structured sections. This means the LLM is only sent the references for the document, it is not sent other parts of the manuscript.
Each extracted citation is then checked independently. First the citation is checked against the Crossref database. If it is not found in Crossref, then an agentic tool uses web-search and fetch to verify whether the citation exists in external sources.
This independence means that in a document with some errors, other documents are not falsely assumed to be hallucinated. It also allows processing the documents in parallel, to return the results back faster.
Models Considered
Because the task involves web-search, when checking citations that are not verified via Crossref, this limits the potential pool of LLM providers. Here I consider two major companies:
Gemini family models
OpenAI family models
In particular, I focus on the cheaper models (flash-lite for Gemini and Nano/Luna for OpenAI) with minimal reasoning. I additionally consider these same models, but using the web search tools provided by Perplexity.
I do not consider Anthropic, as even their cheapest model (Haiku) is considerably more expensive than the models I evaluated here. I only partially evaluated AWS Nova-2 lite with their web search (latency was too long). Grok 4.3 did well in the benchmark tests but was considerably more expensive.
Extraction does not need web search, and so while I evaluated many other models on Bedrock (such as the open source GPT and Gemini models), they were not as accurate as the frontier served models. Extraction takes around 2 to 3 cents per document, so at this time I am not concerned about cost, although in the future likely even smaller open source models (especially if fine tuned) will be sufficient for the extraction task.
I would consider additional open source models in the future (such as DeepSeek flash) for citation checking, although it will be necessary to incorporate custom web search and fetch tools in conjunction with the served models. If the models are not provided out of the box by Perplexity.
The checker assigns each citation one of four statuses:
Verified
Citation confirmed to exist via CrossRef or web search
Minor Error
Citation likely exists but has metadata discrepancies (wrong year, volume, pages)
Hallucination
No evidence the cited work exists as described
Not Found
Unable to confirm or deny – includes bare URL citations and paywalled/very recent works with insufficient search results
Many citations have minor errors (perhaps on the magnitude of 5% in my personal assessment). These include many innocuous things that are likely human introduced errors, like a year off or page numbers wrong.
The major different between a hallucination and a minor error is that hallucinations tend to have whole cloth differences that are difficult to explain. Such as journals entirely replaced. For example, Canessa et al. (2026) has the citation:
Elrod, Linda D. (2006). “A Move in the Right Direction? Best Interests of the Child Emerging as the Standard for Relocation Cases”. Journal of the American Academy of Matrimonial Lawyers 15, pp. 1-54.
This is a real paper, but should be cited as:
Elrod, Linda D. (2006). “A Move in the Right Direction? Best Interests of the Child Emerging as the Standard for Relocation Cases”. Journal of Child Custody 3, 29–61. https://doi.org/10.1300/J190v03n03_03
While the Journal of the American Academy of Matrimonial Lawyers does exist, there is no combination of volumes and page numbers that could reasonably be confused with the article in the Journal of Child Custody.
While no automated tool can never find 100% proof that a citation was hallucinated via a generative AI tool, it is difficult to construct a scenario where a bibliography had such errors manually generated, especially the volume and page numbers that do not exist in any comparable document for the Elrod citation.
Hallucinations can also include when multiple authors are incorrect, but author name errors are much more common, so they are less likely to be flagged. For an example of a minor error Mekonen (2026) has the reference:
Croft TMA, Allen CK, Arnold F, Assaf S, Balian S. Guide to DHS Statistics: DHS-7 (version 2). Rockville, MD: ICF. 2020.
The actual author list includes an Aileen M.J. Marshall, whose initials were concatenated into the first author (see page 2 at https://dhsprogram.com/pubs/pdf/DHSG1/Guide_to_DHS_Statistics_DHS-7_v2.pdf).
Minor errors are incredibly prevalent. Any tool intended to do the same task needs to effectively distinguish between minor errors to be able to identify actual hallucinated citations.
Benchmark
The V1 validation corpus contains 2288 hand-labeled citations across 36 documents. Sources are intentionally heterogeneous:
Papers with known hallucinations identified by others on social media (Chris Carothers on X, David Buil-Gil on BlueSky)
GPTZero’s NeurIPS 2025 audit (GPTZero 2026)
Papers flagged by Reviewer3 (Reviewer3, n.d.)
A ChatGPT deep-research output where all 12 citations are fabricated (Jacques, Wheeler, and Gerstenfeld 2026)
Clean papers (my own dissertation excerpts, open-access criminology, PLoS ONE articles)
MDPI and preprint samples
The corpus spans categories in social sciences, physics, mathematics, as well as many different formats, including many different pre-print and journal format examples.
Of the 2288 citations, 1889 (82.6%) are verified, 204 (8.9%) have minor errors, 141 (6.2%) are hallucinations, and 54 (2.4%) are not found. The corpus skews heavily toward verified citations (as real documents do), which makes false positive rate the critical metric.
There are two different metrics a user should be interested in. First is whether the tool can actually extract out the references accurately. This is non-trivial in and of itself – a skills based approach submitting a large document with an extensive bibliography is likely the generate more errors in this steps.
The second metric is in terms of false positives (e.g. a good citation is flagged as a hallucination), and recall rates (of all the hallucinations, how many does the tool capture). Below are those metrics for this corpus on the current VerusCite tool across different LLM providers.
Extraction Results
Citation extraction uses an LLM to parse the document text and identify individual references. The table below shows extraction accuracy against the ground truth. There are two types of errors that can occur, you can miss a reference, or the tool can add in a reference.
gemini-3.1-flash-lite
2288
2285
3
3
99.9
4.9
7.4
1.15
gpt-5.4-nano
2288
2285
3
4
99.9
12.2
3.0
0.88
gpt-5.6-luna
2288
2284
4
6
99.8
8.2
4.4
0.84
Extraction is very accurate across models. Gemini 3.1 Flash Lite (using pypdfium2 to read in the text, we additionally considered liteparse with markdown) correctly extracts over 99.8% of citations at under $0.03 per paper for the full corpus. The few “extra” rows are typically page footers, footnotes, or duplicate listings that the LLM picks up in addition to the actual bibliography.
I also tested liteparse with the markdown export format to extract out the text from PDFs, which tended to produce slightly more artifacts – it does not distinguish line breaks for individual citations as cleanly as pypdfium2, producing a few more missed or merged entries. Results were largely similar overall (and I may migrate to using liteparse with markdown in the future).
Both the extraction part of the pipeline and the citation check part of the pipeline have fallback models, as it is common for these LLM providers to have downtime with models. The current production version of VerusCite uses 3.1 flash lite (served by Google) as the primary extraction model (due to both accuracy and cost). The fallback model is currently gpt-5.6-luna. While OpenAI (both the 5.4 nano and 5.6 luna models) are somewhat cheaper, due to the lower accuracy and time, it is not used at this point. (The cost only saves around 1 cent per paper.)
Generally the prompts are constructed in a way that advanced reasoning is not necessary. So going up to larger reasoning models (and expanding the reasoning budget) does not result in higher accuracy. For OpenAI, you can see that the recent luna model did not result in any more accurate results than nano.
Citation Checking Results
The citation checking results break down the benchmark against both the model, and the provider. When Perplexity is the provider, it means the tool called Perplexity servers (using their internal web search), whereas if the provider is gemini it uses Googles web search tool, or OpenAI uses its internal web search tool. Otherwise all prompts are the same across each of the models.
The citation checking results input in correct data – extraction and checking are done in independent benchmarks. So each of the 2288 citations, it is independently run though the checking benchmark, swapping out the model used.
gemini-3.5-flash-lite
gemini
0.1% (2)
0.8% (15)
63.1% (89/141)
71.9% (286/398)
google/gemini-3.1-flash-lite
perplexity
0.4% (7)
0.7% (13)
69.5% (98/141)
82.7% (329/398)
gpt-5.4-nano
openai
0.2% (3)
2.2% (42)
60.3% (85/141)
88.4% (352/398)
gpt-5.6-luna
openai
0.2% (3)
2.5% (47)
74.5% (105/141)
88.7% (353/398)
openai/gpt-5.4-nano
perplexity
0.4% (7)
2.5% (47)
33.3% (47/141)
84.9% (338/398)
False Positives
For key metrics, there are a total of 1889 correct citations in the corpus. So false positives (FP) rates are of those 1889 correct citations, how many were falsely flagged by the tool.
The current FP rates for hallucinations are well under 1%, and vary between 2 to 7 total false positives across the different models in this corpus. Hallucination false positive are often attributable to idiosyncratic web search results. (The agentic tools often will return a response, even if the web search tool is currently inaccessible.) Using Perplexity web search generally results in the same accuracy as using Google or OpenAI web search.
For an example of a false positive hallucination in this particular run, my dissertation has the citation:
Pearl, J. (2000). Causality: models, reasoning and inference, Volume 29. Cambridge Univ Press.
And 5.4 nano lists as reasoning for classifying this as a hallucination:
Upstream verdict: book exists and matches author/title/year/publisher, but the cited “Volume 29” element is not supported by the primary publisher/library records surfaced in the tool results and is treated as fabricated metadata; therefore status mapped to hallucination.
This behavior is not consistent, either within 5.4. nano or across other models. (This most often returns a “minor error”, although my ground truth I have this as “verified”. Some sources list the published year as 2001, causing additional issues beyond just the “Volume 29” addition.) I have intentionally avoided K-shot examples in the prompt, so there is as little leakage as possible and the prompts should better generalize to out of sample data. But given the stochastic nature of LLMs, some errors will ultimately occur.
FP rates for minor errors are more prevalent, being close to 1% for the Gemini models, but over 2% for the OpenAI models. The current tool in production uses Gemini 3.5 flash-lite (Perplexity is currently having issues consistently calling 3.5 flash-lite).
The general approach I took was to evaluate when both OpenAI and Google models returned false positives. While these do happen in the corpus, in some cases they are inevitable, as CrossRef or other online data provides conflicting information. For one example citation:
[CS22] T. Cieśla and M. Sabok. Measurable Hall’s theorem for actions of abelian groups, J. Eur. Math. Soc., 24 (2022), 2751-2773 (cit. on pp. 3, 44)
Consistently produces a “minor error” category in the tool due to the citation year. Going to the website, the citation year is correctly 2022. But crossref lists the publication year as 2021.
Many of the false positives are minor errors like this (and likely some of the labels in the ground truth should be updated – feel free to contact me if you believe an articles classification is not correct in the ground truth). False positives for not found tend to be due to unreliable web search, and so are intermittent and not consistent due to the tool.
Recall is the proportion of the true errors that are captured by the current tool. These are more variable across the different tools, with OpenAI models having greater recall (which comes with more false positives). Hallucination recall for the gemini models is currently ranges between 60 and 70%, whereas OpenAI luna is at 75%.
Minor error recall is similarly lower for Gemini (with the current model in production 3.5 flash-lite), at 52.7%. OpenAI has higher recall (often higher than 70%) for minor errors. Not found recall is near perfect across all models.
The final category, not verified, collapses the categories of minor error, hallucination, and not found. So if many hallucinations were classified into minor error, the direct recall rates would be low, but not verified (which will typically trigger a human review) would still be high. These are consistently over 80% across all model runs, with the exception of Gemini 3.5 flash lite (which is mostly due to low recall on minor errors).
Given Gemini is currently much faster as well, this is the model chosen for production. The fallback model is currently OpenAI luna.
Cost Breakdown for Citation Checking
For the 2288 total citations, approximately 1310 citations were verified via Crossref after extraction (about 57%) for each of the models. These are largely automated, and so incur no additional LLM cost. When those fail however, an agent based LLM tool needs to use web search and fetch to identify whether the citation is correct. Thus costs incur for both token usage as well as web search. Web search costs $5 per 1000 searches on Perplexity, $7 per 1000 searches for OpenAI, and $14 per 1000 searches for Google.
The prompts are generally short enough that token caching does not occur at all for the Gemini models (needs over 4000 tokens). Some token caching does occur for OpenAI (although these costs do not include that), but it is relatively small (and the majority of token costs are output).
gemini-3.5-flash-lite
gemini
6.14
8.44
14.59
0.41
google/gemini-3.1-flash-lite
perplexity
12.15
7.72
19.87
0.55
gpt-5.4-nano
openai
7.17
16.28
23.45
0.65
gpt-5.6-luna
openai
6.84
11.49
18.33
0.51
openai/gpt-5.4-nano
perplexity
7.34
4.84
12.18
0.34
The newer 3.5 flash lite model has resulted in lower token costs (despite being slightly more expensive than 3.1 flash-lite). One can see though across this corpus, costs are typically around 50 cents per paper to process, with often more than half of the cost being devoted to web search.
VerusCite uses only zero data retention (ZDR) models from all providers. Uploaded documents are not used for model training by any third party.
Additionally, the pipeline minimizes what is sent to external LLMs. Citation extraction uses text search on the locally-extracted PDF text first – only the citation strings themselves are sent to an LLM for structured parsing. During verification, only the parsed citation metadata (title, authors, year) is sent to web search, not the full document text. The full PDF content never leaves the server.
Limitations and Next Steps
The V1 corpus is heterogeneous by design, but 36 documents is a limited sample. The next evaluation (V2) will use the same prompts and pipeline but a completely different validation set – different documents, different domains, different sources of known hallucinations. This ensures the system is not overtrained on the particular examples in V1.
Known limitations of the current evaluation:
Ground truth labels for some documents have ambiguous categories that likely could be reasonably changed. This is particularly true for the minor error category.
“Not found” is an inherently ambiguous category – some citations are bare URLs, others exist but are difficult to locate via web search (paywalled, very recent, or in non-English databases).
The corpus over-represents arXiv preprints relative to other domains (law, humanities, clinical medicine).
Reproducibility
All data for this report is public at github.com/apwheele/veruscite-data. The v1/ directory contains:
ground_truth.csv – hand-labeled citations with expected_status
extraction_run/ – raw extraction outputs per model
checking_run/ – raw checker outputs per model
manifest.json – run IDs used in this report
To regenerate this report:
This paper was prepared with AI assistance. Drafting and earlier iterations used Claude Opus 4.6 (Anthropic), reviewing prior works by Andrew Wheeler (see A. P. Wheeler (2026) for that workflow). Updates for the 2026-07-30 extraction and checker refresh (including DOI link-annotation recovery notes, gpt-5.6-luna extract/check runs, and Perplexity openai/gpt-5.4-nano), speed/cost metrics, and this disclosure were assisted by Grok 4.5 (xAI). Ground-truth labels, model-default decisions, and final review are human.
References
Canessa, Stella, Gordon B Dahl, Anna Hasselqvist, Costas Meghir, Susan Niknami, Mårten Palme, Helmut Rainer, Olof Rosenqvist, and Pengpeng Xiao. 2026. “Life After Divorce: Effects of Joint Custody on Parents and Children.” Working Paper 35482. Working Paper Series. National Bureau of Economic Research. https://doi.org/10.3386/w35482.
Emi, Bradley. 2024. “All about False Positives in AI Detectors.” https://www.pangram.com/blog/all-about-false-positives-in-ai-detectors.
GPTZero. 2026. “GPTZero Finds 100 New Hallucinations in NeurIPS 2025 Accepted Papers.” https://gptzero.me/news/neurips/.
Jacques, Scott, Andrew Wheeler, and Joshua Gerstenfeld. 2026. “Open Access, Generative Artificial Intelligence, and the Criminology Evidence Base.” Evidence Base 1 (2): 2658591. https://doi.org/10.1080/30679125.2026.2658591.
Mekonen, Enyew Getaneh. 2026. “Prevalence and Associated Factors of Intimate Partner Violence Against Reproductive-Age Women in Africa and Asia Regions: Insights from 2022–2024 DHS Datasets.” PLOS ONE 21 (7): 1–16. https://doi.org/10.1371/journal.pone.0353669.
Reviewer3. n.d. “Reviewer3: Live arXiv Reference Checking.” https://reviewer3.com/live/arxiv.
Topaz, Maxim, Nir Roguin, Pallavi Gupta, Zhihong Zhang, and Laura-Maria Peltonen. 2026. “Fabricated Citations: An Audit Across 2.5 Million Biomedical Papers.” The Lancet 407 (10541): 1779–81. https://doi.org/10.1016/S0140-6736(26)00603-3.
Wheeler, Andrew. 2026. Large Language Models for Mortals: A Practical Guide for Analysts with Python. Crime De-Coder.
Wheeler, Andrew P. 2026. “Using Claude Code to Help Me Write.” https://andrewpwheeler.com/2026/03/20/using-claude-code-to-help-me-write/.
Zhao, Zhenyue, Yihe Wang, Toby Stuart, Mathijs De Vaan, Paul Ginsparg, and Yian Yin. 2026. “LLM Hallucinations in the Wild: Large-Scale Evidence from Non-Existent Citations.” https://arxiv.org/abs/2605.07723.
dlvr.it
