Five Minutes to Spot the Unsourced Numbers in an AI Report

You can tell in five minutes whether a report deserves an hour of your time. Most of the warning signs are visible without opening a single source. The ones that catch invented material are not, and those are exactly the checks a busy reviewer skips when the formatting looks professional.

Why not just ask the AI whether it's true?

Because it will vouch for itself. In 2023, a New York lawyer used ChatGPT to find case law for a filing in Mata v. Avianca. He then asked it, "Is Varghese a real case" and "Are the other cases you provided fake." It told him the cases were real and could be found on Westlaw and LexisNexis. Six of the judicial opinions did not exist.

On June 22, 2023, Judge P. Kevin Castel of the US District Court for the Southern District of New York fined the two lawyers and their firm $5,000, jointly. He also ordered them to write to each judge who had been named as the author of a fake opinion, enclosing the sanctions ruling and the fake opinion.

The ruling isn't hostile to the technology. It opens by saying there is nothing inherently improper about using a reliable AI tool for assistance, but that existing rules give attorneys a gatekeeping role to make sure their filings are accurate. In a company, the gatekeeper is whoever reads the report. Asking the tool that wrote it to check it is like letting a student grade their own exam.

Paid consultants are not immune

In 2025, Deloitte's Australian practice delivered a review to the federal Department of Employment and Workplace Relations. Its subject was the IT system that penalizes welfare recipients who miss obligations such as job search appointments. The contract was worth AU$440,000. After the report was published in July, Christopher Rudge of the University of Sydney found fabrications in it: a quote attributed to a Federal Court judgment that the judgment doesn't contain, and references to reports that don't exist.

The department posted a corrected version on September 26, with more than a dozen references and footnotes removed. It also added a line to its methodology disclosing a generative AI tool chain based on Azure OpenAI GPT-4o. Deloitte agreed to refund the final installment of the contract.

The detail that bothers me most is who caught it. An outside reader, after publication. Both the firm that wrote it and the department that received it had a chance first.

What to check in the first five minutes

Start with what you can see without opening anything. Finish by opening a few sources. My order goes roughly like this.

Year and unit. A percentage with no year, an amount with no currency or no word on whether tax is included. Stop there for a second. A figure that changes over time and has no "as of" date will be useless in six months.

Who is in the denominator. "Foreign staff are 20%" of what? All employees or permanent ones? The Japanese subsidiary or the whole group? A number without a denominator can't be compared to anything.

How far from the source it is. A figure quoted in an article, an AI summary, a line from the company's own website: send each one back to the original. A company's description of itself is a claim at best.

Whether the precision fits. In a survey of 200 people, each respondent is half a percentage point, so every share is a multiple of 0.5. A report that says 37.42% either has a different sample or a different calculation. Very round numbers deserve a question too. "About a third" with no source is often just something someone said in a meeting.

Whether the numbers agree with each other. A market worth ¥50 billion on page 3 and ¥80 billion on page 9, with no change in definition explained, means at least one is wrong.

Last come the slowest checks and the ones that catch fabrication. Pick three numbers, open their links and look for the sentence on the page. If the report quotes someone, copy a phrase and search for it.

Invented sources and quotes only show up when you open the original

Checks for a report written by AI or by your team, sorted on two axes. Cases: Mata v. Avianca (2023); Deloitte Australia's review for the Department of Employment and Workplace Relations (2025)

Across: left = visible without opening sources, right = needs the original. Down: top = catches fabrication, bottom = catches definition errors

Contradictions, impossible precision

The same figure differs between pages. 37.42% from 200 respondents can't come from any whole number of people

Open the links, search the quotes

The link is dead, or the page doesn't say it. The six fake opinions in New York and the court quote in Deloitte's report were exposed here

Year, unit, denominator

Numbers without a year, shares without saying who was counted. The cheapest to find, and probably the most common

Primary source or not

Quoted figures, AI summaries and self-reported company numbers go back to the original to match definitions

The cleaner the formatting, the easier it is to skip the last step. Pick three numbers and open the originals.

A working link isn't the end of it

A link that opens only gets you through the door. In 2023, William H. Walters and Esther Isabelle Wilder reported in Scientific Reports on 636 references that ChatGPT produced for 84 short papers on 42 topics. Of the references from GPT-3.5, 55% were fabricated. For GPT-4 it was 18%. Among the references that did exist, 43% from GPT-3.5 and 24% from GPT-4 had substantive errors.

Those are models from more than two years ago, and today's rates may be different. The second pair of numbers is the one I'd remember. The reference is real and the content doesn't match, and you won't see that by opening the link and reading the title. So for the three numbers you picked, find the actual sentence in the original and check that the year and unit match too.

My bet: wrong definitions outnumber fabrication

I haven't counted this, so take it as my view. Invented numbers are the minority. Most reports, I think, go wrong somewhere else: the number is real and the definition is off.

The same company can report four different female manager ratios depending on the denominator and what counts as a manager. I took that apart in One Company, Four Female Manager Ratios. Monthly overtime in Japan can be 13 hours or 21, depending on which survey you use and who it counts, as in Overtime in Japan: 13 hours a month, or 21? AI is very good at finding a real number and dropping it into a sentence where it doesn't belong.

These errors are harder to catch than fabrication because every number is correct on its own. Only when you write out the denominators do you see that two of them can't sit side by side.

Send it back. Don't fix it

If you correct it yourself, you will get the same kind of report next time.

Three marks are enough. For a number with no source, "not disclosed" or a request for the source. For one whose source doesn't match, the sentence from the original. For a judgment written up as a fact, "to verify." I use these terms the way I set them out in my ground rules.

Five minutes won't verify a report. What it does is tell you whether to spend an hour reading it, or to hand it back and ask for every number's source first. That second option feels rude. I've come to think it is the gatekeeper's whole job.

---