How to Tell When AI Is Making Up Financial Data

Short version: an AI model isn’t looking anything up unless it’s explicitly wired to a real source, so a confident-sounding number is not evidence the number is true. Before you act on any AI-generated financial claim, ask one question: can I click through to the actual filing, report, or data point it came from? If the answer is no, the number hasn’t been checked, it’s been guessed convincingly.

Where the hallucination actually comes from

A language model is predicting the next plausible word, not retrieving a fact from a database. When you ask it for a company’s quarterly revenue or a stock’s current price, it produces the kind of answer that sounds right, built from patterns in what it was trained on, not a lookup against the actual filing. Most of the time that pattern-matching lands close to the truth, which is exactly what makes it dangerous: it’s right often enough that you stop checking, until the one time it isn’t.

This is true even when the model cites a source. Unless the system is specifically built to pull the number from a live, verifiable feed and show you the line it came from, a “source” next to a number can just be more plausible-sounding text, generated the same way the number was.

The failure mode that’s worse than “I don’t know”

A model that says “I’m not sure” costs you nothing. A model that states a wrong number with total confidence and a citation that looks legitimate costs you a bad decision made with false confidence. I ran into this myself: I asked an AI to help me understand a stock, got a clean, well-written answer with eleven sources listed at the bottom, and started checking them one by one. The first source didn’t contain the number the AI had quoted. The second didn’t mention the claim at all. A few were dead links or paywalled. Twenty minutes later, I still hadn’t verified the original answer, and that pattern repeated across different companies and different questions.

The tone of the answer told me nothing about whether it was true. That’s the actual problem: fluency and accuracy are produced by different mechanisms, and nothing in how an answer sounds tells you which one you’re getting.

A checklist for verifying an AI-generated financial claim

Before you act on a number an AI gave you:

Find the actual source. Not a citation that name-drops a filing or report, an actual link you can open.
Open it and search for the number. Not the topic, the exact figure. If it’s not on the page, the claim isn’t verified, no matter how specific it sounded.
Check the date. A real number from six months ago can be presented with the same confidence as one from this morning. Markets move fast enough that the gap matters.
Notice if the AI hedges. A model that says “I’m not certain” or “I couldn’t confirm this” on some answers and not others is giving you a real signal. Treat unhedged confidence as the default state, not as evidence of accuracy.
If you can’t verify it in under a minute, don’t act on it. The number might still be right. But acting on an unverified confident answer is the exact failure mode that produces the worst outcomes, because you find out it was wrong after you’ve already traded on it.

FAQ

Can you trust ChatGPT for stock analysis? Not on its own, for anything you’re going to act on financially. A general-purpose model isn’t connected to a live, licensed data feed by default, so a number it gives you about a stock is a plausible-sounding guess, not a retrieved fact, unless you’ve verified it against the actual source yourself.

Why do AI citations not check out? Because the model can generate a citation the same way it generates any other text: as something that fits the pattern of what a citation looks like. A citation being present tells you the model was trained on text that includes citations. It doesn’t tell you the citation is real, or that it supports the specific claim next to it.

What should a verifiable answer actually look like? One where every number links back to the actual document it came from, so you can click through and see it yourself instead of taking the model’s word for it. If an answer can’t show you where a number came from, treat it as unverified, regardless of how confident it sounds.