The Confidence Trap — David Chadderton

Why we trust machines that don’t know what they don’t know — and what happens when we stop noticing.

You know the feeling.

You are in a meeting. Someone answers a question with total conviction. Clean sentences. No hesitation. The kind of delivery that fills a room. And you know — you actually know, somewhere in the back of your mind — that what they have just said is not quite right. Maybe it is outdated. Maybe it is a half-truth. Maybe it is flatly wrong.

But you do not speak up.

Not because you are afraid. Not because you do not have the answer. But because the confidence with which the statement was delivered has already shifted the energy in the room. To contradict it would require you to match that confidence, and confidence against confidence feels like confrontation. So you let it go. The room moves on. The wrong answer becomes the working assumption.

Now multiply that dynamic by a billion interactions a day, and replace the confident colleague with a machine that never hesitates, never hedges, and never admits it does not know.

That is where we are.

The Jim Principle

There is a scene in the American sitcom Taxi that captures this dynamic perfectly. Jim Ignatowski, played by Christopher Lloyd, is the lovable, eccentric former Harvard student who has, through various life choices, become somewhat detached from conventional reality. In one memorable exchange, Jim delivers a series of completely wrong “facts” with absolute, unwavering confidence. And nobody corrects him. Not because they agree. Not because they are being polite. But because his delivery is so certain, so devoid of doubt, that challenging it feels socially expensive.

The scene works as comedy because we recognise the pattern. We have all been in rooms where the person with the most confidence, not the most knowledge, sets the direction.

Watch the scene here

Jim does not know what he does not know. He is not lying. He is not manipulating. He is simply generating answers with complete conviction, regardless of whether those answers have any relationship to reality.

If that sounds familiar, it should. It is a near-perfect description of how large language models work.

The data behind the comedy

Artificial Analysis recently published the AA-Omniscience benchmark, which measures AI models on two dimensions: accuracy (how often they get the answer right) and non-hallucination rate (how often they admit they do not know something, rather than inventing an answer).

The results are not comedy. They are an industry-wide indictment.

The best-performing model, Claude Fable 5, achieves a non-hallucination rate of 45%. That means 55% of the time it is uncertain about something, it fabricates an answer rather than admitting ignorance. And this is the best model available. Every other model tested performs worse.

Command A+, the most honest model in the benchmark, still fabricates 14% of the time. At the other end, DeepSeek V3 Flash hallucinates 96% of the time when uncertain.

Most of the models people actually use for content generation, research, and daily decision support sit somewhere in the 30-60% range. A third to half of the time these tools are generating information they are not confident about, they are presenting it as established fact.

No hedging. No “I’m not sure about this.” No signal to the reader that the next sentence might be invented.

Just clean, confident, authoritative-sounding text.

Why we fall for it

The uncomfortable question is not why AI models hallucinate. That is an engineering problem, and engineers are working on it.

The uncomfortable question is why we keep believing them.

The answer lies in how human cognition processes confidence. Behavioural scientists call it the authority bias: our tendency to assign greater credibility to information delivered with certainty, regardless of whether that certainty is warranted. It is the same bias that makes us trust a doctor who speaks decisively more than one who says “I’m not entirely sure, but I think…”

Dual Process Theory, one of the seven psychological pillars of the STAR Framework, explains the mechanism. Our System 1 processing — the fast, intuitive, automatic mode of thinking — evaluates fluency and confidence as proxies for accuracy. A well-structured sentence feels more true than a hesitant one, even when the content is identical. AI output is almost always fluently structured. It almost always sounds confident. So System 1 accepts it, and System 2 — the slow, deliberate, analytical mode — never gets engaged to check.

This is not a rational evaluation. It is a cognitive shortcut. And AI exploits it at industrial scale.

Automation bias makes it worse. Studies have shown that people will override their own correct judgements to accept an incorrect computer recommendation, simply because the machine presented its answer with certainty. We do not just fail to question AI. We actively suppress our own knowledge to defer to it.

In the meeting analogy, automation bias is the equivalent of knowing the answer is wrong but deciding the machine probably has access to information you don’t.

The compounding problem

The danger of AI hallucination is not just that individual outputs are sometimes wrong. It is that the errors compound.

When an AI confidently states a fabricated fact — a property that does not exist, a price that was never offered, a study that was never conducted — that fact enters circulation. It gets cited. It gets repeated. It gets indexed. It enters the training data for future models. Each repetition adds another layer of apparent confirmation, making the fabricated fact increasingly difficult to dislodge.

Cognitive Bias Theory describes this as confirmation bias operating at industrial scale. Once a piece of information is accepted as true, subsequent encounters with it are processed as confirmation rather than new claims requiring verification. The hallucinated fact becomes increasingly “real” with each repetition, not because evidence accumulates, but because the repetition itself creates the illusion of evidence.

This is how you end up with AI systems confidently citing sources that do not exist, recommending businesses that closed years ago, and quoting statistics from studies that were never conducted. Not because the models are malicious. Because the information ecosystem they operate in has no reliable mechanism for distinguishing between “stated with confidence” and “actually true.”

The Jim Knows Stuff problem at scale

In the Taxi scene, Jim’s confident nonsense affects one room. A handful of people. The social cost is low — someone might be mildly misinformed about a trivial fact, and the comedy writes itself.

Now put Jim in every search result. Every chatbot. Every AI assistant. Every content generation pipeline. Every customer-facing interaction. Every email draft. Every research summary.

And make Jim never stop talking.

The scale of the problem is not that AI sometimes gets things wrong. Every information source does. The scale of the problem is that AI gets things wrong with the same fluency and confidence with which it gets things right, and there is no signal — no hesitation, no hedge, no qualifying language — to help the reader distinguish between the two.

A human expert who is uncertain will say “I think” or “if I remember correctly” or “you should verify this.” Those hedges are not weaknesses. They are critical information. They tell the listener to engage their own judgement. They activate System 2.

AI does not hedge. It does not qualify. It does not signal uncertainty. It presents everything — the known and the invented — with the same clean, authoritative delivery.

And we, being human, process that delivery as a signal of reliability.

What this means for brands

If you are a brand operating in a world where AI assistants are forming consideration sets before your customers ever visit your website, the implications are immediate.

The “Invisible First Click” framework describes a reality in which the moment of brand selection is increasingly happening inside AI-generated answers. A student asks “What is the best student accommodation in Leeds?” and the AI responds with a confident recommendation. That recommendation might be based on real, current, accurate information. Or it might be hallucinated — a property that does not exist, a price that was never offered, a location that is wrong.

The student has no way to tell the difference. The delivery is the same. The confidence is the same. The fluency is the same.

If the AI hallucinates a positive recommendation for a competitor that does not exist, the student’s consideration set has been poisoned by fiction. If the AI hallucinates negative information about your brand, you have been damaged by a machine that was never trained on your actual data.

And if the AI simply does not know about your brand at all — if you are absent from its training data, its citation sources, its structured knowledge — then you are not being misrepresented. You are being erased. Not by malice. By absence.

The trust equation

Appraisal Theory, another of the STAR Framework’s seven pillars, explains what happens when we discover we have been misled.

When a human expert gives us wrong information, our emotional response is shaped by how we appraise the situation. If we believe they acted in good faith, we feel disappointed but not betrayed. If we believe they were careless, we feel something closer to anger.

With AI, the appraisal lands differently. We do not attribute intention to the machine. We cannot be angry at a algorithm. But we do attribute reliability to the system that deployed it. When an AI hallucinates, the emotional response transfers to the brand, platform, or organisation that put the AI in front of us.

This creates a trust asymmetry: the trust we extend to AI output is high (because of fluency and authority bias), but the trust we withdraw when that output proves false is directed at the human institution behind the AI.

Every confident, fluent, authoritative-sounding AI output that turns out to be wrong does not just fail to inform. It actively erodes trust in the institution that delivered it.

What to do about it

The practical response is not to stop using AI. The tools are genuinely useful when they work. The practical response is to treat AI output the way you would treat a confident colleague who might be Jim Ignatowski: useful as a starting point, dangerous as a final answer.

Verify before you publish. If AI generates content that will be seen by customers, students, or stakeholders, check the facts. Not the tone, not the structure, not the plausibility — the facts.

Choose your models deliberately. The AA-Omniscience benchmark shows that hallucination rates vary dramatically across models. Model selection is a risk management decision, not just a cost decision. A cheaper model that hallucinates 60% of the time will cost you more in credibility than a more expensive one that hallucinates 15%.

Build uncertainty into your AI interfaces. If you are deploying AI in customer-facing contexts, design for uncertainty. Give the AI permission to say “I don’t know.” Show confidence indicators. Make it clear that the output is generated, not verified.

Invest in being accurately represented. If AI engines are forming consideration sets for your products and services, invest in making sure those engines have access to accurate, structured, up-to-date information about your brand. Schema markup, llms.txt files, structured data, and consistent citation sources are not SEO tactics. They are trust infrastructure.

The uncomfortable truth

Jim Ignatowski is funny on television because the stakes are low and the audience knows he is a character. The joke works because we can see the gap between his confidence and his competence.

In the real world, that gap is not funny. It is structural. And we have filled it with machines that deliver Jim-level confidence at global scale, with no laugh track, no context, and no visible signal that the next sentence might be invented.

The question is not whether AI will get better. It will. The question is what happens to trust in the meantime — and whether the confidence trap will cost us more than we can afford to lose.

David Chadderton is the creator of the STAR Framework and the author of three books: The STAR Framework: Rewriting the Rules of Consumer Engagement (NYC Big Book Award 2025), The STAR Operating System: Decode Mindset, Understand Motivation, Transform Human Behaviour, and Dear Algorithm, It’s Not Me, It’s You. He spent his twenties and thirties as a military aviator and instructor, studying how people make decisions when the stakes are highest. He now applies those principles as a Chief Marketing Officer, bringing behavioural science to performance marketing at scale. He writes about human behaviour, AI, and the psychology of decision-making on The Unoptimised Human.