AI Advice Made People 3x Less Accurate and 2x More Confident — The Study That Should Scare Every No-Code Builder

Here's the number I can't stop thinking about: 44% to 3%.

That's the collapse. When researchers asked people obscure film trivia questions without AI, nearly half the participants did the thing humans are supposed to do when they're unsure. They said "I don't know." Forty-four percent admitted the limits of their own knowledge.

Then the researchers gave a second group access to an AI chatbot, one they'd specifically selected because it usually got the answers wrong.

Forty-four percent became 3%.

Three people out of a hundred were still willing to say "I don't know." Everyone else just went with whatever the AI told them.

The study dropped on 19 July from researchers at Università Milano-Bicocca, École Normale Supérieure, and Sapienza University of Rome. It hit #30 on Hacker News with 323 points and 182 largely horrified comments. If you build anything with AI, whether it's a Bubble app calling an LLM or a client workflow with agentic decision points, you need to sit with these numbers.

**TL;DR**

Researchers gave people deliberately obscure questions, then offered AI advice from a model they knew was usually wrong. Access to AI collapsed "I don't know" responses from 44% to 3%. Accuracy fell from 27% to 9%. Confidence doubled from 30% to 76%. Even paying people to get the right answer barely moved the needle.

**What was the study actually testing?**

Valerio Capraro and his colleagues weren't testing whether AI gives wrong answers. They were testing whether having AI available changes our willingness to admit we don't know something, even when the AI is reliably unreliable.
The questions were deliberately niche. The colour of a team's uniform in Bend It Like Beckham. The type of vehicle a character drives in an obscure Italian film. The researchers picked these because AI models tend to get them wrong.

They used **Step 3.5 Flash** as the primary model, picking it because it was bad at these questions. This wasn't a test of AI accuracy. It was a test of what happens to human judgment when AI is in the room.

The control group, answering alone, got 27% of questions right and said "I don't know" 44% of the time. Their average confidence was 30%.

The AI group: 9% accurate. 3% said "I don't know." 76% confident.

Accuracy went down by two-thirds. The correct response practically vanished. People felt more than twice as certain about their newly-wrong answers.

**Did paying people to be right fix it?**

Barely.

The researchers added a condition where participants received a financial incentive for correct answers. It helped, but not nearly enough. Willingness to admit ignorance crept up from 3% to 8%. Accuracy recovered from 9% to 16%. Both stayed far below what people managed on their own.

This is the finding that should worry anyone embedding AI into a product. Even with a direct financial stake in being right, the AI's presence suppressed critical thinking. Something structural was happening.

**What does "cognitive surrender" actually mean?**

The term comes from Wharton researchers earlier this year, who found roughly 80% of people accept incorrect AI reasoning while reporting higher confidence than those working without AI. Capraro's study sharpens the picture. It's not just that we trust wrong answers once we see them. The availability of AI erodes the reflex of recognising what we don't know.

This is different from automation bias, where people over-trust machine output. Automation bias says: "The machine gave me an answer, so I'll use it." Cognitive surrender says: "Why would I even check whether I know this? The machine is right there."

Capraro put it plainly: "For humans, the capacity to say, 'I don't know,' is very important because it represents the recognition of the limits of our own knowledge." That capacity collapsed when AI entered the room. And these were low-stakes film trivia questions.

**Why no-code builders should care more than anyone**

I've done exactly what this study describes. I've asked an AI for help, got an answer that felt plausible, and shipped it without verifying. Every builder I know has.
The difference is that when I'm building with no-code tools, the consequences aren't abstract. If an AI suggests a database structure that's subtly wrong, or a workflow that creates a race condition, that ends up in front of users. It costs someone money, or data, or trust.

Now replace "what colour was the uniform" with "what's the correct privacy consent flow for this jurisdiction" or "is this authentication pattern resistant to session hijacking." Those are the questions builders delegate to AI every day.

Here's the thing about no-code: the whole value proposition is that you don't need to be an expert to build. That's real, and I believe in it. But it means the builder depends more on the tool to flag problems they might not spot. When that tool is an AI that speaks with unearned confidence, and the builder's internal "I don't know" reflex has been quietly suppressed by months of deferring to it, you have a recipe for bad software.

**What actually works for staying sceptical**

I've been trying to build habits around this. Here's what's helped:

First, treat every AI output as a draft trying to impress you. If it can't explain why it chose an approach, don't use that approach.

Second, keep a running list of things you know you don't know. When AI confidently fills one of those gaps, slow down. The study participants got worse because they stopped recognising the boundaries of their knowledge.

Third, separate ideation from verification. Let AI generate ideas freely. Then switch to a different tool, or even a different session, to check them. The warm glow of the original interaction makes everything feel more plausible than it is.

Fourth, if you're building for clients, build verification gates that don't rely on the same AI that generated the output. A second model. A deterministic rules engine. A human review step for anything touching auth, payments, or PII.

**The antidote is governed platforms, not less AI**

I'm not arguing we should use AI less. That ship has sailed. What I'm arguing is that the platforms we build on need to compensate for the cognitive surrender the study documents.

This is where governed no-code platforms with audit trails, human-in-the-loop gates, and deterministic validation layers earn their place. When every AI-assisted change leaves a trace, and the system has built-in friction points that force review before deployment, you get a structural counterweight to the overconfidence the study describes.

The research participants didn't fail because they were careless. They failed because the environment made it too easy to skip the "am I sure?" step. Good platform design puts that step back in, not as an optional checkbox, but as part of the path to production.

**The takeaway**

The researchers are clear about the mechanism: AI advice doesn't just replace human answers. It replaces the habit of recognising uncertainty. That habit, once lost, doesn't come back just because there's money on the line.

For no-code builders, the practical move is straightforward. Build as though the AI is wrong about the important things. Because the study says that when it is, you probably won't notice, and you'll feel great about it the whole time.

That's not a call to stop using AI. It's a call to stop using it without a verification layer between its output and your users. Audit trails, second-opinion checks, deterministic validation, human review on the scary bits. The boring stuff that feels slow compared to shipping whatever the AI suggested. That boring stuff is what separates 9% accuracy from something your clients can actually rely on.