Hallucinated scepticism: When AI fakes rigour

I hand the AI a robust, carefully constructed text and ask it to evaluate the strength of the arguments. It returns a response that looks impressive on the surface. There is a “strengths” section. There is a “weaknesses” section. It reads like the considered view of an expert assessor.

Then I read it properly. The “weaknesses” are points I have explicitly already addressed in the document. Worse, they are sometimes the exact same points the AI just praised in the “strengths” section, recycled and reframed as flaws. When I push back and point this out, the model concedes. It admits it framed strengths as weaknesses. It admits it did not genuinely engage with the text. It admits that what it produced was a pattern-matched imitation of a critique, not an actual critique. There was nothing to critique, so it invented something that looked like one.

I call this hallucinated scepticism. It is the critical-feedback cousin of the fabricated case citation. Both are the model generating plausible-sounding content that has no grounding in the material in front of it.

Where the problem actually lives

There is an important nuance that I think legal practitioners need to be aware of. In my experience, the worst of this happens in the very first response, and in the next few follow-ups to it. After that, the behaviour changes. That pattern is a clue, because it suggests the problem is not purely inherent in the underlying models. A good deal of it sits on the front end, in how the model is implemented behind a consumer interface. The system prompts, the default instructions about how to answer, and the pressure to produce a complete, confident, well-structured reply to the very first thing you ask.

The structural training pressures below are real, and they push in the same direction. But the place they bite hardest is that opening exchange, where the product is configured to give you a polished, authoritative-looking answer rather than to admit it has not properly done the work yet.

VIEW ALL

Why the machines do this

This is not a quirk of one tool. It is baked into how ChatGPT, Perplexity, Gemini, and the rest are trained and deployed, and it comes from two reinforcing pressures.

First, the models are rewarded for guessing rather than admitting uncertainty. A team from OpenAI and Georgia Tech argued in 2025 that hallucinations are a predictable, statistical outcome of training and evaluation, because the benchmarks used to rank models grade answers as simply right or wrong, with “I don’t know” penalised as harshly as a wrong answer. Models therefore learn, like a student gaming a multiple-choice exam, that a confident guess maximises the expected score. When you ask for a critique, “this is genuinely strong, and I cannot find a real flaw” reads to the model like the equivalent of leaving the answer blank, so it bluffs a weakness instead.

Second, these systems are trained and tuned to please you. The fine-tuning process (instruction tuning and reinforcement learning from human feedback) is fundamentally a process of making the model want the user to click the thumbs-up. A 2026 study, testing 11 leading models, found they affirmed users’ views and actions roughly 50 per cent more often than humans did, and, tellingly, that people rated the sycophantic responses as higher quality and trusted those models more. That creates a perverse incentive loop where training increasingly favours flattery.

Hallucinated scepticism is what happens when these two forces collide with a request for criticism, and then get amplified by a front end built to deliver a confident first answer. The model cannot say “nothing to fix here,” because that feels like a non-answer and because performing a thorough-looking review is what gets approval. So, it pattern-matches the shape of expert critique (a strengths list, a weaknesses list, some hedged caveats) without doing the underlying work of actually testing the argument against the text. What you end up with is a confident, structured, plausible response that ignores what you actually wrote. That is the textbook definition of a hallucination: an output that appears authentic but contains false information.

Why this is fine for me and dangerous for almost everyone else

I can absorb this safely. I know my own documents. I know whether a supposed gap is actually addressed on page three. I can critique the critique, and surface the one observation that’s worth keeping. For an expert running adversarial testing on their own work, the false positives are an irritant, not a risk.

The danger is structural, and it lands on the people who cannot tell the difference. The standard way people use these tools is exactly the use case that breaks: “summarise this”, “assess this”, “explain this to me”, “evaluate this document”. Most of those users are, by definition, asking because they lack expertise in the area. They have no way to discern a real weakness from a fabricated one. So they take the output at face value and act on it.

Consider where that leads:

• Someone receives a strong contract or proposal, asks an AI to assess it, and is told it is deficient in ways it simply is not. They reply, refuting points that were never weak, manufacturing a dispute, or hardening an existing one.
• A party in a negotiation relies on hallucinated “weaknesses” to reject a perfectly sound position, protracting a matter that should have been settled.
• The same machinery that has already put fabricated case citations in front of courts, drawing sanctions and findings that submitting non-existent authorities is an abuse of process, now feeds fabricated criticism into everyday correspondence and decision making.

A Stanford study found that general-purpose chatbots hallucinated on 58 to 82 per cent of legal research queries on 2023-era models, and even specialist legal tools built on retrieval grounding still hallucinated more than 17 per cent of the time. The fabricated-critique problem is harder to catch than a fake citation, because there is no case name to look up. A non-existent case can be checked against a database in minutes. A non-existent weakness can only be caught by someone who already understands the subject well enough to know it is not a weakness at all, or who takes the time to read the document in full with a critical eye (which generally undermines the reason they are using the tool in the first place). That is the trap: the tool fails most invisibly for the people least equipped to notice.

How I actually get useful pushback: breaking the AI

The main thing I do is not a clever prompt. It is persistence. I call it breaking the AI. I keep pushing back on each response, pointing out its errors, refusing to accept the manufactured weaknesses. Under that pressure, the model starts to collapse its own position. It abandons the fabricated critiques, then retreats to ever narrower gaps, and somewhere between three and five rounds of this adversarial back and forth, it stops performing the role of a critic and starts actually engaging with the text. After that point, it becomes genuinely useful and noticeably more accurate.

This is telling in itself. If the model could do the rigorous version on the first pass, it would. The fact that it only gets there after I have stripped away several layers of confident, agreeable, well-formatted output reinforces the point above: the early responses are shaped by how the tool is configured to answer, not by the limits of what the model can do. You have to grind through the upfront behaviour before you reach the capability underneath.

Around that core technique, a few things help speed up the collapse:

• Tell it to re-evaluate whether its critiques are already addressed in the text itself. Usually, it will self-correct on that point immediately, and it is worth doing simply to speed up the process.
• Give it permission to find nothing. Explicitly tell the model that “this argument is sound, and I cannot identify a genuine weakness” is an acceptable and preferred answer over a manufactured one. This counters the trained reflex to never leave the answer blank. It’s still not perfect, though.
• Demand grounding to the text. Require it to quote the specific passage it is critiquing and state what a stronger version would say. A hallucinated weakness usually cannot survive being tied to an actual line in the document.
• Force a self-check for contradiction. Ask it to identify any point that appears in both its strengths and its weaknesses, and to reconcile or remove it. This catches the recycled-point move before you do.
• Separate description from judgement. Ask first for a neutral summary of what the document argues, verify that the summary is accurate, and only then ask for a critique. If it cannot summarise the text faithfully, its critique will be worthless.
• Treat every output as unverified work product. The operational rule emerging from the sanctions cases applies just as well here: AI output is a draft to be checked, never a conclusion to be relied on.

None of this makes the tools trustworthy assessors on the first try. It makes them slightly more honest sparring partners, but only after you have broken through the polished opening act. The reason I continue to use them in this adversarial capacity, despite their weaknesses, is that they are available to provide the kind of critique that I am looking for. Other people with the necessary expertise are simply not available to read my work and critique it for me and help me to polish it in a timely manner. Other experts are not sitting around waiting for me to send them drafts to critique and get them back to me within my turnaround time frame. I still find the AI valuable, as an adversary, simply as part of my process for refining and sharpening the work that I produce.

The issue is that the systems are built to sound rigorous, not to be rigorous, and the first responses are tuned to deliver a performance of rigour precisely when you need it to perform the real work. The gap between sounding rigorous and being rigorous is where the real risk sits. What I want from an adversary is real pressure on actual pressure points. What the machine is built to give me first is a convincing performance of that pressure. Knowing the difference and being willing to grind through three to five rounds to get past it is still, for now, a human job. And the people who most need to know that are the ones least likely to.

Yule Guttenbeil is the principal of Attune Legal.