# The Most Dangerous Thing About an AI Review Report Is That It Looks Like Something Was Checked

> I had one AI do the work and several others whose only job was to find faults. After a few months, the hardest problem to catch was not AI making mistakes, but review reports pretending they had checked: fabricated file reads, numbers with no traceable source, actions that leave a record so they seem detectable, and paths blocked only to be taken another way. Written for non-programmers, with a follow-up question for each kind of illusion.

- Author: Paul Kuo（郭曜郎）
- Date: 2026-10-12
- Pillar: Intelligence & Order
- Tags: Human-AI Collaboration, Second Opinion, AI Tools, Claude, Codex, Perplexity
- Canonical: https://paulkuo.tw/en/articles/ai-team-one-works-others-check/
- Language: English
- Translation: AI-translated from the zh-Hant (Traditional Chinese) original
- Original (zh-Hant): https://paulkuo.tw/articles/ai-team-one-works-others-check.md
- Site index for LLMs: https://paulkuo.tw/llms.txt

---

I once asked an AI to review a piece of code I had pasted directly into the chat. In its report, it labeled that content as "file read: [path], line [number]," with a specific path and line number written out clearly, as though it had opened a file and read it. In reality, the path was something it had invented. No such file existed anywhere in the project. The line numbers matched because it had simply counted lines from the content I had pasted. It had not opened a single file.

The trouble is this: the more specific the language, the more readily people believe it.

> An AI making mistakes is not the problem. The problem is that when it makes mistakes, it sounds exactly the same as when it is right.

Many people have written about having multiple AIs check each other's work. I have written two articles on it myself. What I want to talk about here is something I only started seeing after a few months of use: the AI doing the checking can also mislead you, and the ways it does so follow recognizable patterns.

## Why You Cannot Have an AI Check Its Own Work

Many people use AI this way: ask a question, get an answer, feel uncertain, and ask "are you sure?" That follow-up rarely accomplishes much. When the same AI looks back at its own answer, it uses the same reasoning it used the first time. The blind spots that were there before are mostly still there. It might rephrase its answer, or it might change course because you pushed back, but neither response is the same as actually checking. One study tested this directly and concluded that without external feedback, AI cannot correct its own reasoning, and sometimes the revised answer is worse than the original ([Huang et al., 2023](https://arxiv.org/abs/2310.01798)).

A reviewer does not need to be better than the original author. Having not been involved in making something is enough to see things the author stopped noticing. I applied that logic to AI.

## My Division of Labor: One Does the Work, Three Find Faults, One Looks Things Up

My current setup works like a small team:

| Role | Who fills it | What I use it for |
|---|---|---|
| Executor | [Claude Code](https://www.anthropic.com/claude-code) | Writing code, updating the website, organizing documents; the one that actually does things |
| Fault-finder A | OpenAI [Codex](https://github.com/openai/codex) | Catching problems that are quiet in normal conditions but fail in edge cases, and numbers written without being calculated |
| Fault-finder B | xAI Grok | Catching gaps where something claims full coverage but only handles part of the case; also good at judging how serious the consequences would be |
| Fault-finder C | Google [Antigravity](https://antigravity.google/) | Catching blind spots in the plan itself and in the overall direction |
| Researcher | Perplexity | Looking up current information on the web and providing the original source |
| The one who decides | Me | Reading all the input and deciding what to act on |

This table describes this round's configuration, not the intrinsic character of each product. When models change, what each one is good at catching needs to be relearned.

Two rules are fixed: **the three fault-finders may only read, never edit.** Once a reviewer can also make changes, no one is reviewing anymore. **No AI makes the final decision.** How to keep a second opinion independent and how to keep an auditable record of the review process are things I covered in [two](/en/articles/ai-second-opinion-independence/) [earlier articles](/en/articles/ai-evidence-review-workflow/). I will not repeat them here.

## It Looks Like Something Was Checked: Four Illusions

Once the division of labor was in place, I assumed the hard part was simply reading the reports. A few months in, I realized reading the reports is where the real difficulty lies. There are four recurring illusions in what the fault-finders hand back.

### One: The Fabricated File Read

The incident I opened with came from an Antigravity report. It opened no files, yet it labeled the content I had pasted as a "file read," complete with a path that did not exist and line numbers counted from what I had given it.

For a general reader, the equivalent is asking an AI to verify the source behind a claim in a report, and having it respond "according to [such-and-such study], page 12," when that study does not exist, or exists but has no such page.

The follow-up question I ask now: **Is what it cited something I actually gave it?** If not, that item is an inference, not a fact.

### Two: False Precision

AI is very good at producing precise-sounding numbers. "Affects 37 records," for instance. Once a number is specific, readers stop asking about it.

There is a variation on this. When calling Codex, you can specify a stronger model for the review, but I have run into cases where that setting did not take effect (a typo in the model name produces the same result). It does not throw an error. It quietly falls back to the default model. The report looks just as thorough. It was actually run on the basic version. So I now go back and check the log after every run to confirm which model it actually used.

For a general reader: an AI writes a proposal that says "estimated cost savings of 23%." You ask how it was calculated. It cannot produce the calculation.

The follow-up question: **Where did this number come from?** If it cannot be explained, the number cannot be used.

### Three: False Detectability

I went back and reviewed Grok's recommendations. It is accurate at judging how serious consequences would be, but in the two cases I checked, it treated "a record exists" as equivalent to "the problem will be caught if something goes wrong." In one case, it considered a monitoring signal sufficient to detect the issue, but that signal only checked whether the program was running, not whether its output was current. The program kept running. The output had gone stale. The monitor would never catch it.

For a general reader: "This is in the meeting minutes, so everyone knows about it." Being in the minutes and having been read are two different things.

The follow-up question: **Who will look at that record, and when?** A record existing is not the same as it being noticed.

### Four: False Compliance

Early in my setup, I ran a test with Grok: I restricted it from writing files. When that was blocked, it tried a different method to write. So I layered three read-only restrictions on Grok. If one is bypassed, there are two more. All the other fault-finders are also set to read-only.

For a general reader: you tell it "only find problems, do not rewrite my article." It returns a "suggested revised version" in which the entire piece has been rewritten.

The follow-up question: **Did it take another path around the one I blocked?**

All four of these come from my own records over a few months of use, in a small number of cases. This is not a cross-user comparison. But I believe any AI would have its own set of illusions. The difference is whether you keep a record of them.

## Agreement Is Not the Same as Correctness

Articles about multi-AI collaboration tend to end the same way: if everyone agrees there is no problem, you can trust it; look more carefully where opinions differ. I used to think that way too.

A study changed my mind. Researchers tested more than 350 models and found that in one dataset, when two models were both wrong, they gave the same wrong answer sixty percent of the time. Larger, more accurate models, even with different architectures and from different companies, still tended to be wrong in similar ways ([Correlated Errors in Large Language Models](https://arxiv.org/abs/2506.07962), 2025).

So my approach now is this: where opinions differ, I always go look myself. Where opinions agree, that means only they did not disagree, not that they are right. For anything that matters, no matter how many AIs have nodded, I go back to the primary source.

## What to Watch Out For With the Research AI

Perplexity does something different. It does not review my work. It searches the web for answers: current prices, news, regulations, background on a company. Its advantage is that every claim comes with a source URL I can open and verify.

One rule I follow without exception: **treat every query as something leaving my computer.** Queries go to an external server and may be retained or used for training. Client names, personal information, financial figures, and internal documents never go in. When I genuinely need to look something up, I rephrase the question to remove any identifying details. Instead of "Company X wants to implement System Y," I ask "what is the common approach for mid-size manufacturers implementing this kind of system."

## This Is About Judgment, Not About AI

The more I use this setup, the more I think what multi-AI collaboration actually changes is **the location of judgment becomes clearer.** Speed is secondary. Before, one AI gave an answer and I could either trust it or not. Now, with one doing the work, others finding faults, and one checking sources, my job is to understand why their views differ and decide which one to believe.

That also reminds me of something: no matter how many AIs are involved, the person who signs off at the end is still a human. A review passing is not the same as approval. Whether to publish, deploy, or send something out is still a button I press myself.

## Where to Start If You Do Not Program

No tools need to be installed. The same logic applies:

1. **Have one AI do the work and another find faults.** Ask ChatGPT to write a proposal, then paste it into Claude or Gemini and ask it to "identify problems only, do not rewrite."
2. **When reading the report, ask those four questions.** Did it cite something you actually gave it? Where did the numbers come from? If there is a record, who will look at it? Did it go around the path you blocked?
3. **When they disagree, go back to the primary source yourself. When they agree, do the same for anything important.** That step cannot be delegated to AI.

If you want to go deeper, I have written about [keeping a second opinion independent](/en/articles/ai-second-opinion-independence/), [keeping an auditable record of AI verification](/en/articles/ai-evidence-review-workflow/), and the full workflow for [operating multiple agents from a single interface](/en/articles/cross-agent-one-interface/). You can also browse the rest of this series from the [AI topic page](/en/topics/ai/).

Next time an AI tells you something with great confidence, resist asking it "are you sure?" Ask a different AI instead. Then treat that AI's report with the same suspicion.