You can have a Claude Max subscription or a ChatGPT Pro subscription and still make fairly basic mistakes in the work you're using them for.
That sounds obvious when written down. It becomes less obvious when the model has just built you an application, produced a convincing report, or explained its own decisions in language you couldn't have written yourself.
There's a temptation to treat access to a capable model as evidence of your own capability. You use the best tools. You know which model to choose. You've got a workflow. Surely that means the work is good.
I think there's a particular kind of overconfidence developing around this. AI can improve the quality of what someone produces much faster than it improves their ability to judge what they've produced. The output changes immediately. Their understanding may barely have moved.
The application works, so the design must be good
Consider a document application that needs to recognise whether a piece of text has already been saved.
Someone asks an agent to build it. The agent creates a table, stores the documents, and implements duplicate detection by comparing the full text of each incoming document with the stored text. There's no suitable index for that lookup. For a batch of documents, it generates a large query with one equality condition after another.
The essential shape looks like this. This is an illustrative example, with parameter placeholders standing in for whole documents.
SELECT id
FROM documents
WHERE body = :document_1
OR body = :document_2
OR body = :document_3;
With a few dozen short documents, the application feels fine. It finds duplicates. The interface is clean. The person who built it has a working demonstration and a reasonable sense of achievement.
But each lookup may require checking a large amount of stored text. As the collection and batch size grow, that decision can become expensive. The user sees the correct result without seeing how much work the database did to produce it.
An experienced builder would want to know what counts as a duplicate, how records are identified, and what the query plan actually does. Are we matching exact contents, a normalised version of the contents, or the same document after an edit? Those are different requirements.
A stored, indexed content hash could help with exact-content lookup, with collision handling and a clearly defined input to the hash. A stable document ID could solve a different identity problem. An ordinary index could be appropriate for other lookups.
The qualification is important. Text equality isn't inherently slow, and a database without hashes isn't inherently badly designed. PostgreSQL supports equality lookups through B-tree indexes as well as hash indexes. A scan can even be the sensible choice for a small table. The mistake is accepting a design without checking whether it suits the data and the work it will have to do.
The agent may be perfectly capable of finding and fixing this. But if the person supervising it doesn't know that there's a question to ask, a successful demo can be enough to stop the investigation.
Dunning–Kruger is a useful starting point
Dunning and Kruger's original research explored a difficulty in self-assessment. In the tasks they studied, some of the knowledge needed to perform well also helped people recognise whether they were performing well. Poor performers could lack both.
It's a more useful idea than the casual version where everyone you disagree with is too stupid to know they're stupid.
It's also an idea to handle carefully. Later research has challenged the explanation and examined statistical artefacts in the familiar pattern. I wouldn't treat Dunning–Kruger as a diagnosis of AI users, or pretend that an older psychology experiment proves what today's agents are doing to people's confidence.
What interests me is the related problem of recognising quality in a field you don't yet understand.
AI adds something unusual to that problem. A beginner can now produce work with many of the visible signs of competence. The code is formatted properly. The report uses the right vocabulary. The essay has a coherent structure. The explanation sounds considered.
Previously, difficulty producing those things might have made the gap in understanding fairly obvious. Now the person has something substantial in front of them. Their confidence has evidence behind it, but the evidence may support a narrower claim than they think.
“I got this application working” is a real achievement. It doesn't establish that the data model will hold up under a different workload.
The graphic below is how I'd sketch the risk. The shaded space is work you can now produce but can't yet confidently assess. Learning and testing can help close it, though neither happens automatically just because you keep using the tool.
The model's explanation can become part of the problem
A model can supply both a decision and a plausible account of why that decision was sensible.
That can be helpful when you're learning. It can also make a weak choice harder to question. You aren't just looking at unfamiliar code. You're looking at unfamiliar code accompanied by an explanation that appears to settle the issue.
Suppose the agent says full-text comparisons keep the implementation simple and avoid unnecessary complexity. For a tiny prototype, that might be a reasonable judgement. For a growing document service, it might be an assumption that needs testing. The same explanation can sound convincing in both situations.
Someone with enough database knowledge can ask what volume the decision assumes, inspect the query plan, and measure performance with representative data. Someone without that knowledge may hear a complete answer.
There are similar examples outside software. A research summary can combine studies that use different definitions of the thing being measured. An essay can explain a quotation fluently while using it to support a claim the source never makes. A forecast can contain correct calculations built on an implausible assumption.
In each case, the surface gives you reasons to feel reassured. The weakness sits somewhere you need subject knowledge, source checking, or a well-chosen test to inspect.
That's why “I asked the AI to check it” doesn't tell me very much on its own. A second review can catch errors. It becomes more convincing when it produces evidence beyond another reassuring paragraph.
AI really can make beginners better
There's a version of this argument that becomes an excuse for experienced people to dismiss anyone who has learned to build with AI. I don't agree with it.
If someone couldn't make a useful application before and can now make one, their practical capability has increased. They don't need to reproduce every line unaided for the result to count.
Research supports real gains. In Generative AI at Work, Erik Brynjolfsson, Danielle Li and Lindsey Raymond studied the introduction of AI assistance among customer-support workers. Productivity, measured as issues resolved per hour, increased by about 15% on average, with larger benefits for less experienced and lower-skilled workers.
That is a specific work setting, rather than proof about every use of AI. But it makes “AI only helps people who are already experts” too sweeping a claim.
The effects also depend on the task. In the experiment behind Navigating the Jagged Technological Frontier, consultants using GPT-4 performed better on tasks within the model's capabilities. On a task chosen to fall outside that range, the AI groups were about 19 percentage points less likely to reach the correct answer than the control group.
Those are results from a particular model and experiment, not a measurement of Astra or Fable today. They show why a successful experience with one task doesn't establish that the next task is safe to delegate in the same way.
The distinction I want to preserve is between getting a better result with assistance and knowing when that assistance has let you down. Both can improve. They don't necessarily improve together.
Knowing the models is a different kind of knowledge
I've written about preferring Astra for writing and Fable for coding. Choosing a model that suits the work can make a substantial difference to the experience.
But knowing which model I prefer for an essay doesn't tell me whether my argument is sound. Knowing which coding agent is strong doesn't tell me whether a schema represents the problem properly.
There's knowledge of the tool, and there's knowledge of the work. They overlap, but it's easy to become quite sophisticated about the first while remaining a beginner at the second.
You can know about context windows, reasoning settings, agents, and subscription limits while missing a basic issue with how your application identifies a record. A conversation about the best model won't necessarily expose that gap. A slow database eventually will.
This is part of what I mean when I say AI can make expertise more valuable. Experience gives you more places to look when something appears to work. It also helps you decide which imperfections are acceptable for the job at hand.
That doesn't make experienced people immune. An expert working outside their field can make the same mistake, especially if earlier successes have made the agent seem dependable.
Use the output to learn what to check
The useful response is to make the work answer questions the demonstration leaves open.
For the document application, I'd want a clear definition of duplication, tests for the awkward cases, and measurements on a collection large enough to expose the lookup cost. If the agent proposes a hash, I'd want to understand what gets hashed and what happens when two records match. If it proposes an index, I'd want evidence that the query uses it appropriately.
AI can help with all of that. It can explain the design, suggest alternatives, build test data, and investigate the results. You can use it to learn the questions you didn't know to ask.
Over time, the learning should show up in what you notice. You recognise a repeated scan before the application slows down. You ask what a claim rests on before polishing the paragraph. You can explain the circumstances in which your chosen approach would stop being suitable.
For the database example, that's a much better basis for confidence than the plan you're paying for. You've seen what the system does under realistic conditions, you understand the trade-off, and you know what you still haven't tested.
Try this prompt
I'm using AI to build something in a field where my knowledge is incomplete. Review this work for mistakes a beginner could miss even if the demonstration succeeds. Identify the three most consequential assumptions and explain how each could fail. For each one, propose an observable check and tell me what result would count against the current design. Distinguish checks you've actually run from checks you're only recommending. Explain the underlying concepts so I can recognise similar problems next time. Ask for missing context before making a confident recommendation.
