AI: How Can Something So Smart Seem So Stupid?

Some weeks back, I asked ChatGPT to help me with a research project. I wanted to know how Atlanta’s eviction rates compared to other parts of the country before and after COVID. Not surprisingly, ChatGPT did a pretty good job: reporting back with a list of academic articles. It summarized them and pulled out the findings I cared about.  

Then I asked it to turn the list of articles into a report. All the facts came through fine. But the citations did not. In one instance, to support the fact that evictions lead to mental health problems for school children (a real and well-documented finding), ChatGPT cited a medical journal article about a drug study for the relief of rheumatism in the toes. (Yep. True story.) 

In another project, Claude started calling me James. I asked it directly, “What is my name?” It answered, correctly, “Michael.” And then, a sentence later, it called me James again. It knew the right answer. It just wouldn’t use it. And just today, after I built a tool to process my receipts, Claude Opus 4.8 told me “to run the program by just typing two words: run my expenses.” “That’s three words,” I typed back. Claude cheerily quipped, “Yep, you’re right. I should have counted.” 

If you have spent any real time with ChatGPT, Claude, Gemini, etc., you have your own version of these stories. Your AI suddenly feels less like a helpful tool and more like a brilliant colleague who, every so often and with great confidence, loses their mind. ChatGPT just took a trip around the entire internet on your behalf and produced a very nice summary of what it found, and suddenly can’t do basic math. Or it invents a research study, vacation destination, or news article that never existed.  

Training, Context, and Memory

Are these tools broken in some way? Are they just spewing random nonsense you can’t rely on? Nope. The outputs, even the hallucinations, are not random, and hallucinations don’t mean your ChatGPT is broken. Many of the reliability challenges you have with AI are due to how AI (particularly large language models–the brains behind ChatGPT, Claude, etc.) “know” things. Once you understand that an AI knows things in three different ways, the strange moments start to make sense. And, more importantly, you can start to steer around them. 

The three ways of knowing for ChatGPT, Claude, Gemini, Grok, etc. are training, context, and memory. I write about each below with some tips to help you manage them and get the best out of your AI. And the easiest way to use my tips is to just cut and paste them into a chat and ask the AI about them and how to best use them. It will tell you. 

Training: what the model already learned

Most of what an AI seems to know comes from “training.” AI companies and builders “train” a model on vast amounts of information, usually from the internet. These builders collect the internet, compress it, and feed it into the “model,” like GPT-5.6 or Claude Opus 4.8 or Fable 5. The model processes that information and uses it to figure out how one piece of information relates to another. Eventually, after a variety of training steps, the model has two salient qualities: 1) it’s capable of writing generally about most anything found on the internet up until its training ended (without looking anything up) and 2) it has been trained to be helpful, to be an “assistant.” 

But what explains the toe-rheumatism citation? In its training, the model didn’t store exact copies of everything it was trained on. It did not store all the research articles on the internet at the time of training. But it did learn the shape of things, like what a citation looks like and what housing research tends to say. And when I asked for a citation it did not actually have, it produced something shaped like a citation. Confident, fluent, and wrong. People call this “hallucination.” It is really just the model filling a gap with something plausible because it assumes that a complete answer is better than a correct one. 

Try this: do not trust facts, quotes, or citations an AI produces from its training. Give it the real sources or ask it to find and identify real sources. Tell it to use only those. And tell it, in very clear language, that you would rather have an incomplete answer than a confident wrong one. Say that “I’m not sure,” or a clearly marked gap, is a good answer and not a failure. I have had good luck with this approach. AI tools will do this when you ask. But beware: Left to their own habits, they would rather sound complete and confident than be right. (Just like humans.) 

Context: what is in front of it right now

The second way an AI knows things is called “context.” Basically, context is the conversation you are having right now. Everything you have typed, everything it has said back, and the document you pasted in is context. Context can matter more than training. What is in front of the AI right now shapes its answer more than what it learned long ago. 

That is what happened when Claude started calling me “James.” Somewhere early in that conversation the name got into the context, and it stuck. Maybe a James was an author of an article that I was using in a paper that Claude was editing for me–I don’t remember. When I asked my name directly, the AI could reach for the right fact and say “Michael.” But for some reason its behavior kept getting pulled back to the James sitting there in the conversation somewhere. It knew the right answer but got stuck on the wrong one too. 

Try this: when an AI starts going sideways, the fastest fix is usually the context. Clear the clutter. Ask the AI to summarize what matters from the chat, then start a fresh chat and paste the summary in. A short, clean conversation with the right information in it will beat a long, messy one almost every time. 

Memory: what it carries from one time to the next

The third way is memory: what the AI keeps after the conversation ends. By default, most chats have no memory after you close the chat. Close the window and it forgets you completely, unless the app’s memory feature is turned on. And even with memory turned on, it remembers only a little of what you talked about. That’s why you can find yourself explaining basic facts about your work and needs over and over again. 

But you can give it a memory. The ChatGPT and Claude apps will save some information about you automatically. (If you explore the settings, you can find and read and edit that memory.) You can also use the apps to create a “project,” with a set of saved instructions (basically, just a file it reads every time you start) and a bunch of docs you upload into the project. It will reference these docs for every chat in that project. Set up a project once and you stop starting from zero. An even more advanced approach is to use the apps as an agent. You can point Claude or ChatGPT to docs on your computer or your email and they will gather context for you. (Once you get comfortable with using agents, the product you get from AI gets much, much better.) 

Try this: if you find yourself giving AI the same background over and over again, create a project (look to the upper left of your app interface) and put that background permanently in the project instructions or source docs. Most tools now let you save standing instructions or set up a project for exactly this.  

Putting it together

Training, context, memory. They are the cause of almost every “how is it this smart and this dumb” moment. The fake citation was training filling a gap. The James problem was context running the show. The endless re-explaining is missing memory.  

These are not the only ways AI can stumble. But they cover most of the everyday frustration. Once you notice which one is in play, the surprises get smaller and fewer, and you get a great deal more out of your AI tools. 

If you try any of these tips, let me know how it goes.