Why do AI models make things up or hallucinate? OpenAI says it has the answer and how to prevent it

Artificial intelligence (AI) company OpenAI says algorithms reward chatbots when they guess, the company said in a new research paper.

OpenAI is referring to “hallucinations” when the large language models (LLMs) used to train the chatbots guess answers when they are unsure, instead of admitting that they don't know. 

The researchers say that hallucinations come from an error in binary classification, when the LLMs categorise new observations into one of two categories.

The reason hallucinations continue is because LLMs are “optimised to be good test-takers and guessing when uncertain[ty] improves test performance,” the report said. 

The researchers compared it to students who guess on multiple-choice exams or bluff on written exams because submitting an answer would receive more points than leaving the entry blank. 

LLMs work with a points scheme that rewards them with a point for a correct answer and none for blanks or for saying that they don't know the answer. 

The paper comes a few weeks after OpenAI released GPT-5, the model the company  says is “hallucination-proof” with 46 per cent fewer falsehoods than predecessor GPT-4o. 

However, a recent study from the US company NewsGuard found that ChatGPT models in general spread falsehoods in 40 per cent of their answers. 

Some questions ‘unanswerable’ by AI

Through pre-training and post-training, chatbots learn how to predict the next word in large amounts of text. 

OpenAI’s paper found that while some things, such as spelling and grammar, follow very clear rules and structure, there are other subjects or types of data that will be hard or even impossible for an AI to identify. 

For example, algorithms can classify pictures when they are labelled either “cat or dog,” but if the pictures were labelled after the pet’s birthday, the chatbot wouldn’t be able to categorise them in an accurate way. 

This type of task that an AI performs would “always produce errors, no matter how advanced the algorithm is,” the report found. 

 One of the key findings by researchers in the report is that models will never be 100 per cent accurate because “some real-world questions are inherently unanswerable”. 

To limit hallucinations, users could  instruct the LLM to respond with an “I don't know” if it does not know the answer and modify the existing points system for the types of answers it gives, OpenAI said.