This post focuses on one of the points covered in the Far Phase board training session on Generative AI, and complements the previous AI risk post on intellectual property.
If you have heard anything about Generative AI, you have heard about its “hallucinations”. They provide great fodder for the media, who can point out many silly things that Gen AI has produced since appeared on the AI scene (around 2021-2022). However, as Gen AI gets adopted by more organisations, how should Directors and Executives think about hallucinations, the resulting risk to their organisations, and what approach can be taken to probe whether the risks are being managed? Are Gen AI hallucinations a new risk, or do we already have tools and approaches to manage them?
In this post, I’m going to mostly avoid using the term hallucinations, and instead use the word “mistakes” since practically this is what they are. A Gen AI system produces an output that should be grounded in reality but is actually fictitious. If a person did this, we’d call it a mistake.
Types of Gen AI mistakes
The fact that Gen AI can make mistakes is sometimes a surprise to people. We are used to the idea that software systems are designed to produce valid output, unless there is a bug. Even with the previous generation of Predictive AI, care was taken to avoid the system making things up, e.g. a Google Search would return a ranked list of websites that actually exist. However, Gen AI is not like Predictive AI, and it is designed to be creative. Creating things that didn’t exist before is a feature, not a bug. However, sometimes we want an output grounded in reality, and when Gen AI fails to do this, from our perspective, it has made a mistake.
I’ve come across four main causes of Gen AI mistakes in commonly used AI models: (i) there is something missing from the training set, (ii) there was a mistake in the training set, (iii) there was a mistake in the prompt given to the Gen AI system, or (iv) the task isn’t well-suited to Gen AI.
Additionally, sometimes the guardrails that an AI firm puts around their AI model can stop the model performing as desired, e.g. it refuses to answer. I don’t consider this an AI mistake, as it is the implementation of a policy by the AI firm, and may not even be implemented using AI techniques.
Let’s quickly look at some examples of these four sources of mistakes, noting that such examples are valid only at a point in time. The AI firms are continually improving their offerings, and actively working to fix mistakes.
1. Something missing from the training set
An AI model is trained on a collection of creative works (the “training set”), e.g. images, web pages, journal articles, or audio recordings. If something is missing from the training set, the AI model won’t know about it. In the above example, the Claude 3.7 Sonnet model was trained on data from the Internet from up to about November 2024 (its “knowledge cut-off date”). It wasn’t trained on the news that Bashar al-Assad’s regime ended in December 2024, and, as of March 2025, Syria is now led by Ahmed al-Sharaa.
In fields where relevant information is rapidly changing, e.g. new software package releases, new legal judgements, or new celebrity relationships, it should be expected that an AI model can output incorrect information. It is also possible that older information is missed from the training set if it was intentionally excluded, e.g. there were legal or commercial reasons that prevented it from being included. Lastly, if a topic has only a relatively small number of relevant examples in the training set, there might not be an close match for what the AI is generating, and it might make up a plausible-sounding example instead, e.g. a fictitious legal case.
If there is an overwhelming amount of data in the training set that is similar for a particular case, and missing much in the way of more diverse examples, the AI model will generate output like the former. In the above example, the Gemini 2.0 Flash model was trained on a lot of examples of analogue watches that all showed the same time, with images of watches showing 6:45 missing from the training set, and so it generates an incorrect image of a watch showing the time 10:11 instead.
A possible way to overcome gaps in the training set is to introduce further information as part of the prompt. The technique known as RAG identifies documents that are relevant to a prompt, and adds these to the prompt before it is used to invoke the model. This will be discussed further below.
2. A mistake in the training set
An AI model developed using a training set that contains false information can output false information also. In the above example, the ChatGPT-4o model was trained on public web pages and comments on discussion forums that talked seriously about the imaginary animal called the dropbear. It’s likely that ChatGPT could tell it was a humorous topic as it ended the response with an emoji, but this is the only hint, which implies that ChatGPT may be trying to prank the user. In addition, it has made a mistake saying “Speak in an American Accent” when the overwhelming advice is to speak in an Australian accent.
Outside amusing examples like this, more serious mistakes occur when AI the training set captures disinformation or misinformation campaigns, attempts to “poison” the AI model, where parody/satirical content is not correctly identified in the training set, or where ill-informed people are posting content that greatly outnumbers authoritative content. For Gen AI software generators, if they have been trained on examples of source code that contain mistakes (and bugs in code are not unusual), the output may also contain mistakes.
3. A mistake in the prompt
The training set is not the only possible source of errors – the prompt is a potential source also. In the above example, ChatGPT was given a prompt that contained an error – an assertion that Sydney ran out of water and replaced it with rum – and this error was picked up by the AI model in its output. Often AI models are designed so that information in the prompt is given a lot of weight, so a mistake in the prompt can have a significant effect.
In the above example, in an interactive session with the Claude chatbot, the chatbot initially gives the correct answer, but a second prompt (containing an error) causes it to change to an incorrect answer.
The prompt as a source of mistakes is particularly relevant for when the RAG technique (shown in the diagram above) is being used to supplement a prompt with additional documents. If there are mistakes in the additional documents, this can result in mistakes in the output. Something akin to a search engine is used to select the most relevant documents to add as part of RAG, and if this search engine selects inappropriate documents, it can affect the output.
4. Task is ill-suited to Gen AI
Gen AI is currently not well-suited to performing tasks involving calculations. In the above example, to perform the requested task, the ChatGPT chatbot needed to count the words being used. It was asked to use exactly 21 words, but instead it used 23 words (which it miscounted as 22 words), and its second attempt was an additional word longer.
Newer Gen AI systems try to identify when a calculation needs to be performed, and will send the calculation to another system to get the answer rather than rely on the Gen AI model to generate it. However, in this example, the calculation cannot be separated from the word generation, so such a technique can’t be used.
What do to about Gen AI mistakes
Despite a huge investment in Gen AI systems by AI firms, they continue to make mistakes, and it seems likely that mistakes cannot be completely prevented. The Vectara Hallucination Leaderboard shows the best results of a range of leading Gen AI systems on a hallucination benchmark. The best 25 models at time of writing (early March 2025) range from making mistakes between 0.7% to 2.9% of the time. If an organisation uses a Gen AI system, it will need to prepare for it to make occasional mistakes.
Organisations already prepare for people to make mistakes. The sources of error above could equally apply to people, e.g. (i) not getting the right, or getting out-of-date, training, (ii) getting training with a mistake in it, (iii) being given incorrect instruction by a supervisor, or (iv) being given an inappropriate instruction by a supervisor. Organisations have processes in place to deal with the occasional human mistake, e.g. professional insurance, escalating to a different person, compensating customers, retraining staff, or pairing staff with another person.
In November 2022, a customer of Air Canada interacted with their website, receiving incorrect information from a chatbot that the customer could book a ticket and claim a bereavement-related refund within 90 days. Air Canada was taken to the Civil Resolution Tribunal, and it claimed that it couldn’t be held liable for information provided by a chatbot. In its February 2024 ruling, the Tribunal disagreed, and Air Canada had to provide the refund, damages and cover legal fees. Considering the reputational and legal costs it incurred to fight the claim, this turned out to be a poor strategy. If it had been a person not a chatbot that made the original mistake, I wonder if Air Canada would have taken the same approach.
Gen AI tends to be very confident with its mistakes. You will rarely get an “I don’t know” from a Gen AI chatbot. This confidence can trick users into thinking there is no uncertainty, when in fact there is. Even very smart users can be misled into believing Gen AI mistakes. In July 2024, a lawyer from Victoria, Australia submitted to a court a set of non-existent legal cases that were produced by a Gen AI system. In October 2024, a lawyer from NSW, Australia also submitted to a court a set of non-existent legal cases and alleged quotes from the court’s decision that were produced by a Gen AI system. Since then, legal regulators in Victoria, NSW and WA have issued guidance that warns lawyers to stick to using Gen AI systems for “tasks which are lower-risk and easier to verify (e.g. drafting a polite email or suggesting how to structure an argument)”. A lawyer wouldn’t trust a University student, no matter how confident they were, to write the final submissions that went to court, and they should treat Gen AI outputs similarly.
As you can see, organisations already have an effective way to think about Gen AI mistakes, and that is the way that they think about people making mistakes.
Recommendations for Directors
Given the potential reputational impact or commercial loss from Gen AI mistakes, Directors should ask questions of their organisation such as:
- Where do the risks from Gen AI mistakes fit within our risk management framework?
- What steps do we take to measure and minimise the level of mistakes from Gen AI used by our organisation, including keeping models appropriately up-to-date?
- How well do our agreements with Gen AI firms protect us from the cost of mistakes made by the AI?
- How have our customer compensation policies been updated to address mistakes by Gen AI, e.g. any chatbots?
- How do our insurance policies protect us from the cost of mistakes made by Gen AI?
- How do we train people within our organisation to understand the issues of Gen AI mistakes?
In conclusion
All Gen AI systems are prone to hallucination / making mistakes, with the very best making mistakes slightly less than 1% of the time, and many others 3% or more. However, people make mistakes too, and the tools and policies for managing the mistakes that people make are generally a good basis for how to manage the mistakes that Gen AI systems make. It’s not a new risk.
That said, Gen AI systems make mistakes with confidence, and even very smart people can be misled into thinking Gen AI systems aren’t making mistakes. It is important to ensure that your organisation is tackling AI mistakes seriously, by ensuring it is appropriate covered in risk frameworks, contractual agreements, processes, policies, and staff training.
