How Do I Deal With AI Accuracy Issues?

We should probably start by defining exactly which type of AI the question is about, and whether the asker was conscious of this distinction or not. When issues of accuracy come up, it’s almost always about Large Language Models (LLMs), commonly referred to as chatbots, with the best known being ChatGPT. There are plenty of areas of generative AI where accuracy is not an issue, as well as plenty of ways of using LLMs where it isn’t either. The current concerns about accuracy mostly stem from the use of chatbots to get factual answers to questions. There are no doubt several reasons why we tend to view AI chatbots as cousins to search engines, but it’s unfortunately leading to a common perception that Q&A is their main function.

The ability of the big LLMs to answer questions on pretty much any topic is indeed impressive but it’s actually one of the weaker capabilities and areas of potential they have. There is a much wider field of work they are underway with transforming (through pattern recognition, process automation, summarisation, translation, novel discovery etc. etc.) that is at risk of being overshadowed in the public consciousness by their oracle-like image.

Anyway, let’s get to the question because it’s true; they sometimes make mistakes on factual points. These are known as hallucinations. The first step to dealing with them is to understand how and why they occur. Unlike the bugs in computer code, they are not errors but are in fact features of the design of the technology. In very high level terms, the LLM works as a kind of hyper-powered autocomplete tool. It takes a sequence of words, assesses the probability of a range of possible words that could come next and makes a decision on one, based on lots of points of reference and context. If it is presented with the words “the cat sat on the”, it will compile a list of likely next words (mat, floor, bed, laptop etc.) and pick one via a probability calculation. That calculation will use all the context it can from the input you’ve given it. If you’ve been writing a lot about living rooms in your discussions with it, it would rank “sofa” with a higher probability of being the next word than if you’d been writing about bedrooms.

It can do all this because it has read millions of sentences that contained the sequence “the cat sat on the” and knows how many times any subsequent word follows and in what circumstances. So, why does it sometimes say something wrong or unusual? This happens because, while it has read an unfathomable amount of text, there are still things you can ask it, or ways you can ask those things, that don’t match with many examples in its knowledge. Therefore, it hasn’t got as many examples to create probability scores and therefore has to guess a bit. The issue is that it will not readily admit (or realise) that it is guessing, because it’s just doing what it’s meant to do – predict the next word based on the info and context it’s been given.

I hope that makes sense and if you’re still reading, let’s look at what you can do about this behaviour.

Use the Hallucinations

Consider not using chatbots for research, or at least not for work that requires perfect accuracy. Instead, use it as a muse, to bounce ideas off, check your own work and thinking, run thought experiments with. Here, any inaccuracies can become surprising avenues of thought that jolt new ideas and ways of thinking from the ether.

In fields like marketing and creativity, there are often no completely right or completely wrong answers and there are usually many ways to achieve a goal. Here, LLMs excel at ideation precisely when they say the wrong thing. The “right” thing is almost always something that’s been done before or some bland, unoriginal concept. The 'wrong' ideas are the path to new thinking.

Review and Edit the Work

If you are in a position where you delegate work to junior staff, you are probably reviewing that work before publishing/releasing/handing it to the client. The diligence of that review probably correlates with the experience and ability of the team member. Well, chatbots shouldn’t be treated any differently. They’ll make mistakes, just like people, but they’ll make them faster.

Write Better Prompts

Learn about prompting and how providing context, examples and clear instructions can reduce error rates. Learn about how to talk to the AIs and what reasoning they respond to. Simply saying “if you don’t know, say so” in a prompt can improve things immediately.

Too often, people underestimate AI because they’ve dabbled with it, seen mediocre responses and written it off, assuming the hype is just the tech industry getting overexcited again. People should be thinking about AI more like an instrument. If I picked up a violin and started scratching away, I’d be asked to stop by anyone in earshot. Would they blame the violin for the awful sound, or would they say I just needed to get better at using it?

Wait

AI is improving at an incredible pace. Every month the state of the art is moved forward. With each release, the AIs get smarter and more capable of spotting their own errors. If you really want to avoid inaccuracies, just wait until the technology has improved so that they fall below the threshold you’ll accept. That might only be another few months.

As with most things, the more you use LLMs, the more you’ll understand how and when to use them, where their strengths and weaknesses lie. They have reached the point where even the free tools have knowledge and intelligence greater than the average human, so using them brings a world of advantage.