AI prompt engineering is hard

Generally speaking, AI is pretty dumb. "Sacrilege!" I hear you shout. Isn't AI taking over the world and replacing the entire working age population? Well, not quite, so hear me out.

AI certainly has many uses - streamlining operations, increasing security, helping find cures for diseases and many others. However, it is only as good as the input. This includes the dataset (i.e. the data used to train the LLM) and the prompt (i.e. asking it for a response). In computer science this is known as GIGO ("garbage in, garbage out") - basically, if you use bad data to start with you can't expect any kind of decent result.

In this article I'm not concerned with the data being used to train the LLM (I'm going to assume for the sake of brevity that it's decent), but rather the mechanism of getting the data out of it.

There is a whole specialism appearing known as "Prompt Engineering" - it's even started being included in job listing requirements. But how hard can it really be to write a prompt? As it turns out, pretty hard.

Taking the initiative

The issue with AI is that it has difficulty using initiative. Although there has been an exponential growth in the complexity and functionality of AI (and by proxy the amount it is used) it is still in the somewhat early stages. We are not at the stage where it is sentient and can think for itself like a human (though this could only be decades away). As stated earlier, it is using available data and giving an answer - it is not using critical thought.

Therefore if the prompt is ambigious, the output will be poor. If the AI doesn't know something you tend to get one of two scenarios occuring:

1. It will draw a blank
2. It will make something up that is completely innaccurate (known as a hallucination) 

The below is a good example of a hallucination from ChatGPT 4o. To be fair to ChatGPT, the prompt is poor as it isn't specific (we will look at that later on), but it doesn't ask for any clarification.

It confidently states that Sheffield United played 38 Premier League games in 2024, which is incorrect. As the season spans over 2023/2024, only a subset of these games were played in year 2024, not all of them.

The interesting thing is it doesn't ask follow up questions such as "do you mean the calendar year or football year?" or "do you want to include cup games and friendlies?". It just makes an assumption and is then confidently spits out garbage.

Until it starts to learn initiative we need to produce better prompts.

Prompt club

In many ways prompting AI is like talking to a child - you need to be very specific when asking them to do something. If you ask a child to "please bring me that book" they will probably bring you any old book. But if you ask them to "please bring me the only red book on the first shelf of the bookcase" you will get the book you are after (probably).

This is the same with AI. But what is the anatomy of a good prompt? There are a few things that we must include if we want to counteract GIGO. I've created an acronymn to help us here, QUBA:

• Qualifier - the context, background or audience
• User - the role that needs to enacted
• Behaviour - the action required
• Artefact - the format of the output

You also don't have to write it in plain English - instead of a long paragraph feel free to use bullets to break it up. 

Using my earlier example, let's improve it. My original prompt was "How many games of football did Sheffield United play last year?". Using the QUBA acronym we can turn it into:

Qualifier: I am doing some research into the 2023/2024 football season for Sheffield United but only for the calendar year 2024.
User: You need to take the role of a football fan.
Behaviour: Find all the fixtures that Sheffield United played during the year 2024.
Artefact: Print out bullets to separate the league and cup games.

The result:

This time it has gone super detailed - too detailed in fact. I originally just wanted to know how many games Sheffield United had played, but it has now listed them all out. The prompt needs further tweaking:

Qualifier: I am doing some research into the 2023/2024 football season for Sheffield United but only for the calendar year 2024.
User: You need to take the role of a football fan.
Behaviour: Find all the fixtures that Sheffield United played during the year 2024 and create totals of league and cup games.
Artefact: Print out bullets to separate the league and cup games. Do not include the individual results.

The result:

This time I have what I wanted - the totals listed out. It made up "the Blades demonstrated resilience and determination throughout these fixtures" but that's a different story!

If you are not getting the output you are expecting try giving this simple prompt a go. And if you're still not getting what you need drop me an email.