Pac-Man Lego and an LLM epiphany

On my 42nd birthday, I was handed a challenge wrapped in a gift box - a 2651-piece Lego Icons Pac-Man set. As I unwrapped the box, a wave of nostalgia washed over me both for Lego and of course Pac-Man. The bright, colourful blocks, the familiar click as they snapped together, and the anticipation of seeing a pile of pieces transform into a tangible, recognisable form – it was a trip back to the carefree days of my childhood.

But this was no ordinary Lego set. With its 2600+ pieces and an instruction guide an inch thick, it was a behemoth that demanded patience, precision, and a keen eye for detail. Speaking to a large number of users and engineers over the last year, I couldn't help but draw parallels between my Lego endeavour and what is being asked of large language models the majority of the time.

‘Achieve this hard task, but I’m only going to give you a little if any of the nuance and context that I would give to a person if I were asking them to do the same job’.

Language, like this Lego set, is not a simple, straightforward entity. It is a complex structure, a jigsaw puzzle with countless pieces. Each word, each phrase, each idiom carries a weight of history, nuance, and context. They are not just building blocks of communication; they are the carriers of culture, the markers of time, the reflections of our collective consciousness.

As I poured over the instructions, something particularly struck me: the organisation of pieces into numbered bags. These bags make it easier to locate the next brick required, but they also constrain my choice, effectively directing my actions towards the correct sequence for assembly. In much the same way, the addition of nuance and context to a prompt given to a large language model narrows down the pool of potential next tokens, making the model’s response more aligned with the intended query.

Consider a general prompt like, "Tell me about climate change." A large language model could respond with a broad spectrum of answers, ranging from the science behind climate change to its socio-political implications. Now, contrast this with a more nuanced prompt, such as, "Explain the impact of climate change on polar ice caps in the last decade." The latter, with its added context and specificity, restricts the model's potential outputs. The range of relevant tokens narrows significantly, honing the model's focus and driving it toward a more precise and relevant response.

Just as the numbered bags in my Lego set served to filter out irrelevant pieces, leading me to the exact components I needed to complete a specific section, contextual cues and nuanced prompts steer a language model toward generating more relevant, coherent, and valuable outputs. In both instances, whether assembling intricate Lego structures or generating human-like text, the devil is in the details—nuance and context act as invaluable guides in the building process.

Expecting AI to understand and replicate this complexity based on surface-level information is like expecting to build my Pac-Man set by just looking at the box. It's not just about knowing what the final product looks like; it's about understanding the 'why' and 'how' of each piece, each step in the process. It's about recognising patterns, making connections, and anticipating outcomes.

As engineers in this new world it's imperative that we do more than just write code; we must be architects of a new digital-human interface. While it's tempting to rely solely on the advancements in language models, or place the burden on users to adapt, such an approach sells the technology short and limits its transformative potential.

We must take the helm in actively shaping how these models interact with users. We are the mediators who must make AI not just powerful, but also accessible, intuitive, and deeply attuned to human needs. Failing to do so risks leaving an enormous gap between what AI could be and what it becomes, a gap filled with missed opportunities to genuinely improve the human experience.

The onus is on us to not just engineer but to envision, to pave the way for a future where technology truly serves humanity. Anything less is an abdication of our role as the architects of tomorrow's digital world.

As I sat there with my Lego set, painstakingly placing each block, following each step in the guide, for two whole days. I couldn't help but marvel at the complexity of the set and the attention to detail in the instructions.

Building the Lego Icons Pac-Man set was a journey, a challenge, and a joy. It was a reminder of the beauty of complexity and the thrill of creation. And it gave me an insight in to the potential of AI, a promise of a future where technology understands us, aids us, and enriches our lives in ways we can't even imagine yet.

So, as AI engineers, our task is clear. We need to understand the complexity, appreciate the nuance, and respect the context. We need to create tools that help users be more productive, do better, and more creative work. And we need to remember that our goal is not just to build technology, but to build bridges between technology and humanity.

Here's to the journey ahead, filled with challenges, discoveries, and breakthroughs. Because with the right tools, the right approach, and the right mindset, who knows what we can build?