The Prompt Whisperer

20.22, Thursday 4 Aug 2022

Mia flicked the rim of the cat food tin with her finger and the 3D model span on both their screens, the double cat face illustration with tortoiseshell hair that whirlpooled into three, no, four eyes, four and a half, blinking alternately with the UPC on the rear of the spinning can, every rotation a flashlight right to the back of the visual cortex, a half second involuntary scramble parsing the dazzle of fur and ears and whiskers, and of course those eyes before, brief blessed release, the model turned again to the barcode.

“Anybody who’s even thinking of switching brands picks it off the shelf,” said Mia. “We’re bleeding customers. And we’ll lose the client too unless we figure out a way to counter it.”

“It’s certainly hypnotic,” said Charlie. Mia watched him tear his eyes away from the animation to hunt around on his desktop.

“I can replicate the basic look,” he said and he dragged a sequence of other stills into their call, a dozen, “but never the compulsion. If I could get that then we would at least be able to work our way to something similar, to make it a fair fight for eyeballs.”

It was true. Superficially the AI-generated swirling feline images were the same, but there was none of that arresting affect.

Mia tabbed over to chat and typed into the #prompt-engineers channel.

> hey, anyone around to take a look at a weird packaging reconstruction problem?

> free, coming

said Selby.

“I’m on costuming for the new IKEA store,” said Selby. “There’s this whole outdoor range launching. The models are the same so we don’t need more characters but they all need new clothes to pose in the product shots.”

He panned around a spider’s web of lines, words and photos covering his entire screen. At the centre, a dense paragraph with rays coming out of it connected to colour-coded phrases and thumbnails of caps and clothes and accessories. He traced a path out from the centre through a dozen sunshield t-shirts, each silvered shirt a mutation of the one before.

A screen for getting work done.

“You don’t get to see the prompt with my mood board app,” said Charlie.

Writing the paragraph in the middle, the prompt, was what Selby was good at. The prompt tells the AI what image to generate. It’s an inexact science.

Charlie jumped them over to the mood board. It was a collage of photos of cats, close-ups of eyes, and Picasso’s Portrait of Dora Maar.

“Because of the two perspectives of the same face,” said Charlie.

Then some less literal images arranged round the edges: slit-scan photographs; faces in funhouse mirrors; views through old, uneven glass; intricate crayons by people on acid trips; maps of global wind patterns; a grab-bag of AI stimulus copy-and-pasted from the internet.

“I’ll go knead my dough while you run your mood board,” said Selby. “I’d like to see what it comes up with.”

“I should get back into bread again,” said Mia.

“Linseed, sunflower, hemp, poppy,” called Selby over his shoulder, standing at the kitchen counter, folding the wet dough in on itself. The camera had switched from his desk to the wide-angle unit across the room.

“Sesame, definitely,” he said. “I had a loaf delivered from a new bakery last week and the flavour was so distinct. I’m reverse engineering the seeds.”

“I’ve got the first image. Let me make a few variations so you can see,” said Charlie, tapping the run button again.

“Fennel or anise,” said Selby. “Anise, I’m sure.”

He washed his hands, walked back to his laptop, and leaning over, still standing, scrolled through the neat grid of almost identical generated images, each a soup of various combinations of cat facial features swimming in fur.

Mia and Charlie saw Selby lift both hands and type a four-fingered chord on his keyboard, then a rattle of a few more keys, then a solitary enter. Selby sat as a long paragraph of small text replaced the latest cat-soup image.

“Oh you can see the actual prompt like that?” said Charlie, surprised. It was in English but barely. Reading it was like a transcription to a room full of people all talking about the same thing but not listening to one another, snatches of words, a cut-up: basket of kittens/ two cat faces/ fur in the style of a whirlpool/ f/22 35mm/ and so on it went.

“Let me see the original?”

Charlie dragged the graphic of the competitor cat food tin back into the window and it outshone the picture generated by the mood board AI with blinding ferocity.

They gazed.

“The video’s frozen,” said Mia.

Selby’s face was large on Mia’s and Charlie’s screens, filling the webcam view as he had lent in for a closer look. Light brown hair cut short, roughly; eyes and mouth too big for his face, slim without being gaunt, quick to smile, unshaven – an even gaze, quizzical, caught between a question and a laugh.

10 seconds.

“I’m going to restart the–” said Mia, and then a flurry of keys from across the call.

“Let’s give that a minute,” said Selby, the connection stutter unacknowledged.

A progress bar began its crawl.

“You were on the right track with Picasso,” he said, “but cubism as a visual style is a gravity the AI can’t resist. You’re going to get the look but what we want is the cognitive impact of cubism. That’s what makes the cat food grab you so much. Your face-detecting neurons are hyperactivated. You’re seeing a face but you’re also seeing a face! It overheats your perception, doubleplus nature. So we need a phrase in the prompt that directs the AI in latent space, but concisely. There’s a professor who has linked cubism and cognition in this way. We can use his name as a token.”

Selby had barely altered the original prompt on screen, adding to the end just one word:

> ramachandran

“The neuroscientist?” said Mia. Then the progress bar completed, and the AI-generated image from Selby’s prompt blinked up, next to the original.

Two cat food tins rotated lazily together and identically, the original and its twin plucked, impossibly plucked from the infinity of latent space. Charlie and Mia stared, caught in the headlights of a miracle.

“How–” said Charlie.

“I need to tell the boss,” said Mia.

Selby grinned.

From Mia to Lawson:

> uncanny accuracy from one of our prompt engineers. you need to see this

> let’s get on a call

replied Lawson.

“If you think of what a square of pixels can be,” said Mia, “it’s every Picasso sketch. It’s every kid’s drawing. It’s the Coca Cola logo. It’s every stock photograph ever.

“It’s a photograph of the night sky – whatever you like, any constellation there is and any constellation there isn’t. It’s a diagram of a new computer chip that goes ten times faster. It’s a handwritten shopping list, and it’s a concise proof to Fermat’s Last Theorem. It’s an illustration for cat food packaging that somehow you can’t take your eyes off.

“That’s latent space. The space of all possible images. It’s endless.”

Lawson had founded and continued to run the agency, although these days spent most of his time talking to clients. Mia had seen his work from the early days – sharp, deft, and even better with words than visuals. A storyteller of other people’s stories. Only two or three years ago AI assistants for creative work were a novelty; he’d stopped being hands-on before they graduated into essential tools.

Lawson and Selby were side by side on Mia’s screen. Lawson was standing, AirPods in. It was a large room. All cream, a low cream leather sofa against the back wall, another, identical, to its right at 90 degrees, the two neatly framing a square coffee table. The camera had panned over from the dark wood table when he crossed the room earlier, and now he stood in-front of the large print by the sofas and listened, intent.

Selby sat wearing large headphones, kitchen in the background, arms by his sides. Apprehensive, guessed Mia. Selby was unlikely to have been in a room with Lawson before, virtual or otherwise.

“A prompt is a pirate’s map,” said Mia, “Directions to hidden treasure in latent space.

“A prompt is just words, that’s all. A paragraph. Maybe two. The AI that understands the words has been trained on every sentence ever written. You can’t ask it questions but it gets any and every reference you might make.

“How do you describe your way to a picture that you have in mind? It’s a process of trial and error. That’s why we have prompt engineers. People who can deconstruct what a client wants to see and write in the form of a prompt, and develop it from there. They’re explorers. They write tools to automate exploring. It’s tough going, hacking your way through the jungle of possibilities.

“How do you look at a picture and figure out what prompt got you there?

“You can’t. It would be like me showing you a photo of a grain of sand and you telling me on what beach, at what spot, at what precise point… at best you could make an educated guess of a part of the world. But putting your finger on the exact grain? No.

“Yet Selby can.”

“There’s a Bridget Riley behind me, a study for High Sky,” said Lawson, looking back at a colourful, grid-like geometry framed on the wall. “Can you write a prompt that will generate something like that?”

“Yes,” said Selby, “but I wouldn’t need to.”

“Training. Don’t forget the AI has already seen every image on the internet,” said Mia.

Lawson walked to the table and opened a sketchbook; the camera followed him.

“Okay, something new then.”

He drew a large loop in a single bold stroke, a circle coming to a point at the top right. Inside, next to one another, two smaller circles.

“Craters,” he said, “16 Psyche. A logo for a startup I know. Work in progress.”

He held the paper to the camera. Mia could see gravitas and upward force even in the strokes of the hand-drawn sketch. A glimpse of the original Lawson, she thought, a talent.

Selby examined his keyboard and even over video Mia could see his embarrassment.

“I can reverse engineer prompts,” he said, “that’s what Mia saw. Give me something that was generated and I can tell you the prompt that was used to get there. But not anything that wasn’t made by an AI. What’s it good for? It’s a party trick.”

Lawson stood and looked steadily through the screen.

“Or rather,” – he was grasping for words now – “I can engineer a prompt for one of your originals. That’s what I do. It won’t be exact but we can work with it. Show me the logo again and let me see what I can put–“

He trailed off and started typing something. Opening applications, setting parameters. Fake busy.

“Lawson…” said Mia.

Lawson shifted his weight onto his other foot and put two fingers on his chin, over his mouth. He continued looking forward, in silence.

Then he turned and walked past the table. The camera panned as far as it could until he walked off screen. Mia waited. Selby continued with his tapping.

A lilac astronaut on a lilac planet; a low-shot fisheye photo, boot striding forward over the sand dune, a woman’s face through a transparent visor staring with strength down the barrel of the lens. Hips cocked. “COSMOPOLITAN” stamped in blue across the top, and the familiar telltale in the bottom right, a visual tag marking this image as being AI-generated.

“This magazine cover was never published, or rather,” said Lawson, from behind the framed print, lifted from the wall, “not this variation. It’s from my own collection.”

Selby looked up and leaned into the camera. He held his stare without blinking. One breath. Another. Two more, slowly. Then the sound of him typing for a few seconds.

“It’ll take a minute to come back,” he said.

Mia exhaled.

“It’ll make our projects much quicker,” she said. “And cheaper. It’s something our clients ask for. We can build a whole proposition around this.”

“No,” said Lawson.

“You said something about computer chips,” he said.

“I haven’t been outside for two years,” said Selby.

“Lawson says that everything is designed by AIs now,” said Mia, “like how chips work and how phones look. It’s why chips are so fast, he says.”

The two of them on their screens, talking together the next day, no-one else.

“I stopped and didn’t start again. It hasn’t come up. So now I stay in,” said Selby, “and I really think we can carry on over video.”

He paused kneading and stood at the kitchen counter, leaning on his hands. The camera hadn’t switched over so Mia couldn’t see his face; she saw him side-on, silhouetted. But she could see him breathing and the effort to control those breaths.

“But none of these breakthroughs are shared, he says. When they come up with a new chip then it’s ten times faster but the prompt is secret.

“What you can do is not just a trick and Lawson asked me to ask you to visit so he can see it in person,” said Mia. “He says that progress should be for all of us.”

“Progress should be for all of us,” repeated Selby.

“I didn’t know that you’re indoors-only,” said Mia.

“If we unlock the prompts then we unlock drugs, medicine, better batteries,” said Selby. “It means everyone will have access. Cheap access.”

“I understand it would be difficult for you. You should stay at home. I’m sure we make accommodations.”

“No more prompt-squatting billionaires,” said Selby, dusting his hands on his apron and turning towards Mia.

“Don’t travel. We’ll find another way.”

Selby: “I’ll come.”

Sat low on the cream sofa, Mia looked around Lawson’s Kensington apartment. It was strange, looking back at the never-seen camera, seeing the familiar room flipped into its mirror image: the dark table to her right and not on the left. Beyond it she could see bookshelves and, through an arch, windows in the next room. A dining room? A boardroom? On the other side, also off-camera, there was an open staircase leading down – they had come up that way. Soft carpets throughout. They had been asked to remove their shoes. Closed doors presumably led to bedrooms, other living spaces, it was hard to tell. Was this a home or an office? Lawson’s family wealth was more obvious this side of the lens.

Selby sat to Mia’s left, having pushed himself into the corner of the seat. Arms folded, hands tucked in, his chin pushed into his chest.

On the other sofa: a woman, Hope, who Mia had met for the first time today, then Lawson, tailored in black as always, hands pressed together as if praying. Concentrating.

Hope had just asked something.

“It’s a kind of synaesthesia I think,” said Selby, replying. “I see the thing and I can read the prompt. I know there aren’t any words. I can’t see any words. But it feels like reading.”

Lawson had introduced Hope as a “computational astrophysicist,” someone he knew somehow who he asked about things. He had been vague.

Hope pulled her phone from the pocket of her jeans and swiped over to a circuit diagram in her photos.

“It’s one of the new AI-generated chip cores,” said Hope, “not the whole thing and not a proprietary one. One that was published. I downloaded it earlier. Nobody really knows how it works except the machine and it’s not telling. But it’s fast.”

It looked like a fingerprint. She handed the phone to Selby who took it, reaching out with one hand as little as possible to do so.

“Um,” he said, and hunched even further over.

They all sat for a minute. Lawson pressed his hands together more tightly. Selby, rock still. And then–

Selby stretched forward to the tablet on the tablet and typed using the on-screen keyboard: one paragraph, two, three paragraphs. Half English and half algebra.

“Try that,” he said, and nodded slowly, still not looking up.

“I believe you,” said Hope. She had been testing Selby for half an hour already. She had an open, friendly face, and was smiling in the way Mia imagined she would smile at a nervous child.

Hope reached over to her phone (Selby clasped it) and swiped to the next photo. A diagram of a folded protein, all coils and ribbons. The watermarked telltale showed it was AI-generated.

“This breaks open the SARS-CoV-2 capsid. It’s why we no longer get Covid. It’s used in drugs that are sold for… a lot. The DNA sequence that leads to it is unpublished. Secret.”

This time Selby bent over the phone for long enough that Lawson and then Mia stood to stretch their legs. Mia walked to the table and picked up the framed Cosmopolitan cover that had been left there. Lawson paced slowly and deliberately across the length of the room and back, and then again. Hope sat with him looking curious, calm.

Mia looked up when she heard Selby typing on the glass of the tablet.

She felt she was witness at an event as silent and as momentous as would be an unexpected eclipse of the Sun. Mia gave an involuntary glance out of the window, cold in a sudden invisible shadow. No. The sunlight persisted, bright, fierce, blind.

“Who would like a coffee?” said Lawson. He strode to a Nespresso brewer and tray of mugs on a console table that would be unseen, if they were on-screen again. Yet here they were all together.

“Could you do this phone?” asked Hope. “Industrial design is done by prompt engineers now.”

Selby turned Hope’s phone in his hands, examining its curves and construction, and gave a small nod.

“I was just wondering,” she said, and gently took her phone back. She glanced at it and took in the notifications. The phone wallpaper was a broad oval speckled yellow, blue and red.

“What’s that?” said Selby.

“The cosmic microwave background,” said Hope, “the uneven pattern of the universe as far back in time as we can see. It’s a radio map of the sky, before any stars or any galaxies. A seed! Everything that we see now” - she gestured outwards slightly with both hands - “is an evolution of that pattern. It’s a prompt too, in a way. The prompt that made the universe. Ha.

“Lawson told you I’m in astrophysics?”

“Something like that,” said Selby. “He said you look at things for him.”

“I do,” said Hope, “sometimes. I like it when he calls. It makes a change from the lab. And paperwork.”

“So why is the sky speckled like that then?” said Selby.

“Well that’s the mystery, isn’t it,” said Hope, and she handed her phone back to Selby so he could see. “Why should it be anything at all? This is the first full map, made back in 2013. Seeing this is why I got into astrophysics. Gazing into the face of God, right?

“And now it’s my wallpaper. We don’t know why the universe looked like this a single second after the Big Bang. If it had looked any different then it wouldn’t be us sitting here. So it turned out alright.”

Selby looked down again at the map of the ancient sky. Wonder.

“Mia,” said Lawson. She walked over to where Lawson was feeding capsules into the coffee machine, making a new cup with each one.

“We don’t need to blow up trade secrets with this, not publicly,” he said. “There’s more value in digging up the treasure ourselves.”

Mia frowned.

“Pirate maps,” said Lawson, “remember? We get the prompts for the secret AI-generated chips and drugs and the rest, then we prompt engineer our way around those x-marks-the-spots and sell the artefacts to the highest bidder. A better place in the value chain.”

“I’m–” said Mia, “I’m not sure that’s why Selby left his house today.”

“Hope,” said Lawson, “come show me how you want your coffee.”

Mia, Lawson and Hope were standing, talking over their coffees, when Mia turned and saw that Selby hadn’t moved, and in fact was still sat curled over Hope’s phone, its screen now dark.

Mia hurried to the sofa and touched Selby on the shoulder, then shook him after the lack of response. His eyes were wide and fixed unseeing on the phone held in both hands, rictus fingers become taut claws.

“Selby,” said Mia, and shook him harder this time. Selby gave a sharp single nod and a strangled grunt and clenched his jaw even tighter, his whole body locked.

Lawson and Hope came to the sofa.

“Get an ambulance,” shouted Mia. “What’s your address? What’s your address? I think he’s wet himself.”

Lawson began to dial.

Hope sat next to Selby and put an arm around him. He didn’t move. The tablet was on the coffee table in front of them both, its glowing screen showing the history of prompts that Selby had been typing to the AI.

“They’re on their way,” announced Lawson.

“Hold on,” said Hope softly to sightless Selby, this rigid body. “There’s an ambulance coming.”

She held him more tightly and the room shrunk to just them, their sofa, the table. Hope felt trapped in the moment, stuck in a web, claustrophobic. Her eyes darted then landed on the tablet, a record of the afternoon.

There were prompts for art, they’d started there. Then the cat food packaging, to see that again.

Then the chip. Then the protein. Several paragraphs, both of those.

Then a prompt Hope didn’t recognise, one that that he must have typed while the other three had not been watching. Selby’s final synaesthetic reading, the cosmic microwave background itself.

> let there be light

he had written.

More posts tagged:
Follow-up posts:

If you enjoyed this post, please consider sharing it by email or on social media. Here’s the link. Thanks, —Matt.