I Asked 4 Robots to Do Improv
I’ve been documenting my efforts to better understand the creative limitations and capabilities of AI — for now, and for the fast approaching future. To focus my investigation, I’ve been using the lens of a popular fantasy: What if you could ask AI for another episode of your favorite TV show?
In my case, I’ve chosen Douglas Adams’ The Hitchhiker’s Guide to the Galaxy. This is to give the state-of-the-art in AI what I consider to be an impossible task. I’ve asked it to emulate the satirical and philosophical wit of Douglas Adams, an author who seemed to choose his words with such deliberate care and precision, like a comedic brain surgeon, accurately pinpointing the desired effect on his audience. That he created a body of work so impactful, timely and most importantly, funny — that it is held in such esteem by so many people, including me, shows that his work has achieved a cultural relevance (across many cultures), which I believe is the pinnacle of human artistic achievement.
So I wanted to set a high bar for AI to attempt. I expect I could pretty easily get it to create a new episode of a formulaic procedural crime drama (the clue’s in the name — procedural). Or even, given the mainstream lack of familiarity with the works of Shakespeare, along with the outdated style of dialogue, I could probably get AI to write something ‘Shakespeare-ish’ enough to fool the average layperson. However Douglas Adams’ work is contemporary enough that fans should readily hear something that sounds like a dissonant note — not in keeping with the rest of his work. Considering that his work also exemplifies originality — retreading over previous storylines should also stick out like a sore thumb, even if pointed up in a hitchhikerly direction.
The story so far…
I wrote about the disappointing first draft of this next chapter, for which I simply told it to take an example reference from the radio play and extrapolate a new chapter with the premise that: Arthur Dent reawakens from an 100 year sleep
Interestingly enough, when I asked it to criticize its own attempt at writing this, its analysis was fairly insightful in terms of its own shortcomings.
With AI, we’ve trained an analysis engine more than a creative engine
Even still when I asked it to take its own criticism into account and to rewrite its own work, the result wasn’t notably better.
Prior to that, I was curious to see if you could put it on a loop of self-reflection in which coal going in, my emerge as a ‘heart of gold’ many iterations later. Sadly, it seems to ‘rob peter to pay paul’ with regards to its own work, in that improvements in some area come with new missteps in others, leaving you roughly where you began in terms of quality.
A carpenter blames his tools
Another explanation for the poor results could be that perhaps I set the AI up to fail. The AI cottage industry has introduced the term ‘prompt engineering’. The idea is that AI can produce the correct results given a suitable prompt — the context and question being asked of the AI.
Maybe GPT-4 is already an all-knowing oracle and we need to develop the esoteric rites of our High Priests, or Prompt Engineers, to divine its secrets
I think that’s only partly true. My experiments with language models have revealed some clear limitations when it comes to their domain of expertise, in addition the the well-documented aspect of their making stuff up.
I don’t know how much of that is due to nature of the training data, or the knowledge and backgrounds of the human subjects used for reinforcement learning — did those folks grade GPT-4 on creativity and originality, or just accuracy and how human-like it sounded? I don’t know that OpenAI has disclosed those details.
Still, it remains true that changing and improving your prompt gives better results. Thinking about this, I came up with a better process for creating dialogue between characters.
Four AIs walk onto a stage…
One of the apparent shortcomings of AI produced narrative seems to be character dialogue. Some people have posted ‘AI written’ stories which are purely exposition, and about as exciting as reading the IMDB plot summary. As I listened to the original radio version of Hitchhiker’s and compared it to the chapter created by AI, I realized how much of the original story is dialogue between the characters — very little actually happens in a given scene.
So how could I prompt AI to write mostly dialogue?
I thought of an improv exercise, along the lines of how TV shows like Curb Your Enthusiasm are produced — where you have the basic story beats, and the actors improvise their lines within the scene.
First, I created a profile for each of the main characters in the scene, and prompted ChatGPT to imagine that I’m the director and the actors have been given each those profiles to use for an improv performance through the scene.
The Result
Granted I’ve taken more of an active role in steering the conversation, but this definitely achieved the goal of getting ChatGPT to spend more time on dialogue before moving the story forward.
Is it good? It is better dialogue — but it feels like filler. It hasn’t developed the narrative in any meaningful way. So in terms of creating an original story, it doesn’t get us any closer to our goal, but I think it could be a useful tool if you have a clear storyline in mind and want to flesh out the dialogue while directing it very tightly.
