Skip to main content

Illustrating Dystopian Future Poems with Stable Diffusion and Dall.e 2 AI's - Comparing Directed with Non Directed Output

Her addiction was the feeling of scoring a bargain. Image by Hugging Face Stable Diffusion Demo based on a prompt by David Arandle (TET).
Her addiction was the feeling of
scoring a bargain.

Image by Hugging Face Stable Diffusion Demo
based on a prompt by David Arandle (TET).

Far from being fearful of AI image generators like Dall.e 2 and Stable Diffusion I feel a new skill set is emerging for artists who embrace the technology. Chiefly among those skills is the ability to write 'quality' text inputs, and the ability to curate the best outputs i.e. images, if they are being fed into a bigger project or art piece.

Case in point. One of my FB friends shared an article by Jesus Diaz, AI was made to turn David Bowie songs into surreal music videos, in which YouTuber aidontknow fed the lyrics of Bowie's Space Oddity into Midjourney AI and then curated the resulting images into a video clip for the song (embed below). 

While aidontknow says they used minimal changes to the lyrics, such as to clarify characters being spoken about in the song, from my experience you don't get quite that cohesive range of images without also suggesting an art style and give more detailed direction such as camera shots etc.

Regardless, this inspired me to revive one of a series of dystopian future poems I wrote between 2005 and 2006 dealing with the human condition, virtually reality consumerism, and AIs. The poem is titled, Rachel. The video of me reciting it, is actually the very first entry in this blog (because this blog was initially going to an art piece telling the story of those poems. If you read that post it's actually the start of a story rather than a blog post).

One at a time I entered each line from my ten line poem into both Dream Studio's Stable Diffusion AI and Dall.e 2's AI with no modification to the lines other than appending; An Oil painting the style of Blade Runner the movie. Wide angle lens to the end of each prompt. This was to give every image a unified look and, when I wrote the poems, I always imagined a Blade Runner style to the art. 

You can see this Blade Runner influence in the digital art  image I created for another of the poems (below) called, Stealing. This image is the first and only complete artwork I made at the time.

Stealing. One of nine dystopian future poems written by TET in 2005-2006.
Stealing. One of nine dystopian future poems written by TET in 2005-2006.
Art by TET

Incidentally you can read more about my whole concept, and read another of the poems, The Fabulous Machine, in my TET Life blog article, Virtual Reality Addiction Meets Online Shopping and Death! I digress.

Feeding my ten lines into both AI's, for each line I generated eight images with DreamStudio (which was free at the time) and four with Dall.e 2, which I only had enough free credit left to enter nine of my ten lines.

I curated the best images from both AI's into a video presentation that includes the poems words, plus YouTube library music and other sound effects. Hopefully it gives you a sense of the poem, its mood, and what it's trying to convey.

Most of the more polished, cleaner looking, images are by Dream AI, which tended to fixate on the neon lit darkness of the city depicted in Blade Runner. While the more painterly images are Dall.e 2's, which must feel the textured oil paint look is the definition of 'oil painting'.

At this point I thought I was going to finish the project but then I started to wonder, how would my video presentation look with more directed images that actually describe more of the type of image I had in mind for each line of the poem?

For example, for the first line of the poem I originally entered this prompt:

Rachael patched in a circuit wired to her brain. An Oil painting the style of Blade Runner the movie.

For my second video presentation I entered this prompt:

Rachael, sitting on her bed, wearing a VR headset wired to a computer in her cyberpunk style bedroom, patched in a circuit wired to her brain. An Oil painting the style of Blade Runner the movie. Wide angle lens.

As you can see, a lot more detailed and not the exact line verbatim. Below is a side by side comparison of what I feel is the best produced image for each prompt, both generated by Dream AI.

Side by side comparison of Dream AI's output with an unedited, direct interpretation of the first line of my poem used as a prompt on the left. On the right the prompt included more description, along the lines of what I had in mind for an image when I wrote the poem.
Side by side comparison of Dream AI's output with an unedited, direct interpretation
of the first line of my poem, used as a prompt on the left. On the right the prompt
included more description, along the lines of what I had in mind for an image,
when I wrote the poem. 

It's entirely subjective on which image is a better interpretation of the first line. Especially as what I envisioned in my head is not necessarily the same vision anyone would imagine, reading my poem for the first time, because no one else has all the additional context I do.

Below is my updated video presentation, using the best images generated with my more detailed input prompts (same music and sound effects just to save time).

Note that I didn't use Dall.e 2 this time because I didn't want to pay for more credit. I also didn't use Dream's AI directly either for similar credit issues (I'm not like Rachael, spending all my money on zeroes and ones for fun). Instead I used Hugging Faces demo version of Stable Diffusion which is slower but essentially the same AI with a few less settings, and completely free at the time of writing this. (Insert rant here about all these AI's putting up pay walls rather than going the free, ad supported route).

Anyway, what do you think of my second video presentation?

Creating works like this really does show that the human element of generating quality prompts is very much a skill to be learned, as is the curation of the output. Not every prompt produces the results you are hoping for. Particularly if the AI fixates on the wrong part of a prompt as the main subject to highlight.

Several times in my second presentation I completely scrapped detailed prompts that I thought should get good results but were just producing garbage (never more is the computing quote "garbage in, garbage out" personified than with text to image AI generators).

As I said in my previous musing on AI's, Is Your Next Design or Writing Partner an AI?, these algorithms do not actually think for themselves. Even if you were to use a writing AI to randomly generate prompts for an image AI neither would have any concept of the output as an abstract concept, or how that concept might relate to other prompts. The human element is still key in getting the best images.

I'm tempted to try this with all my poems in this series. It seems very appropriate to use AI to generate images for poems about AI and how humans are finding more ways to hook themselves into 'the machine' for longer and longer periods at a time (not to mention the rise of corporate money machines, passively draining your bank account - did I mention all the text to image AI's being put behind paywalls yet?).

One of my original concept sketches for Stealing drawn alongside the poem in 2005.
One of my original concept sketches for Stealing
drawn alongside the poem in 2005.
I guess the ultimate experiment would be for me to execute the project in the way I initially envisioned back in 2005, with a combination of digital collage images mixed with my own hand drawn sketches. I'd also need to write the accompanying narration that links the poems together. Which I think was a kind of future noir detective story. Not sure because my first blog post in this blog is the only part of that I actually wrote.

The question is, now that I've been influenced by AI text to image generators, could I even produce what I had in mind back in 2005?  


Comments

Popular posts from this blog

Featured Animator: Christian Haynes - 'Zack In Time' An Original, Independent, Animated Series on the Rise

Christian Haynes - Zack In Time.  If you've ever wanted to create an animated TV series staring your own original characters and stories then Los Angeles based writer, director, and animator, Christian Haynes is taking those next steps of putting together a team, developing a pitch/trailer for their series, Zack In Time. Featuring professional studio quality animation, they hope the show will get picked up by an animation studio for an official series. The path he and his team are taking is one you could easily follow as they deal with real life commitments, and building a following on Instagram and Tik-Tok showcasing their work behind the scenes. TET: Tell me a little about yourself. Who you are, and why you started animating? My name is Christian Haynes and I've loved animation ever since I was a kid. I would constantly be drawing cartoon characters from TV shows and movies and making my own little homemade comic strips.  As I got older, I became a lot more interested in st

Shotcut - Free Open Source Video Editor for Windows, Mac, and Linux

Shotcut Open Source Video Editor. I've been on the hunt for a while now for the best, free, open source, video editing application out there. In Shotcut , by Meltytech , which has versions for Windows, Mac, and Linux, I think I may have found a real front runner. This won't be a feature filled review, rather it will be my first impressions after having used Shotcut on a few of my YouTube videos so far. One of my key criteria for a video editor is the ability to import any video format directly into the project. This may seem like an odd focus initially but having convert video to something your video editor can use is annoyingly time consuming, and it creates a new generation of footage, potentially with a loss in quality if you don't really know much about video format specs (that's this guy right here!). Shotcut will happily work with my OBS recordings (.FLV), and .MOV, .MP4 files that I get from two different cameras. Not only that but Shotcut doesn't hold me up

Wonder Unit Storyboarder - Free Storyboarding Software for People Who Can (or Can't) Draw

Wonder Unit Storyboarder.  As an independent and solo animator I'm always tempted to try and skip storyboarding my animated shorts because they're usually only single scene sketch comedy type jokes. As a result I have many unfinished projects that kind of petered out due to having no clear finishing line. Storyboarding your productions, no matter how small, gives you a step by step guide of every shot that needs to be completed (no planning shots as you animate). It also allows you to create an animatic that gives you a rough preview of the finished production. In short, you shouldn't skip storyboards as they, generally, increase the chance of the project being completed. Disclaimer - I'm Not a Fan of Storyboarder Upfront, Wonder Unit's Storyboarder  is not my preferred storyboarding software. However it's completely free, has a number of very compelling featu

Make Disney/Pixar Style Characters with Reallusion's Character Creator and Toon Figure Bases

The Extraordinary Tourist Classic Coat outfit created using Reallusion's Toon Designer for CC3. I've talked before how I've wanted to get into 3D Disney/Pixar style character animation since I first saw the animated cutscenes for the very first Tomb Raider game back in 1996. It's why I initially bought Reallusion's iClone 3D studio app as soon as I could afford a computer that would run it. But then Reallusion released their 3D Character Creator (CC) for iClone and I wanted to create my characters with that (and I did try with Bat Storm ). But the focus of CC was realism, even with ToKoMotion's stylised body morphs . Now with Reallusion's Cartoon Designer bundle for CC3 which features two packs, Toon Figures , and Toon Hair , designing Disney/Pixar style 3D characters just got a whole lot quicker. The two packs are the bare essentials for creating Toon style characters. Five body morphs (2 male, 2 female, and one adolescent body morph that w

Tips on How to Create Background Scenes in Reallusion's Cartoon Animator Using Your Content Library

This background is made from a mix of props from six different content packs. One of Reallusion Cartoon Animator's selling points is that it can speed up your animation workflow with its time saving features. Among those features is the software's Content Library, a collection of premade, characters, props, scenes, motion files, and more, that either came with the application or you've added with purchases from Reallusion's Content Store or Marketplace .  If you've been following along, I've been reviewing applications I'm using in the process of creating my latest animated short featuring my original characters, Mia and the Tourist (and their replica, sentient, R2-D2 droid). Previously I wrote the script with KIT Scenarist , and drafted a storyboard using Wonder Unit's Storyboarder . In this post I'll show you how I designed my backgrounds for my animation using my extensive Content Library collection. I'll give you tips for making your own sc

Learn Moho Pro 12 Free Using SmithMicro's Own 104 Video Tutorials Sorted into a Logical Order of Progression

So you've bought Smith Micro's Moho Pro 12 along with the Moho, 10 hour, 104 Video Tutorials Add On pack so you can get learning right away... only you can't. For whatever reason, the video tutorials aren't sorted into any logical order of progression making them hard to follow. Yeah, I've been there, only I've done what Smith Micro should have done and sorted all the video tutorials into a logical order of natural progression for you. That is, each tutorial builds upon what you learned in previous videos and you won't suddenly come up against some feature you've never used before, unless that feature is what the video tutorial is about.

KIT Scenarist - Free, Open Source, Screenwriting Software that Helps Research Your Ideas Too

KIT Scenarist Script Writing Software's Mascot, Alexander Cat. While you can write a script in any word processing app, if you're writing stories (screenplays) that feature characters and dialogue, a dedicated script writing app can save a lot of time formatting, letting you focus more on the actual story. Script writing apps are also very useful if you plan to send your screenplays out to production companies, or if you're collaborating with actors and other production people, who are used to scripts being in a particular standard format.  [Note: In case you're wondering there are reasons scripts follow a standard format and are always written in Courier (typewriter) font, including but not limited to; being easy to read by actors, plenty of space for notes, and the general rule that one page of a script (in this format) equals approximately one minute of screen time.] KIT Scenarist , in my opinion, is one of the best script writing apps out there for ease of use, simp