Skip to main content

Illustrating Dystopian Future Poems with Stable Diffusion and Dall.e 2 AI's - Comparing Directed with Non Directed Output

Her addiction was the feeling of scoring a bargain. Image by Hugging Face Stable Diffusion Demo based on a prompt by David Arandle (TET).
Her addiction was the feeling of
scoring a bargain.

Image by Hugging Face Stable Diffusion Demo
based on a prompt by David Arandle (TET).

Far from being fearful of AI image generators like Dall.e 2 and Stable Diffusion I feel a new skill set is emerging for artists who embrace the technology. Chiefly among those skills is the ability to write 'quality' text inputs, and the ability to curate the best outputs i.e. images, if they are being fed into a bigger project or art piece.

Case in point. One of my FB friends shared an article by Jesus Diaz, AI was made to turn David Bowie songs into surreal music videos, in which YouTuber aidontknow fed the lyrics of Bowie's Space Oddity into Midjourney AI and then curated the resulting images into a video clip for the song (embed below). 

While aidontknow says they used minimal changes to the lyrics, such as to clarify characters being spoken about in the song, from my experience you don't get quite that cohesive range of images without also suggesting an art style and give more detailed direction such as camera shots etc.

Regardless, this inspired me to revive one of a series of dystopian future poems I wrote between 2005 and 2006 dealing with the human condition, virtually reality consumerism, and AIs. The poem is titled, Rachel. The video of me reciting it, is actually the very first entry in this blog (because this blog was initially going to an art piece telling the story of those poems. If you read that post it's actually the start of a story rather than a blog post).

One at a time I entered each line from my ten line poem into both Dream Studio's Stable Diffusion AI and Dall.e 2's AI with no modification to the lines other than appending; An Oil painting the style of Blade Runner the movie. Wide angle lens to the end of each prompt. This was to give every image a unified look and, when I wrote the poems, I always imagined a Blade Runner style to the art. 

You can see this Blade Runner influence in the digital art  image I created for another of the poems (below) called, Stealing. This image is the first and only complete artwork I made at the time.

Stealing. One of nine dystopian future poems written by TET in 2005-2006.
Stealing. One of nine dystopian future poems written by TET in 2005-2006.
Art by TET

Incidentally you can read more about my whole concept, and read another of the poems, The Fabulous Machine, in my TET Life blog article, Virtual Reality Addiction Meets Online Shopping and Death! I digress.

Feeding my ten lines into both AI's, for each line I generated eight images with DreamStudio (which was free at the time) and four with Dall.e 2, which I only had enough free credit left to enter nine of my ten lines.

I curated the best images from both AI's into a video presentation that includes the poems words, plus YouTube library music and other sound effects. Hopefully it gives you a sense of the poem, its mood, and what it's trying to convey.

Most of the more polished, cleaner looking, images are by Dream AI, which tended to fixate on the neon lit darkness of the city depicted in Blade Runner. While the more painterly images are Dall.e 2's, which must feel the textured oil paint look is the definition of 'oil painting'.

At this point I thought I was going to finish the project but then I started to wonder, how would my video presentation look with more directed images that actually describe more of the type of image I had in mind for each line of the poem?

For example, for the first line of the poem I originally entered this prompt:

Rachael patched in a circuit wired to her brain. An Oil painting the style of Blade Runner the movie.

For my second video presentation I entered this prompt:

Rachael, sitting on her bed, wearing a VR headset wired to a computer in her cyberpunk style bedroom, patched in a circuit wired to her brain. An Oil painting the style of Blade Runner the movie. Wide angle lens.

As you can see, a lot more detailed and not the exact line verbatim. Below is a side by side comparison of what I feel is the best produced image for each prompt, both generated by Dream AI.

Side by side comparison of Dream AI's output with an unedited, direct interpretation of the first line of my poem used as a prompt on the left. On the right the prompt included more description, along the lines of what I had in mind for an image when I wrote the poem.
Side by side comparison of Dream AI's output with an unedited, direct interpretation
of the first line of my poem, used as a prompt on the left. On the right the prompt
included more description, along the lines of what I had in mind for an image,
when I wrote the poem. 

It's entirely subjective on which image is a better interpretation of the first line. Especially as what I envisioned in my head is not necessarily the same vision anyone would imagine, reading my poem for the first time, because no one else has all the additional context I do.

Below is my updated video presentation, using the best images generated with my more detailed input prompts (same music and sound effects just to save time).

Note that I didn't use Dall.e 2 this time because I didn't want to pay for more credit. I also didn't use Dream's AI directly either for similar credit issues (I'm not like Rachael, spending all my money on zeroes and ones for fun). Instead I used Hugging Faces demo version of Stable Diffusion which is slower but essentially the same AI with a few less settings, and completely free at the time of writing this. (Insert rant here about all these AI's putting up pay walls rather than going the free, ad supported route).

Anyway, what do you think of my second video presentation?

Creating works like this really does show that the human element of generating quality prompts is very much a skill to be learned, as is the curation of the output. Not every prompt produces the results you are hoping for. Particularly if the AI fixates on the wrong part of a prompt as the main subject to highlight.

Several times in my second presentation I completely scrapped detailed prompts that I thought should get good results but were just producing garbage (never more is the computing quote "garbage in, garbage out" personified than with text to image AI generators).

As I said in my previous musing on AI's, Is Your Next Design or Writing Partner an AI?, these algorithms do not actually think for themselves. Even if you were to use a writing AI to randomly generate prompts for an image AI neither would have any concept of the output as an abstract concept, or how that concept might relate to other prompts. The human element is still key in getting the best images.

I'm tempted to try this with all my poems in this series. It seems very appropriate to use AI to generate images for poems about AI and how humans are finding more ways to hook themselves into 'the machine' for longer and longer periods at a time (not to mention the rise of corporate money machines, passively draining your bank account - did I mention all the text to image AI's being put behind paywalls yet?).

One of my original concept sketches for Stealing drawn alongside the poem in 2005.
One of my original concept sketches for Stealing
drawn alongside the poem in 2005.
I guess the ultimate experiment would be for me to execute the project in the way I initially envisioned back in 2005, with a combination of digital collage images mixed with my own hand drawn sketches. I'd also need to write the accompanying narration that links the poems together. Which I think was a kind of future noir detective story. Not sure because my first blog post in this blog is the only part of that I actually wrote.

The question is, now that I've been influenced by AI text to image generators, could I even produce what I had in mind back in 2005?  


Comments

Popular posts from this blog

Inochi2D - Free Open Source 2D VTuber Avatar Rigging and Puppeteering Software (Part 1)

Inochi2D Creator - Free Open Source VTuber Software. If you've been looking for a way to live perform as a 2D cartoon avatar on camera, whether it be for a live stream or for pre-recorded content like educational videos, then VTuber software is a low cost (or even no cost) option worth looking into. In my previous post, How to Become a VTuber - 2D and 3D Software for Creating and Controlling Your Avatar , I took a brief look at the relatively new but completely free and open source Inochi2D  which I thought showed great potential for my own needs of creating a live performance character rig for my own TET Avatar that I use for all my promotional materials. While it is possible to live perform my character using Cartoon Animator itself, Reallusion's MotionLive2D capture system isn't great - with lip sync in particular. More importantly though, I can't exactly teach people how to use Cartoon Animator if I'm using Cartoon Animator to control my Avatar. What is Inochi2D

Wonder Unit Storyboarder - Free Storyboarding Software for People Who Can (or Can't) Draw

Wonder Unit Storyboarder.  As an independent and solo animator I'm always tempted to try and skip storyboarding my animated shorts because they're usually only single scene sketch comedy type jokes. As a result I have many unfinished projects that kind of petered out due to having no clear finishing line. Storyboarding your productions, no matter how small, gives you a step by step guide of every shot that needs to be completed (no planning shots as you animate). It also allows you to create an animatic that gives you a rough preview of the finished production. In short, you shouldn't skip storyboards as they, generally, increase the chance of the project being completed. Disclaimer - I'm Not a Fan of Storyboarder Upfront, Wonder Unit's Storyboarder  is not my preferred storyboarding software. However it's completely free, has a number of very compelling featu

Glif (Alpha) - Make Your Own Tiny AI Powered Niche Image Generator Apps

I  first heard about Glif through a YouTube video that mentioned you could get access to Flux Pro (the latest 'game changer AI' generative image model) through the site for free. While I had a vague notion from the video of what Glif was, I wasn't expecting it to be so easy to get started with, and so good with my very first results. Glif is an easy to use, low-code platform for creating tiny AI-powered generators called Glifs. While that may not sound inspiring, what Glifs allow you to do is create a tiny app that niches down to a specific type of AI generation that the user modifies with their own inputs. The best way to really understand is with an example.  My First Glif I've recently been using VivaGo's AI platform  (free and unlimited at the time of writing) to consistently generate full body characters in a front facing T-Pose that I can rig as front facing characters in Cartoon Animator. Unfortunately it can be a bit hit or miss maintaining the T-Pose part

Krita AI Diffusion - Generative Image AI For Krita is Seriously Useful, Powerful and Free (If You Can Install it Locally)

Generative AI sequence of a woman in a business suit. From sketch to refined image using Krita AI Diffusion - by TET G enerative image AI, where you describe an image with a text prompt to an Artificial Intelligence model and it produces a new image based on your prompt, is gaining a strong hold as a tool for many artists. Krita AI Diffusion brings generative AI image tools right into your favourite free and opensource, graphics editor, Krita. Not only that, if you have a computer with decent specs (and at least 10GB of hard drive space), Krita AI Diffusion is completely free. What If I Don't Have a Powerful Computer? If you're in my situation, with a computer that was around before anyone in the mainstream had even heard of generative AI, you can still access Krita AI Diffusion for free, using a cloud based AI server, Interstice  and 300 tokens, to get you started. Once your initial tokens run out, purchase 5000 more for 10€ (approx US$11.00). Tokens never expire. I would

The Ultimate Independent Animator's App and Resource List 2019-2023 - Animation and Video Life

Image created with Cartoon Animator 4. Being an independent animator is not like a studio animation job. There's so much more to do that is indirectly related to the actual task of animating. Over the years I've sought out many apps, tools, and services that can help me achieve that one single task, expressing myself through animation. Below is my Ultimate Independent Animator's Resource List for 2019-2022. It started out as a list of free or low cost apps that could help you in every stage of producing either 2D or 3D animation, and then just kind of grew from there. You may not have been looking for a Time Management App as much as you needed something to get you started in 3D animation but when those commissioned projects start coming in you'll have a head start on maximizing your time. All the apps and services on this list had to meet two main criteria: They had to be useful and relevant to an Indy Animator/artist. The base app/service had to be US$

Dollars Mocap: Full Body Webcam Motion Capture (Including Hands and Fingers) For iClone and Cartoon Animator

Even though I should be further away from the camera Dollars Mocap MONO still does a good job of  tracking my arms, hands and fingers. Ever since I wrote my series on becoming a VTuber , discovering it was possible to do full body motion capture, including hands and fingers, with just software and a webcam, I've been on the look out for any motion capture software that can bring that functionality to Cartoon Animator. Dollars Mocap is a low cost motion capture application with a free trial that I learned about through the YouTube Channel Digital Puppets  and their test video . It can record full body, upper body, arms and hands, and facial mocap from a live video source or pre-recorded video. Investigating further, I discovered not only does Dollars Mocap have a free iClone7, iClone8 character profile file download (look for it at the bottom of the main program download page), so you can use the saved motions with iClone8, they've also got a demo video for how to convert your

Review: Headshot Plugin for Reallusion's Character Creator 3

Headshot for CC3. Quite possibly the best 3D Avatar I've made of myself in any 3D application. Creating a realistic 3D human avatar is a whole lot easier with Reallusion's new Headshot Plugin for Character Creator 3. The plugin is an AI powered extension that can generate 3D digital humans from one photo. Which sounds like an amazing proposition but, in practice, if you're trying to achieve a specific likeness to an actual person, Headshot will give you an excellent base to work from. Headshot has two modes, Auto and Pro. Auto Mode Auto is well worth a try if you have an ideal photo of a front facing person that is properly lit and posed to Headshot's optimum requirements. It's also the only mode that will take a crack at generating a hair model. I grabbed an image of Harrison Ford, dragged it into Headshot without changing any of the default settings (other than specifying 'male' and selecting an 'old male' setting) and this is what I