The Future of Animation? Using Reallusion's Cartoon Animator and Image to Video Generative AI to Animate Sequences

Animation software is already capable of auto generating tweens but will AI take it to the next level and tween entire sequences of motion?

My earlier article, Reallusion's Cartoon Animator Versus PixVerse Image to Video and Lip Sync AI - Battle of the Talking Head Avatars, got me thinking about using generative AI for creating the 'tween' animation in between key poses.

This isn't a new idea, and I have explored tweening with AI before in my article, Five AI Generative Image to Video Tools For Animation You Can Try Free Right Now. That was almost eight months ago and the results were mixed and barely usable.

However PixVerse AI is exceptional at maintaining the art style of whatever image you begin with and will also let you include an end frame, making it an ideal tool for tweening, so... 

What if, in a future version of Cartoon Animator (or your preferred 2D animation software), all you had to do was create the key moments of the scene, and then the software's AI kicked in to animate everything in between each moment? PixVerse can even generate sounds, speech, and lip syncing so it's not that fantastical a thought.

Image to Video Generative AI - Is it That Good?

In a word 'No'. To elaborate, 'No, not yet.' 

I tried a number of AI sites with Image to Video capabilities both with and without adding an end frame. All of them failed my test, including PixVerse. 

In every case I was using the free account for that AI site. Some sites have image to video behind their paywalls so those I completely skipped those.

PixVerse's free account with it's 60 free daily credits is so good, it's the only one, of two sites, I've paid money to for a subscription (the other is ArtFlow - same reason). I also like that PixVerse will let you buy a minimum of 500 credits for USD$5.00 without committing you to a subscription that you need to remember to cancel. 

Incidentally, try PixVerse's free phone app. I'm having a lot of fun with it, taking photos of ordinary, everyday things, and animating them with PixVerse. If you're that way inclined you can create clips of you being hugged by Jesus... or a Polar bear... even a Tiger!

The Test - Highlights

For my test I used the start and end (well almost end) frame from a demonstration video of how to make a character in Cartoon Animator mount and ride a horse. I used the same descriptive prompt in all cases, "A cowboy standing next to his horse, gets on the horse and rides forward toward the left of frame."

On the left is my start frame. Right is my end frame that I used if the AI tool had an end frame option.
In every case I used the prompt "A cowboy standing next to his horse, gets on the horse and rides forward toward the left of frame."

My notes on how each faired are below. No single AI nailed it on the first try, and in most cases I'd used my free credit up so I couldn't retry - which is typically what you do when prompting an AI. You try a prompt. It doesn't quite work so you keep tweaking the prompt and retrying until you get close to what you want, or give up after 100+ tries in frustration.

If you want to see each one's output I've made a compilation video you can view at the end of the article.

PixVerse AI - the only one to not change the art in any way. It animated the cowboy walking toward the horse perfectly, and riding the horse perfectly, including animating the horse, shadows and more. It just completely failed at animating the character getting on the horse. Instead a clone of the character appeared from nowhere on the horse, while the original character almost literally disappeared into the horses behind.

Krea AI - was the best at following the prompt with the cowboy walking to the horse, mounting it, and riding away. How he got on the horse was not in anyway realistic. He just kind of floated into position. The whole animation was let down by the cowboy's artwork being noticeably changed and distorted, and the background morphing between trees appearing and disappearing and even a whole new horizon appearing. Horse was animated correctly and walked well.

Vivago AI - Actually the next best for maintaining the artwork style, and even followed the prompt quite well. Clearly got confused with limbs crossing and morphing into each other in that way AI tends to do quite often.

Toon Crafter - Last time I tried this I couldn't get it to generate. This time the free version worked but only produced two seconds of a whole new cowboy morphing out of thin air and had no idea how to animate the horses legs. Weird because ToonCrafter is intended for this very task. Don't waste your time. Something also failed and I couldn't download the result, so it's not in my compilation video.

Kling AI - Gave me two trial uses of its Professional mode and it did relatively well with keeping the art style and having the cowboy mount the horse in such a way that, initially, you think it's correct until you realize his leg passed through the horses body. 

Pollo AI (Using Kling 1.6) (referral link) - Pollo AI lets you try a bunch of different AI models in the one place so I had another try with Kling. This one had the most interesting cowboy mounting the horse technique where he climbed up the horse's tail. You'd never mount a horse that way, but still believable and the animation on the tail was inspired.

I don't know what Skyreels was thinking but
none of this was asked for in my prompt?

Hedra - known for the most realistic talking heads, Hedra recently expanded their service to include other AI models, making the site a one stop shop. I took the opportunity to try a bunch of models including Skyreels 12V, Hunyuan 12V, and Minimax 12V, none of which produced anything useable. Skyreels in particular introduced a whole bunch of random new elements to the scene.

Trying PixVerse Again with a Detailed Prompt

To be fair my prompt was very basic but easily interpreted by a human (I hope). Just to end the experiment I bought some credit for PixVerse so I could try a more detailed prompt, and then tweak it until I got what I wanted. A screen shot of my final prompt is below.

My final detailed prompt for PixVerse after seven attempts:
"A cowboy, standing next to his horse, walks towards his horse. The horse senses his approach but remains in place. The cowboy holds onto the saddle and lifts his left leg up to the saddle stirrup, letting the stirrup take his weight as he mounts the horse.  The Cowboy, holding the front of the saddle, and horse then ride off at walking pace toward the left of frame."

I gave it seven tries, adjusting the prompt a little each time. The final version is in the compilation video but it's still not right. I could not get my cowboy to realistically mount his horse. Also, through several tries the AI decided to have reins randomly appear out of nowhere.

Honestly, there comes a point where re-prompting just gets ridiculously frustrating. I can't imagine spending my time crafting prompts to get the exact animation I need.

AI is good if you're happy to compromise a lot... a real lot. When you need something very specific, particularly if it involves maintaining characters, art style, and scene consistency, we're not there yet.

The kind of tweens AI can handle, doesn't really require AI. If I gave a human animator my two frames and just said "animate the cowboy getting on the horse and riding away". Even if the animator had never seen anyone get on a horse, or know how a horse actually walks, they're smart enough to look for reference and give me exactly what I asked for the first time.

Still, it's an intriguing idea that one day we may be able to simply create key moments of a scene, and have AI do all the animation in between.

Below is the compilation video of output from various image to video AI's. The second last clip is the final detailed prompt to PixVerse. The last clip is my original demonstration animation, animated by me in Cartoon Animator.

o---o--- ---o--- o---

Did you find this article useful?
Subscribe to my newsletter and get the
latest articles delivered to your inbox.