Skip to main content

How to Become a Virtual YouTuber (VTuber) - 2D and 3D Software for Creating and Controlling Your Avatar

TET VRoid Avatar in VSeeFace.
Young TET VRoid Avatar in VSeeFace.
In a previous YouTube video I demonstrated how you could use a webcam, Cartoon Animator and its MotionLive 2D Facial Capture system to create a virtual avatar that you could perform live during a live stream. Unfortunately the motion capture performance of this system is very average to say the least, and is even more expensive if you want to add hand motion capture via a Leap Motion Controller.

This lead me down a rabbit hole of seeing what other solutions are out there, where I bumped into the world of the VTuber or Virtual YouTuber (creators who puppet and voice an on camera avatar instead of appearing on camera themselves). Specifically VTuber software, which has been a real eye opener for how expensive and far behind Reallusion's upper body motion capture system actually is.

Free 3D VTuber Avatar Creator, VRoid Studio.
The VTuber world is heavily entrenched in the world of Anime so you'll see a lot of 2D and 3D avatars in this style. If fact, if an Anime avatar is your thing, you can be up and running with a customized 3D Avatar within an hour using VRoid studio, a Free 3D Anime character maker specifically designed for this purpose, and VSeeFace, an also free, VTuber avatar puppeteering program.

The Gold Standard in 3D VTuber Software (IMO)


One of the best examples of 3D VTuber software I've come across is Webcam Motion Capture (yes that is literally the application's name). Free to try, and a small monthly fee to remove the aggressive text watermark, it seems like the gold standard of upper body motion capture.

Using a standard webcam and some AI wizardry this software can capture facial expressions including lip sync, head and upper body movement, arms, hands, and individual finger movement, very smoothly in real time.

Webcam Motion Capture in action using my custom VRoid Avatar.
Webcam Motion Capture in action using my custom 3D VRoid, Anime Avatar. This application has everything you would want in upper body motion capture all from a single webcam. While it's completely free to try the aggressive text watermark makes the free version totally unusable for anything but testing. That said it's only US$1.99 per month or you can buy a lifetime license.

The system is so good I feel Reallusion should drop everything they're doing with upper body motion capture and use this instead - seriously. It has full hand and finger motion capture with no extra hardware to buy.

If that wasn't enough it has so many additional settings you can adjust to not just fine tune the motion capture but to also enhance your avatar's performance, like automatic eye blinking, hot key expressions, or idle motions for when you're not doing any movement yourself. As well it supports the .VRM format which is the standard avatar file format most of these systems use (so you can load your VRoid characters into it).

VSeeFace and Kalidoface 3D (Free Alternatives)

If you're looking for a free alternative to Webcam Motion Capture try VSeeFace a very popular, free app used by VTubers. Watch this very good video by Vtuber, Fofamit, to get you started: 

VSeeFace does support the addition of a Leap Motion Controller to capture hand data but it turns out you don't actually need one. Install this free hand motion capture softwareVideo here by VTuber, Lady Aska, to explain how.

If you don't want to install anything then try Kalidoface 3D VTuber app. It's completely browser based and you can use your custom VRoid Avatars with it.

Kalidoface 3D featuring my TET VRoid Avatar.
Kalidoface is free, browser based, and has full body motion capture. Here I've loaded my TET VRoid Avatar into the interface.

Most of the software I'd found so far was 3D based so I kept going to see what was out there for 2D VTubers.

The PNG 2D VTuber (The simplest way to be a 2D VTuber)

This is not the first way I discovered how to make and puppet your own 2D VTuber Avatar but it's so easy to set up using Cartoon Animator and the completely free PNG Avatar VTuber software, Veadotube Mini.

Unlike most VTuber software this one does not use a webcam or motion capture at all. Instead it works based on the sound from your microphone. In essence it's a flipbook style software that switches between four states of your PNG Avatar... 1. Eyes open, mouth closed, 2. Eyes open, mouth open, 3. Eyes closed, mouth closed, 4. Eyes closed, mouth open. This doesn't sound like much but it's enough to create very rudimentary lip syncing with random eye blinks at any time.

Creating my PNG images in Cartoon Animator for use in Veadotube mini shown in the lower right.
By setting up the maximum image size of 2048 pixels square in Cartoon Animator it's easy to create all the transparent PNG images you need to make a PNG VTuber Avatar in Veadotube Mini (shown lower right). All you need is a microphone.

In addition to that you have three different movement states you can apply depending upon what mouth state is being displayed.

For even more flexibility you can add different states of these images and movement, that you can switch with your assigned hot keys. For example I made a second state of my character talking excitedly with slightly wider eyes and hands raised up in the air.

However because this software is literally flipping images you can do almost any pose or style of character you need. Side view, talking horse... whatever you can dream up. VTuber, Syafire, has a full video on how to set Veadotube Mini up in 4 easy steps.

Also worth mentioning... If you like this style of VTuber Avatar there is a low cost, paid software called Honk that is worth checking out. It's a little more complex, supporting layered characters that allows  better lip sync capabilities.

Advanced 2D VTuber Avatar Creation

Once you start looking into creating a more advanced 2D Avatar with the ability to add facial expressions, head turning, physics and more, you may start thinking being a PNG VTuber is a great compromise (or maybe you'll just stick with a VRoid Avatar - the easiest option of all).

Live 2D Cubism Editor

Finding a free or low cost 2D avatar character rigging application is not easy. There seems to be only one name in town, Live2D Cubism. On the plus side they'll give you a 42 day full feature trial of their VTuber rigging software, which is enough time to learn the process. I'm told it's simple once you know how but, at first glance, it's overwhelming.

Live 2D Cubism VTuber Rigging Software with my TET Avatar being rigged.
The early stages of my attempt to rig my character with Live 2D Cubism. I imported the same mouth and eye sprites I used for the character in Cartoon Animator but these may not be usable as Live 2D uses deform tools to create most facial movement and expression.

Preparing your avatar for rigging is a relatively simple case of separating all the sprites into individual parts in a .PSD file with no group folders. Everything is placed in layer order from top to bottom.

While I tried to rig my TET avatar in Live 2D Cubism I wasn't able to learn it quick enough to demonstrate it here. Even with the many YouTube tutorials I was overwhelmed - and my avatar isn't nearly as complex as some of the Anime avatars many VTubers create themselves and use.

Inochi2D VTuber Rigging Editor

I was able to find a free, opensource 2D Vtuber rigging software called Inochi2D Creator. Unfortunately it's a completely new VTuber avatar format that only works with their Inochi2D Sessions puppeteering software. However it can import the same .PSD files you would use for Live2D Cubism as a starting point.

Inochi2D Creator User Interface with my TET Avatar.
As you can see Inochi2D Creator had no trouble importing my .PSD file. From what I can tell the rigging process is similar to Live2D Cubism but this is as far as I got prior to writing this post.

The other issue is Inochi2D is still in development. I wasn't able to determine if it is still an active project or not however the documentation for Creator seemed quite good, while the same can't be said for Session, which is almost non-existent. While I was able to load an example avatar into Session I wasn't able to work out how to hook up a motion capture solution to it, despite it supporting VMC motion capture that I had (and you will too if you download DSSBodyTrackor2 that is used to add hand capture to VSeeFace, mentioned earlier).

I did find an excellent YouTube Tutorial stepping you through the whole process of using both Inochi  Creator, Session, and hooking everything up to OBS by VTuber Dragon Dragonnator96 - How To Rig a VTuber Model with Inochi2D (0.7.4.1 Creator and 0.5 Session). Definitely worth searching YouTube for more as well.

2D VTuber Puppeteering Software

Once you have your VTuber avatar rigged with Live2D (not Inochi Creator because their format only works with Session) you need an application to puppeteer it with. 

Most applications focus on facial motion capture so your avatar will lip sync and emote in line with your own facial movement. Other body motions and physics based motions are usually achieved with pre-set up auto-animated loops, or triggered with hot keys (such as extreme facial expressions to enhance your facial motion capture). Some also let you create animated environmental effects for your avatar too.

As I haven't really been able to try any of this software yet I'm going to leave this up to VTuber, Kappako Channel's video, Which is the best FREE face tracker for Live2d Vtubers? where she compares the four most commonly used applications including:

Still from Kappako's video, Which is the best FREE face tracker for Live2d Vtubers?
Still from Kappako's video, Which is the best FREE face tracker for Live2d Vtubers?

  • VTube Studio - Probably the most used application. Supports Webcam and iPhone motion tracking. Includes hand tracking. Available via Steam for PC users.
  • PrprLive - Also on Steam. Facial motion capture via webcam.
  • VUP - Steam link. Webcam face capture. Does support other motion capture devices for body capture.
  • Animaze - Works with webcam or iphone. Animaze also have their own avatar editing software that supports common 2D and 3D formats. It's purpose is to help optimize your model to take advantage of the special features of  Animaze.

How Do I Live Stream or Just Record My Performance?

If you're goal is to keep things free or low cost then OBS Studio is probably the most widely used application for live streaming. It's completely free and has so many options. 

It's beyond the scope of this article to go into detail on how to set OBS up with your avatar but the general idea is (if you want to overlay your character on something else like gameplay) to record your avatar with a transparent background or single color (usually bright green) background, and set up a Scene in OBS that contains all the sources you want on screen including your Avatar puppeteering software. If you're using a color background you'll need to add a chromakey filter to make this transparent.

Then you either set up OBS to Live stream, or record video to a file for editing later.

Coming Up

I really wanted to have my 2D TET Avatar all set up for this article so I could demonstrate a 2D VTuber Avatar that wasn't Anime based. 

I also wanted to see how useful having a character template from Cartoon Avatar might be. For example can you use all those mouth sprites in a VTuber Avatar (I think you can but sprite switching is not typically how lip sync is done with VTuber Avatars).

My plan is to write a follow up article on my experience of learning the rigging process and then demonstrating my TET Avatar with a video. For now I wanted to make sure you had everything you need to start your own journey into creating a VTuber avatar.

While you may not want to be a VTuber who live streams, a Virtual Avatar can be very useful if you want to create video tutorials that perhaps appeal to a younger audience, or maybe you'd like to do virtual presentations as an animated presenter. There are so many possibilites.
 

Popular posts from this blog

Can You Learn Reallusion's Cartoon Animator 5 for Free Using Their 137 Official YouTube Video Tutorials Sorted Into a Logical Learning Order?

Or you could just buy The Lazy Animator Beginner's Guide to Cartoon Animator . While Reallusion's Cartoon Animator is one of the easiest 2D animation studios to get up and running with quickly, learning it from all of the official, free, video tutorials can be more overwhelming than helpful. With more than 137 videos totaling more than 28 and a half hours of tutorials, spread across three generations of the software (Cartoon Animator 3 through 5) it's hard to know if what you're learning is a current or legacy feature that you either need to know or can be skipped. Many of the official tutorials only teach specific features of the software and don't relate at all to previous or later tutorials. As a result there are many features either not mentioned or are hard to find. To make your learning easier, on this page, I've collected together all of the essential, official, free video tutorials and sorted them into a learning order that makes sense. Simply start at

Artbreeder - Using AI created Character and Background Content in your Animations

A selection of User/AI generated images from Artbreeder. If you're looking for an endless supply of 2D character and background images for your animations then Artbreeder , an online Artificial Intelligence (AI) that generates image mash-ups you can tweak as much as you like, could be the ultimate content library. What is Artbreeder? Artbreeder is free to use though there are various paid plans, that give you additional features, such as higher resolution download images or more settings to play with. All images created on the site are Public Domain (CC0 License) and can be used in commercial projects. Using Artbreeder's online app you can generate head shot portraits, full body characters, landscapes, and other scenes simply by choosing two or more existing images to mash together then, using a series of sliders, to select which traits from each image you wish to lean toward in the final image. Photo Comparison - Top is my original uploaded photo. Bottom is Artbreeder's ap

Jarrad Wright, The Big Lez Show - Who Would've thought Animating with MS Paint Could Take You So Far?

A friend of mine recommended I should check out The Big Lez Show after I mentioned to him I make animations for living. He said the show's creator, Australian animator, Jarrad Wright , just makes episodes from his home using MS Paint. Somewhat shamefully I hadn't heard of The Big Lez Show, but the fact that it was being made with MS Paint absolutely hooked me into checking out. If you've never heard or seen the show then you, like I was, are probably thinking how good could it be? MS Paint has kind of a cult following of hardcore animators but no one would use it as their primary animation tool on a series, right? WARNING - before going any further, you need to know The Big Lez Show and its humor contains some pretty strong language. By strong I mean it's peppered very liberally with the 'F' and 'C' words and is very every day Aussie, blue collar speak. Unapologetically, all of that, is part of why it's so good. There's a good chance you've

AE Juice - Animation Presets, Motion Graphics, Templates, Transitions for After Effects, Premiere Pro, and Other Video Applications

Level up you video edits and animations with AE Juice's motion graphics and templates. Some days you just don't have the time to create flashy motion graphics for your latest video or animation. For some of us it's more a question of our own artistic abilities being a little less than the awesome we'd like them to be. Whatever reason a resource like AE Juice's animation presets, motion graphics, templates, and transitions packs for After Effects , Premiere Pro , and other video applications can really make your work stand out very quickly. AE Juice gives you access to an instant library of free, premade content elements and sound effects, which you can add to with additional purchases of various themed packs from their store. There are three ways to manage their content, all of which can be used in commercial projects . The AE Juice Standalone Package Manager makes it easy to browse previews of all your pack contents and to download and find just the elements yo

Eric W. Schwartz: Cartoonist, Animator and Amiga Die Hard

July 1992 Edition, CU Amiga Featuring Amy the Squirrel. American Cartoonist, Eric W. Schwartz , (whose unofficial Amiga Icon, Amy the Squirrel, is pictured on the July 92 edition of CU Amiga cover on the right) is my only real animation hero. Sure there are the big names like Disney , Chuck Jones , Tex Avery and even Preston Blair whose influences can all be seen in my own cartoons but Eric did what none of the others could. He showed that really great 2D computer animation was within my reach with little more than an Amiga Computer , a copy of Deluxe Paint and Moviesetter . This was at a time when computer based animation was in its infancy (outside of computer game animation) and Flash was something that lights did. There were many great Amiga artists but Eric was really the only one consistently making very funny, traditional style animations. His humor and drawing style is heavily influenced by classic Warner Brothers and Disney cartoons but he managed to build on this,

Make Disney/Pixar Style Characters with Reallusion's Character Creator and Toon Figure Bases

The Extraordinary Tourist Classic Coat outfit created using Reallusion's Toon Designer for CC3. I've talked before how I've wanted to get into 3D Disney/Pixar style character animation since I first saw the animated cutscenes for the very first Tomb Raider game back in 1996. It's why I initially bought Reallusion's iClone 3D studio app as soon as I could afford a computer that would run it. But then Reallusion released their 3D Character Creator (CC) for iClone and I wanted to create my characters with that (and I did try with Bat Storm ). But the focus of CC was realism, even with ToKoMotion's stylised body morphs . Now with Reallusion's Cartoon Designer bundle for CC3 which features two packs, Toon Figures , and Toon Hair , designing Disney/Pixar style 3D characters just got a whole lot quicker. The two packs are the bare essentials for creating Toon style characters. Five body morphs (2 male, 2 female, and one adolescent body morph that w

Reallusion Releases Cartoon Animator 5 - One Version, More Features, Lower Price!

If you're serious about producing 2D animation as quickly as possible, while still achieving professional results, Reallusion's Cartoon Animator 5 makes the most compelling case yet as your animation studio/tool of choice. Cartoon Animator's point of difference has always been its ease of use and accelerated workflow. Creating fast, 2D animation using puppet, bone rigged based characters and props, on a stage with 3D depth for easy scene parallax effects. As it has developed Reallusion has incorporated more advanced features like motion capture for both face and body as well as being able to export scenes to post production tools like After Effects with the addition of plugins. After moving away from Flash based vector image support for a few years, Reallusion is back with full .SVG (scalable vector graphics) support for resolution independent graphics. They've also added Spring Dynamic physics and Full Form Deformation tools, both of which make it ridiculously easy t