How to Become a Virtual YouTuber (VTuber) - 2D and 3D Software for Creating and Controlling Your Avatar

Young TET VRoid Avatar in VSeeFace.

In a previous YouTube video I demonstrated how you could use a webcam, Cartoon Animator and its MotionLive 2D Facial Capture system to create a virtual avatar that you could perform live during a live stream. Unfortunately the motion capture performance of this system is very average to say the least, and is even more expensive if you want to add hand motion capture via a Leap Motion Controller.

This lead me down a rabbit hole of seeing what other solutions are out there, where I bumped into the world of the VTuber or Virtual YouTuber (creators who puppet and voice an on camera avatar instead of appearing on camera themselves). Specifically VTuber software, which has been a real eye opener for how expensive and far behind Reallusion's upper body motion capture system actually is.

The VTuber world is heavily entrenched in the world of Anime so you'll see a lot of 2D and 3D avatars in this style. If fact, if an Anime avatar is your thing, you can be up and running with a customized 3D Avatar within an hour using VRoid studio, a Free 3D Anime character maker specifically designed for this purpose, and VSeeFace, an also free, VTuber avatar puppeteering program.

The Gold Standard in 3D VTuber Software (IMO)

One of the best examples of 3D VTuber software I've come across is Webcam Motion Capture (yes that is literally the application's name). Free to try, and a small monthly fee to remove the aggressive text watermark, it seems like the gold standard of upper body motion capture.

Using a standard webcam and some AI wizardry this software can capture facial expressions including lip sync, head and upper body movement, arms, hands, and individual finger movement, very smoothly in real time.

Webcam Motion Capture in action using my custom 3D VRoid, Anime Avatar. This application has everything you would want in upper body motion capture all from a single webcam. While it's completely free to try the aggressive text watermark makes the free version totally unusable for anything but testing. That said it's only US$1.99 per month or you can buy a lifetime license.

The system is so good I feel Reallusion should drop everything they're doing with upper body motion capture and use this instead - seriously. It has full hand and finger motion capture with no extra hardware to buy.

If that wasn't enough it has so many additional settings you can adjust to not just fine tune the motion capture but to also enhance your avatar's performance, like automatic eye blinking, hot key expressions, or idle motions for when you're not doing any movement yourself. As well it supports the .VRM format which is the standard avatar file format most of these systems use (so you can load your VRoid characters into it).

VSeeFace and Kalidoface 3D (Free Alternatives)

If you're looking for a free alternative to Webcam Motion Capture try VSeeFace a very popular, free app used by VTubers. Watch this very good video by Vtuber, Fofamit, to get you started: 

VSeeFace - The Free Best Application for Vtubers - Walkthrough, Guide, Tips, & getting started

VSeeFace does support the addition of a Leap Motion Controller to capture hand data but it turns out you don't actually need one. Install this free hand motion capture software. Video here by VTuber, Lady Aska, to explain how.

If you don't want to install anything then try Kalidoface 3D VTuber app. It's completely browser based and you can use your custom VRoid Avatars with it.

Kalidoface is free, browser based, and has full body motion capture. Here I've loaded my TET VRoid Avatar into the interface.

Most of the software I'd found so far was 3D based so I kept going to see what was out there for 2D VTubers.

The PNG 2D VTuber (The simplest way to be a 2D VTuber)

This is not the first way I discovered how to make and puppet your own 2D VTuber Avatar but it's so easy to set up using Cartoon Animator and the completely free PNG Avatar VTuber software, Veadotube Mini.

Unlike most VTuber software this one does not use a webcam or motion capture at all. Instead it works based on the sound from your microphone. In essence it's a flipbook style software that switches between four states of your PNG Avatar... 1. Eyes open, mouth closed, 2. Eyes open, mouth open, 3. Eyes closed, mouth closed, 4. Eyes closed, mouth open. This doesn't sound like much but it's enough to create very rudimentary lip syncing with random eye blinks at any time.

By setting up the maximum image size of 2048 pixels square in Cartoon Animator it's easy to create all the transparent PNG images you need to make a PNG VTuber Avatar in Veadotube Mini (shown lower right). All you need is a microphone.

In addition to that you have three different movement states you can apply depending upon what mouth state is being displayed.

For even more flexibility you can add different states of these images and movement, that you can switch with your assigned hot keys. For example I made a second state of my character talking excitedly with slightly wider eyes and hands raised up in the air.

However because this software is literally flipping images you can do almost any pose or style of character you need. Side view, talking horse... whatever you can dream up. VTuber, Syafire, has a full video on how to set Veadotube Mini up in 4 easy steps.

Also worth mentioning... If you like this style of VTuber Avatar there is a low cost, paid software called Honk that is worth checking out. It's a little more complex, supporting layered characters that allows  better lip sync capabilities.

Advanced 2D VTuber Avatar Creation

Once you start looking into creating a more advanced 2D Avatar with the ability to add facial expressions, head turning, physics and more, you may start thinking being a PNG VTuber is a great compromise (or maybe you'll just stick with a VRoid Avatar - the easiest option of all).

Live 2D Cubism Editor

Finding a free or low cost 2D avatar character rigging application is not easy. There seems to be only one name in town, Live2D Cubism. On the plus side they'll give you a 42 day full feature trial of their VTuber rigging software, which is enough time to learn the process. I'm told it's simple once you know how but, at first glance, it's overwhelming.

The early stages of my attempt to rig my character with Live 2D Cubism. I imported the same mouth and eye sprites I used for the character in Cartoon Animator but these may not be usable as Live 2D uses deform tools to create most facial movement and expression.

Preparing your avatar for rigging is a relatively simple case of separating all the sprites into individual parts in a .PSD file with no group folders. Everything is placed in layer order from top to bottom.

While I tried to rig my TET avatar in Live 2D Cubism I wasn't able to learn it quick enough to demonstrate it here. Even with the many YouTube tutorials I was overwhelmed - and my avatar isn't nearly as complex as some of the Anime avatars many VTubers create themselves and use.

Inochi2D VTuber Rigging Editor

I was able to find a free, opensource 2D Vtuber rigging software called Inochi2D Creator. Unfortunately it's a completely new VTuber avatar format that only works with their Inochi2D Sessions puppeteering software. However it can import the same .PSD files you would use for Live2D Cubism as a starting point.

As you can see Inochi2D Creator had no trouble importing my .PSD file. From what I can tell the rigging process is similar to Live2D Cubism but this is as far as I got prior to writing this post.

The other issue is Inochi2D is still in development. I wasn't able to determine if it is still an active project or not however the documentation for Creator seemed quite good, while the same can't be said for Session, which is almost non-existent. While I was able to load an example avatar into Session I wasn't able to work out how to hook up a motion capture solution to it, despite it supporting VMC motion capture that I had (and you will too if you download DSSBodyTrackor2 that is used to add hand capture to VSeeFace, mentioned earlier).

I did find an excellent YouTube Tutorial stepping you through the whole process of using both Inochi  Creator, Session, and hooking everything up to OBS by VTuber Dragon Dragonnator96 - How To Rig a VTuber Model with Inochi2D (0.7.4.1 Creator and 0.5 Session). Definitely worth searching YouTube for more as well.

2D VTuber Puppeteering Software

Once you have your VTuber avatar rigged with Live2D (not Inochi Creator because their format only works with Session) you need an application to puppeteer it with. 

Most applications focus on facial motion capture so your avatar will lip sync and emote in line with your own facial movement. Other body motions and physics based motions are usually achieved with pre-set up auto-animated loops, or triggered with hot keys (such as extreme facial expressions to enhance your facial motion capture). Some also let you create animated environmental effects for your avatar too.

As I haven't really been able to try any of this software yet I'm going to leave this up to VTuber, Kappako Channel's video, Which is the best FREE face tracker for Live2d Vtubers? where she compares the four most commonly used applications including:

Still from Kappako's video, Which is the best FREE face tracker for Live2d Vtubers?

• VTube Studio - Probably the most used application. Supports Webcam and iPhone motion tracking. Includes hand tracking. Available via Steam for PC users.
• PrprLive - Also on Steam. Facial motion capture via webcam.
• VUP - Steam link. Webcam face capture. Does support other motion capture devices for body capture.
• Animaze - Works with webcam or iphone. Animaze also have their own avatar editing software that supports common 2D and 3D formats. It's purpose is to help optimize your model to take advantage of the special features of  Animaze.

How Do I Live Stream or Just Record My Performance?

If you're goal is to keep things free or low cost then OBS Studio is probably the most widely used application for live streaming. It's completely free and has so many options. 

It's beyond the scope of this article to go into detail on how to set OBS up with your avatar but the general idea is (if you want to overlay your character on something else like gameplay) to record your avatar with a transparent background or single color (usually bright green) background, and set up a Scene in OBS that contains all the sources you want on screen including your Avatar puppeteering software. If you're using a color background you'll need to add a chromakey filter to make this transparent.

Then you either set up OBS to Live stream, or record video to a file for editing later.

Coming Up

I really wanted to have my 2D TET Avatar all set up for this article so I could demonstrate a 2D VTuber Avatar that wasn't Anime based. 

I also wanted to see how useful having a character template from Cartoon Avatar might be. For example can you use all those mouth sprites in a VTuber Avatar (I think you can but sprite switching is not typically how lip sync is done with VTuber Avatars).

My plan is to write a follow up article on my experience of learning the rigging process and then demonstrating my TET Avatar with a video. For now I wanted to make sure you had everything you need to start your own journey into creating a VTuber avatar.

While you may not want to be a VTuber who live streams, a Virtual Avatar can be very useful if you want to create video tutorials that perhaps appeal to a younger audience, or maybe you'd like to do virtual presentations as an animated presenter. There are so many possibilites.