Skip to main content

How to Become a Virtual YouTuber (VTuber) - 2D and 3D Software for Creating and Controlling Your Avatar

TET VRoid Avatar in VSeeFace.
Young TET VRoid Avatar in VSeeFace.
In a previous YouTube video I demonstrated how you could use a webcam, Cartoon Animator and its MotionLive 2D Facial Capture system to create a virtual avatar that you could perform live during a live stream. Unfortunately the motion capture performance of this system is very average to say the least, and is even more expensive if you want to add hand motion capture via a Leap Motion Controller.

This lead me down a rabbit hole of seeing what other solutions are out there, where I bumped into the world of the VTuber or Virtual YouTuber (creators who puppet and voice an on camera avatar instead of appearing on camera themselves). Specifically VTuber software, which has been a real eye opener for how expensive and far behind Reallusion's upper body motion capture system actually is.

Free 3D VTuber Avatar Creator, VRoid Studio.
The VTuber world is heavily entrenched in the world of Anime so you'll see a lot of 2D and 3D avatars in this style. If fact, if an Anime avatar is your thing, you can be up and running with a customized 3D Avatar within an hour using VRoid studio, a Free 3D Anime character maker specifically designed for this purpose, and VSeeFace, an also free, VTuber avatar puppeteering program.

The Gold Standard in 3D VTuber Software (IMO)


One of the best examples of 3D VTuber software I've come across is Webcam Motion Capture (yes that is literally the application's name). Free to try, and a small monthly fee to remove the aggressive text watermark, it seems like the gold standard of upper body motion capture.

Using a standard webcam and some AI wizardry this software can capture facial expressions including lip sync, head and upper body movement, arms, hands, and individual finger movement, very smoothly in real time.

Webcam Motion Capture in action using my custom VRoid Avatar.
Webcam Motion Capture in action using my custom 3D VRoid, Anime Avatar. This application has everything you would want in upper body motion capture all from a single webcam. While it's completely free to try the aggressive text watermark makes the free version totally unusable for anything but testing. That said it's only US$1.99 per month or you can buy a lifetime license.

The system is so good I feel Reallusion should drop everything they're doing with upper body motion capture and use this instead - seriously. It has full hand and finger motion capture with no extra hardware to buy.

If that wasn't enough it has so many additional settings you can adjust to not just fine tune the motion capture but to also enhance your avatar's performance, like automatic eye blinking, hot key expressions, or idle motions for when you're not doing any movement yourself. As well it supports the .VRM format which is the standard avatar file format most of these systems use (so you can load your VRoid characters into it).

VSeeFace and Kalidoface 3D (Free Alternatives)

If you're looking for a free alternative to Webcam Motion Capture try VSeeFace a very popular, free app used by VTubers. Watch this very good video by Vtuber, Fofamit, to get you started: 

VSeeFace does support the addition of a Leap Motion Controller to capture hand data but it turns out you don't actually need one. Install this free hand motion capture softwareVideo here by VTuber, Lady Aska, to explain how.

If you don't want to install anything then try Kalidoface 3D VTuber app. It's completely browser based and you can use your custom VRoid Avatars with it.

Kalidoface 3D featuring my TET VRoid Avatar.
Kalidoface is free, browser based, and has full body motion capture. Here I've loaded my TET VRoid Avatar into the interface.

Most of the software I'd found so far was 3D based so I kept going to see what was out there for 2D VTubers.

The PNG 2D VTuber (The simplest way to be a 2D VTuber)

This is not the first way I discovered how to make and puppet your own 2D VTuber Avatar but it's so easy to set up using Cartoon Animator and the completely free PNG Avatar VTuber software, Veadotube Mini.

Unlike most VTuber software this one does not use a webcam or motion capture at all. Instead it works based on the sound from your microphone. In essence it's a flipbook style software that switches between four states of your PNG Avatar... 1. Eyes open, mouth closed, 2. Eyes open, mouth open, 3. Eyes closed, mouth closed, 4. Eyes closed, mouth open. This doesn't sound like much but it's enough to create very rudimentary lip syncing with random eye blinks at any time.

Creating my PNG images in Cartoon Animator for use in Veadotube mini shown in the lower right.
By setting up the maximum image size of 2048 pixels square in Cartoon Animator it's easy to create all the transparent PNG images you need to make a PNG VTuber Avatar in Veadotube Mini (shown lower right). All you need is a microphone.

In addition to that you have three different movement states you can apply depending upon what mouth state is being displayed.

For even more flexibility you can add different states of these images and movement, that you can switch with your assigned hot keys. For example I made a second state of my character talking excitedly with slightly wider eyes and hands raised up in the air.

However because this software is literally flipping images you can do almost any pose or style of character you need. Side view, talking horse... whatever you can dream up. VTuber, Syafire, has a full video on how to set Veadotube Mini up in 4 easy steps.

Also worth mentioning... If you like this style of VTuber Avatar there is a low cost, paid software called Honk that is worth checking out. It's a little more complex, supporting layered characters that allows  better lip sync capabilities.

Advanced 2D VTuber Avatar Creation

Once you start looking into creating a more advanced 2D Avatar with the ability to add facial expressions, head turning, physics and more, you may start thinking being a PNG VTuber is a great compromise (or maybe you'll just stick with a VRoid Avatar - the easiest option of all).

Live 2D Cubism Editor

Finding a free or low cost 2D avatar character rigging application is not easy. There seems to be only one name in town, Live2D Cubism. On the plus side they'll give you a 42 day full feature trial of their VTuber rigging software, which is enough time to learn the process. I'm told it's simple once you know how but, at first glance, it's overwhelming.

Live 2D Cubism VTuber Rigging Software with my TET Avatar being rigged.
The early stages of my attempt to rig my character with Live 2D Cubism. I imported the same mouth and eye sprites I used for the character in Cartoon Animator but these may not be usable as Live 2D uses deform tools to create most facial movement and expression.

Preparing your avatar for rigging is a relatively simple case of separating all the sprites into individual parts in a .PSD file with no group folders. Everything is placed in layer order from top to bottom.

While I tried to rig my TET avatar in Live 2D Cubism I wasn't able to learn it quick enough to demonstrate it here. Even with the many YouTube tutorials I was overwhelmed - and my avatar isn't nearly as complex as some of the Anime avatars many VTubers create themselves and use.

Inochi2D VTuber Rigging Editor

I was able to find a free, opensource 2D Vtuber rigging software called Inochi2D Creator. Unfortunately it's a completely new VTuber avatar format that only works with their Inochi2D Sessions puppeteering software. However it can import the same .PSD files you would use for Live2D Cubism as a starting point.

Inochi2D Creator User Interface with my TET Avatar.
As you can see Inochi2D Creator had no trouble importing my .PSD file. From what I can tell the rigging process is similar to Live2D Cubism but this is as far as I got prior to writing this post.

The other issue is Inochi2D is still in development. I wasn't able to determine if it is still an active project or not however the documentation for Creator seemed quite good, while the same can't be said for Session, which is almost non-existent. While I was able to load an example avatar into Session I wasn't able to work out how to hook up a motion capture solution to it, despite it supporting VMC motion capture that I had (and you will too if you download DSSBodyTrackor2 that is used to add hand capture to VSeeFace, mentioned earlier).

I did find an excellent YouTube Tutorial stepping you through the whole process of using both Inochi  Creator, Session, and hooking everything up to OBS by VTuber Dragon Dragonnator96 - How To Rig a VTuber Model with Inochi2D (0.7.4.1 Creator and 0.5 Session). Definitely worth searching YouTube for more as well.

2D VTuber Puppeteering Software

Once you have your VTuber avatar rigged with Live2D (not Inochi Creator because their format only works with Session) you need an application to puppeteer it with. 

Most applications focus on facial motion capture so your avatar will lip sync and emote in line with your own facial movement. Other body motions and physics based motions are usually achieved with pre-set up auto-animated loops, or triggered with hot keys (such as extreme facial expressions to enhance your facial motion capture). Some also let you create animated environmental effects for your avatar too.

As I haven't really been able to try any of this software yet I'm going to leave this up to VTuber, Kappako Channel's video, Which is the best FREE face tracker for Live2d Vtubers? where she compares the four most commonly used applications including:

Still from Kappako's video, Which is the best FREE face tracker for Live2d Vtubers?
Still from Kappako's video, Which is the best FREE face tracker for Live2d Vtubers?

  • VTube Studio - Probably the most used application. Supports Webcam and iPhone motion tracking. Includes hand tracking. Available via Steam for PC users.
  • PrprLive - Also on Steam. Facial motion capture via webcam.
  • VUP - Steam link. Webcam face capture. Does support other motion capture devices for body capture.
  • Animaze - Works with webcam or iphone. Animaze also have their own avatar editing software that supports common 2D and 3D formats. It's purpose is to help optimize your model to take advantage of the special features of  Animaze.

How Do I Live Stream or Just Record My Performance?

If you're goal is to keep things free or low cost then OBS Studio is probably the most widely used application for live streaming. It's completely free and has so many options. 

It's beyond the scope of this article to go into detail on how to set OBS up with your avatar but the general idea is (if you want to overlay your character on something else like gameplay) to record your avatar with a transparent background or single color (usually bright green) background, and set up a Scene in OBS that contains all the sources you want on screen including your Avatar puppeteering software. If you're using a color background you'll need to add a chromakey filter to make this transparent.

Then you either set up OBS to Live stream, or record video to a file for editing later.

Coming Up

I really wanted to have my 2D TET Avatar all set up for this article so I could demonstrate a 2D VTuber Avatar that wasn't Anime based. 

I also wanted to see how useful having a character template from Cartoon Avatar might be. For example can you use all those mouth sprites in a VTuber Avatar (I think you can but sprite switching is not typically how lip sync is done with VTuber Avatars).

My plan is to write a follow up article on my experience of learning the rigging process and then demonstrating my TET Avatar with a video. For now I wanted to make sure you had everything you need to start your own journey into creating a VTuber avatar.

While you may not want to be a VTuber who live streams, a Virtual Avatar can be very useful if you want to create video tutorials that perhaps appeal to a younger audience, or maybe you'd like to do virtual presentations as an animated presenter. There are so many possibilites.
 

Comments

Popular posts from this blog

Inochi2D - Free Open Source 2D VTuber Avatar Rigging and Puppeteering Software (Part 1)

Inochi2D Creator - Free Open Source VTuber Software. If you've been looking for a way to live perform as a 2D cartoon avatar on camera, whether it be for a live stream or for pre-recorded content like educational videos, then VTuber software is a low cost (or even no cost) option worth looking into. In my previous post, How to Become a VTuber - 2D and 3D Software for Creating and Controlling Your Avatar , I took a brief look at the relatively new but completely free and open source Inochi2D  which I thought showed great potential for my own needs of creating a live performance character rig for my own TET Avatar that I use for all my promotional materials. While it is possible to live perform my character using Cartoon Animator itself, Reallusion's MotionLive2D capture system isn't great - with lip sync in particular. More importantly though, I can't exactly teach people how to use Cartoon Animator if I'm using Cartoon Animator to control my Avatar. What is Inochi2D

Wonder Unit Storyboarder - Free Storyboarding Software for People Who Can (or Can't) Draw

Wonder Unit Storyboarder.  As an independent and solo animator I'm always tempted to try and skip storyboarding my animated shorts because they're usually only single scene sketch comedy type jokes. As a result I have many unfinished projects that kind of petered out due to having no clear finishing line. Storyboarding your productions, no matter how small, gives you a step by step guide of every shot that needs to be completed (no planning shots as you animate). It also allows you to create an animatic that gives you a rough preview of the finished production. In short, you shouldn't skip storyboards as they, generally, increase the chance of the project being completed. Disclaimer - I'm Not a Fan of Storyboarder Upfront, Wonder Unit's Storyboarder  is not my preferred storyboarding software. However it's completely free, has a number of very compelling featu

Glif (Alpha) - Make Your Own Tiny AI Powered Niche Image Generator Apps

I  first heard about Glif through a YouTube video that mentioned you could get access to Flux Pro (the latest 'game changer AI' generative image model) through the site for free. While I had a vague notion from the video of what Glif was, I wasn't expecting it to be so easy to get started with, and so good with my very first results. Glif is an easy to use, low-code platform for creating tiny AI-powered generators called Glifs. While that may not sound inspiring, what Glifs allow you to do is create a tiny app that niches down to a specific type of AI generation that the user modifies with their own inputs. The best way to really understand is with an example.  My First Glif I've recently been using VivaGo's AI platform  (free and unlimited at the time of writing) to consistently generate full body characters in a front facing T-Pose that I can rig as front facing characters in Cartoon Animator. Unfortunately it can be a bit hit or miss maintaining the T-Pose part

Krita AI Diffusion - Generative Image AI For Krita is Seriously Useful, Powerful and Free (If You Can Install it Locally)

Generative AI sequence of a woman in a business suit. From sketch to refined image using Krita AI Diffusion - by TET G enerative image AI, where you describe an image with a text prompt to an Artificial Intelligence model and it produces a new image based on your prompt, is gaining a strong hold as a tool for many artists. Krita AI Diffusion brings generative AI image tools right into your favourite free and opensource, graphics editor, Krita. Not only that, if you have a computer with decent specs (and at least 10GB of hard drive space), Krita AI Diffusion is completely free. What If I Don't Have a Powerful Computer? If you're in my situation, with a computer that was around before anyone in the mainstream had even heard of generative AI, you can still access Krita AI Diffusion for free, using a cloud based AI server, Interstice  and 300 tokens, to get you started. Once your initial tokens run out, purchase 5000 more for 10€ (approx US$11.00). Tokens never expire. I would

The Ultimate Independent Animator's App and Resource List 2019-2023 - Animation and Video Life

Image created with Cartoon Animator 4. Being an independent animator is not like a studio animation job. There's so much more to do that is indirectly related to the actual task of animating. Over the years I've sought out many apps, tools, and services that can help me achieve that one single task, expressing myself through animation. Below is my Ultimate Independent Animator's Resource List for 2019-2022. It started out as a list of free or low cost apps that could help you in every stage of producing either 2D or 3D animation, and then just kind of grew from there. You may not have been looking for a Time Management App as much as you needed something to get you started in 3D animation but when those commissioned projects start coming in you'll have a head start on maximizing your time. All the apps and services on this list had to meet two main criteria: They had to be useful and relevant to an Indy Animator/artist. The base app/service had to be US$

Dollars Mocap: Full Body Webcam Motion Capture (Including Hands and Fingers) For iClone and Cartoon Animator

Even though I should be further away from the camera Dollars Mocap MONO still does a good job of  tracking my arms, hands and fingers. Ever since I wrote my series on becoming a VTuber , discovering it was possible to do full body motion capture, including hands and fingers, with just software and a webcam, I've been on the look out for any motion capture software that can bring that functionality to Cartoon Animator. Dollars Mocap is a low cost motion capture application with a free trial that I learned about through the YouTube Channel Digital Puppets  and their test video . It can record full body, upper body, arms and hands, and facial mocap from a live video source or pre-recorded video. Investigating further, I discovered not only does Dollars Mocap have a free iClone7, iClone8 character profile file download (look for it at the bottom of the main program download page), so you can use the saved motions with iClone8, they've also got a demo video for how to convert your

Review: Headshot Plugin for Reallusion's Character Creator 3

Headshot for CC3. Quite possibly the best 3D Avatar I've made of myself in any 3D application. Creating a realistic 3D human avatar is a whole lot easier with Reallusion's new Headshot Plugin for Character Creator 3. The plugin is an AI powered extension that can generate 3D digital humans from one photo. Which sounds like an amazing proposition but, in practice, if you're trying to achieve a specific likeness to an actual person, Headshot will give you an excellent base to work from. Headshot has two modes, Auto and Pro. Auto Mode Auto is well worth a try if you have an ideal photo of a front facing person that is properly lit and posed to Headshot's optimum requirements. It's also the only mode that will take a crack at generating a hair model. I grabbed an image of Harrison Ford, dragged it into Headshot without changing any of the default settings (other than specifying 'male' and selecting an 'old male' setting) and this is what I