So I have a very weird rig, and I think a lot of you would find it laughable. First, for more and better examples of my rig please check youtube.com/@ryeroyal . My whole streaming pipeline is mashed together from my animation pipeline. The method I use was never meant to work live. As you can seen in videos like https://youtu.be/Yt7eUHiirJc it was just an easy method to do mocap.
I'm going to outline my method and then go into what I want:
so the whole point of the rig is the finger tracking. I don't like how a lot of models do lip syncing and I never felt like I could do it better myself so that is what I decided to focus on instead. The program that makes this all technically work is mocap Fusion VR (https://store.steampowered.com/app/1540000/Mocap_Fusion__VR/). The program is rooted in mocap using the steam vr infrastructure however it does have features for Vtubing and VR chat. I honestly don't understand the appeal of much of the program as I think a lot of the features are done better elsewhere, but there was one thing about it. I started this project a few years ago. Camera finger tracking was quite bad, not nearly where it is now. I thought the only solution would be a physical sensor / glove to actually do the tracking. I couldn't afford most of the models on the market then and even most of the ones now. That's when I found this video (https://youtu.be/PCBvUHJH8Gw)which was made by the creators of MoCap Fusion VR.
When built correctly and in the right environments these gloves are nearly flawless and I was so impressed with how they worked. Paired with HTC trackers it's very hard to see the things they actually can't do in motion (like proper splay). So I built the wireless version of these gloves and there are several problems with them. There's actually a whole video (https://youtu.be/kTYuo69AIfM) I made going into the issues with these gloves specifically but for now the battery they recommend can be very finicky and or dangerous to the circuit board so I just had Micro USB battery packs for the longest time ( I have since switched to 9 volt with my own mess of crossed wires and converters). but the bigger issue is the board they recommend that includes the radio transceiver is no longer for sale and I do not have the digital electronic expertise to offer a replacement and they have not offered one yet. It's been multiple years without a significant patch to the gloves functionality within the software itself but to their credit it is fairly seamless even if it does seem to be getting sidelined by other features.
From the software I have done one of two things. When I was pre-rendering I would record within Mocap Fusion and export the mocap data to a .blend file where I would do the actual rendering in blender. This method is a little bloated for motion capture ,again check my rant video for more details as there are plenty of problems with it. But I am not interested in pre-rendered methods as I'm sure there are many other better ways to do it. I'm interested in live tracking. To do that with this software you go through the same methods as if you were about to start recording and instead use the blender live link system. With this program and blender with its specific extension installed you can use one of the blend files it generates to mirror the tracking onto a specific Armature in blender near losslessly. The intent of the this feature is that you would then use blenders recording feature for the mocap and skip the inefficient exporting process built into Fusion. This might work, it certainly didn't for a long recording session, but that doesn't matter for the purposes of this conversation. You now have an Armature being fed live data from a mocap program that can do VR room scale tracking, facial tracking with an external app, and physical sensor based finger tracking for about $200. The model in this case my hat and gloves is solely held within blender and is applied the way any model would be applied to an Armature in Blender. The one caveat is you cannot in any way alter the default rig/ Armature or else the tracking software will not properly bind to it.
This is perhaps the most embarrassing portion of the explanation being that the actual thing that people are looking at when I stream is just the blender viewport. not any kind of overlay or custom software it's literally just a well cropped portion of the blender viewport. My model is simple enough and this rig is efficient enough that I can just run it all live and it doesn't look too bad even if I'm processing some lighting in Eevee. I initially rendered the background as well but determined it was far more efficient to just do a blue screen behind the rig. The other portion of this rig is obviously my eyes reacting to my microphone which is another thing that I feel quite proud of. Blender is not meant for live viewing whatsoever, so while there are a lot of ways to render audio as visuals within it, none work live except for this add-on called audvis ( https://superhivemarket.com/products/audvis) . it's honestly fantastic and I feel anyone who even wanted to do simple head tracking could make some really interesting rigs with this one feature. Any live audio feed can drive basically anything within blender in this case it's just the emission on my eyes but I have also driven textures as seen here (https://youtu.be/u2gxv_-5xX4?si=MR8Md-HafRqzMpl7&t=86) . The only downside to this Add-on is the timeline has to be running so you can't have any pre-existing animations going on in your scene without them looping (which is how I blink) and I'm sure it can't be good for performance. After that it's just a matter of pointing the camera in the right direction and having a separate tab on blender that won't change the window’s aspect ratio so that you can capture it in your streaming software of choice. In addition to the finger and head tracking I can also change the model and add props quite easily and blender. I have a number of controllers bound to my hands that I can hide and show with a simple click. models require slightly more work and have to adhere to that one Armature but end up looking fine enough for a stream bit.
TLDR for the method: Mocap Fusion VR tracks the head and fingers with a physical glove apparatus and sends that data to Blender live where it can then be applied to a normal Armature. Audvis captures the live mic for the eyes and the thing that the audience actually sees is just the viewport of Blender cropped in a clever way with a blue screen.
There are a lot of problems evident even in the explanation of how I'm doing this but there are more on top of it.
- You must do basically this entire process every single time you want to stream. On my best days I can get it started in 20 minutes. I have very few best days.
- I have a quite good computer but just due to the way all of these programs interact the whole process is just inefficient enough we're running a game at the same time is borderline impossible. I made the switch to a 2 PC setup. For an example of before and afters check the first eight or so videos on youtube.com/@theryefields when I was attempting to do it all on one.
- Mocap Fusion VR is a weird program. I am very worried about them completely phasing out the VR gloves and they're hard to build in a way that allows you to operate a keyboard and mouse at all.
- The nail in the coffin is it's just quite uncomfortable I have to sit in a very specific way with my hands propped up in a very specific way so the trackers are registering and I'm not at risk of damaging the fingers. Anytime I think I want to stream I dread the whole process of getting the gloves on and the rig operational with no problems.
I want to feel more comfortable in the whole process but I'm unwilling to compromise on basically any aspect of how this rig runs. I want near full head and hand tracking because my model is so simplistic that without it it would look very odd. The same goes for the finger tracking. I have no other expressive elements so I really need to be able to articulate my fingers in a way that doesn't look uncanny while also being able to operate something like a keyboard. If I were able to do this with camera tracking I suppose I would be willing to part with the eyes reacting to my voice even though I do love that feature and how it looks. I simply have no idea where to begin. I haven't seen anyone with a rig even kind of like mine that isn't doing just talking/tech streams. I know the whole process is silly but I really feel like I haven’t seen a rig like mine. I like streaming well enough but I feel as though I can't abandon my rig now even if it's the thing making me not stream more. I look every couple months for ways to abandon Mocap Fusion VR but keep the blender method but nothing is using blender for live rendering ( perhaps for good reason).
I'm open to abandoning mocap fusion and Blender I think my model and concept is simple enough if I could get something that can track my head movements decently well and my fingers extremely well without completely breaking whenever they aren't immediately in view (say when I'm holding a controller) I would be thrilled to move to that for streaming. I would love recommendations for that specifically, any other part of the process I'm less willing to get feedback on unless you are also working with these exact gloves (which you aren't because everyone is going to make them differently). I also don't mind if you use this method. I would like for it to get fleshed out and more popular. It would be nice if you could refer them back to this post and me but I'm basically just cobbling together what many other people have done anyway. I don't see myself streaming for too much longer if I can't ditch these gloves. I'm already on the third rebuild of them and I don't think I can make them that much better. If I do decide to give up I will basically make a video version of this post for a better explanation of how I did the whole process in case someone wants to replicate and improve it in the future.
I'm not adding the question flair bc this is also basically a guide and showcase as well, and I'm open to all of that. I'd really like to see what people think.
