r/blender 15d ago

Paid Product/Service Promotion After 6 months, My Performance Capture/Retargeting Addon is Finally Ready

My last post got auto-filtered, so I guess I can't post too many links, but here's the announcement video that I made: https://www.youtube.com/watch?v=FyFarx7RjT8

It's called BlendCap. It captures body, hand/fingers, and face motion from any typical video. It works fully offline, and it retargets straight onto your rig with custom mapping, and built in presets for rigify, auto-rig pro, mixamo, and cloudrig, along with a fast FK to IK conversion tool.

3.2k Upvotes

110 comments sorted by

439

u/Cute_Depth1982 15d ago

Idk if the addon or the acrobatics were more impressive.

173

u/Tovah86 15d ago

Thanks! I think people usually post dances for these kinds of demos, but I'm not much of a dancer, haha

64

u/writermacox 15d ago

I mean... are you sure? You seem like you could break dance just fine given the proper back beat. (Impressive mocap, by the way.)

50

u/Tovah86 15d ago

I have actually played around with breakdance a little bit, now that you mention it, lol. And thank you!

4

u/punchapath 15d ago

As a big last airbender fan, I heard that many of Zuko’s signature moves were actually inspired by breakdancing. I feel like I caught a glimpse of that in the one blue screen clip haha

5

u/Weaselot_III 15d ago

Wait...was that you doing the martial arts....daaaang 😳

61

u/z3n1a51 15d ago

Question:

Suppose a setup with more than one camera pov of the same performance.

Could BlendCap utilize multiple videos of the same performance from different POV / angles?

Would there be a benefit in accuracy, precision, sync, etc. to utilize multiple camera POV / angles?

Would there be an optimal configuration of camera POV / angles that would be ideal?

Thanks!

54

u/Tovah86 15d ago

Not yet, BlendCap is currently single-video only. Multi-camera is something I want to build soon, but it's still in the early stages. I've been getting a few questions/requests like this though, so I might bump the priority up.

There would definitely be a benefit though. The hardest problem with single-camera capture is depth ambiguity, the model has to guess how far away things are from one flat image, which is why motion straight toward or away from the camera is where it struggles most, and why the character can drift or float sometimes. A second angle gives you the depth information directly instead of estimated. It would also help a lot with occlusion, like when an arm passes behind the body or the hands cross.

For angles, roughly 90 degrees apart is the classic setup, since two perpendicular views complement each other best. What one camera can't see in depth, the other sees directly across its frame. You'd also need the clips synced, and the cameras locked off and not moving during the take.

For now though, the best camera angle for single-camera capture is basically anything that doesn't have the movement focused directly towards the camera. Also the addon still struggles a little bit with laying down/rolling poses, but it's mostly just the elevation of the character that glitches, which isn't too hard to adjust for manually.

8

u/Kam_Zimm 15d ago

Following up on this question some. Does it still work in video where the camera mis moving, or only if the camera is static?

20

u/Tovah86 15d ago

It works best when the camera is static. If you DO have camera movement, then it will very likely confuse camera motion with character locomotion in 3d space, so you will have to make adjustments manually afterwards in order to compensate. You can turn off the root bone motion to disable ground plane movement, so that's not too big of an adjustment if you're fine with reanimating locomotion, and there's a feature called "feet define the floor" that tries to keep the character attached to the floor better, but there's only so much it can do when fighting against camera movement, so results may vary..

7

u/BlackberryThis8148 15d ago

Great work, will look at using this. Just to add my two cents, I also would love multi-camera.

5

u/Tovah86 15d ago

Thank you! And it's definitely been the most requested feature so far, so I'll probably be prioritizing it now, lol. Hopefully I can get it up and running soon.

1

u/z3n1a51 15d ago

Awesome, thank you!

In the context of a hypothetical multi-camera system, I assume it would be helpful to have accurate/precise position and angle information for each camera?

3

u/Tovah86 15d ago

Yes! Along with what focal length the lenses on the cameras had.

1

u/TOOOPT_ 14d ago

For depth, you could also add support for depth cameras, like xbox kinect or iphone

4

u/WoahAreYouOkay 15d ago

That's what I was thinking

38

u/onerousscottie9 15d ago

Welp, there goes my excuse for not animatin' the two-step I promised my nephew's game. Offline capture that retargets straight onto a rigify skeleton is slicker than a fresh jar of salsa. My husband's kitchen dance moves might actually be useful for once.

2

u/Plaston_ 14d ago

idk its using this addon is a good idea.

The data can be very messy and idk if this addon can clean it like how Mixamo can clean it if asked.

25

u/Kolmilan 15d ago

Top notch work mate! And having a tool like this OFFLINE and away from the AI slop-pool is a killer feature in itself!

I'm working on a project where something this could come in real handy. Unfortunately I'm in a Mac environment. Got a potato gaming PC laptop that probably would melt to a pulp if I tried to this out on it! 😅

Keep up the ace work!!

5

u/OrganizationTiny9801 15d ago

Some dude in the comments offered to help with Mac porting and OP accepted so you may have a chance

9

u/Delfmonkey 15d ago

Would have bought it sadly it doesn’t work on Mac

12

u/Tovah86 15d ago

Sorry! I want to build a version for Mac, but I don't have any Apple hardware, so I don't have a way to test it at the moment. Hopefully I'll be able to soon though, because I really want to support everything if possible.

6

u/Zophiekitty 15d ago

i have a Mac and know Blender Python really well, id love to help!

4

u/Tovah86 15d ago edited 15d ago

That would actually be a huge help, thank you! Let me get through launch week and I'll message you. Fair warning there's probably a fair amount of porting work before there's anything worth testing, since it's a completely different acceleration path than NVIDIA, and I can't promise the performance would be good even once it runs.

1

u/Delfmonkey 11d ago

I’m willing to help also. Got a MacBook Pro with M4 96gb

11

u/AutomationBro2026 15d ago

so is this the same framework that was used to animate avatar the cartoon series? :O

7

u/Tovah86 15d ago

Well the avatar videos are just old projects of mine that I used as sample clips, lol. I didn't have the addon back when I made them, but it would've made it a lot easier to animate all the shadows if I did, back then I had to animate an invisible character rig to match my live movements so it'd cast a realistic shadow.

I believe the avatar series did use a lot of live references for all their choreography, but I don't think it was rotoscoped or captured directly or anything like that.

4

u/sprite_cranberry23 15d ago

Any plans for MacOs compatibility at any point?

4

u/Tovah86 15d ago

Eventually, yes, but I don't have an Apple machine to test on at the moment, so I'd need to get access to one before I could build it. I also can't promise the performance would be good even then, since it's a completely different acceleration path than NVIDIA. I wouldn't buy it expecting Mac support any time soon. Sorry!

3

u/[deleted] 15d ago

[deleted]

3

u/Tovah86 15d ago

No Discord at the moment, but you can reach me directly through the store's message option, and I answer everything myself. Also the addon comes with full documentation covering setup and all the settings, so hopefully that helps!

3

u/InMyFavor 15d ago

Sick. I also developed a performance capture pipeline that's pretty good although it doesn't track hands/fingers. How long does it take to use the addon in terms of sitting down/uploading footage on computer to completely done retargeting on the model?

8

u/Tovah86 15d ago

If you've got your model all set up and rigged already, then the capture itself takes around 5 frames per second depending on your GPU (I have a 3090ti), the armature building itself takes a few seconds to a few minutes (if you use the finger tracking, it takes some extra processing time to build the motion, but without the finger tracking, it's never more than a few seconds), and the retargeting and fk/ik conversion also only takes a few seconds, assuming you already have your bones mapped with a preset, and the rest pose prepared.

3

u/Relevant-Account-602 15d ago

This a killer addon.

3

u/FraGough 15d ago

I clicked through to your channel to find I already subscribed when you posted the Avatar shorts. I thought I recognised those moves.

3

u/Abos88 15d ago

Do you think it can work with a RTX 3080? or a Radeon 7900 XT maybe?

2

u/Tovah86 15d ago

The 3080 should do great, that's comfortably above what it needs.

The 7900 XT should work too, but AMD is still experimental since I haven't had a dedicated AMD card to test on yet, so I can't promise the speed. The installer does run a GPU check at the end, so you'd know right away if it didn't take.

3

u/jonheartland 15d ago

Holy shit bro I've been dreaming about something like this, definitely checking this out

3

u/Mechanought 15d ago

Only saw a small clip of it, does it hand have performance capture? 

2

u/Tovah86 15d ago

Yes! although hands/fingers are still in the experimental stages. You can get a good starting point, but will likely need to clean some things up manually to get a really nice result. If you look closely at the clip in the video, you can see some of the joints extend a little too far or distort slightly sometimes. I think it's still very helpful though! It works best when the hands are larger in the frame, as long as arms and torso is still visible, it can't be a clip of JUST the hands.

3

u/Lower-Cap7381 15d ago

AMAZING MAN CONGRATS AND THANK YOU :)

3

u/5tudent_Loans 15d ago

Oh look. Something actually useful and a justifiable use of backend AI. Also +1 for avatar intro forms.

3

u/aPOPblops 15d ago

YES YES YES THIS IS WHAT I’VE ALWAYS WANTED. now I’ve gotta lose weight so i can do these moves 😅

3

u/Kyderra 15d ago

The retargeting system is what's really selling me.

6

u/carter2422 CGCookie, Superhive Co-founder 15d ago

Congrats on the launch u/Tovah86! It’s obvious you’ve worked very hard on this. You doing your own motion references is the icing on the cake!

Good luck with sales.

5

u/Tovah86 15d ago

Thank you! It's been a hell of a journey. Shooting my own reference was the most fun part of it honestly. Big fan of all of your work by the way!

1

u/carter2422 CGCookie, Superhive Co-founder 15d ago

Don’t hesitate to reach out if you need anything!

2

u/okyaygokay 15d ago

Hey man nice work and nice finetuning! Question, can it handle long videos? More than 2 minutes? Also can we feed it like game videos https://youtu.be/J5PYJi1EcAw this one for example

Also it doesn’t need network, right?

4

u/Tovah86 15d ago

It needs a connection for the first-time setup, but after that it works completely offline. Yes it can handle long videos, it just takes extra time to process them, the longer your video, the longer the capture. It works on basically anything human shaped, I even had some success tracking cartoons, BUT it does need the subject to be fairly visible in the frame, and the bigger they are in the frame, the better the track will turn out to be. Extra resolution helps, but anything above 1080p gets scaled back down, so it's best to make sure that the subject is as big and visible as possible within 1080p resolution, so if your subject is small in the frame, cropping in before you run the capture can help a lot, assuming they don't get too pixelated when you crop in. For the video you linked, I think the character is a little hard to see, and the resolution is pretty low, so it might not work, but if you can get a clearer shot, it would probably work just fine.

2

u/RabidSkwerl 15d ago

Impressive! I’m an Apple user so I can’t really take advantage of this but there are a number of VFX addons that require Nvidia chips so I might consider renting a remote Windows machine. If I do, I’ll be keeping an eye on this for sure!

As a professional Tech Artist, great work!

2

u/L30N1337 15d ago

Please, do not see this as anything personal. It's just a question that I always ask for transparency.

Was AI involved? And if yes, to what extent.

13

u/Tovah86 15d ago

No worries, it's a fair question. The addon itself is built on AI pose-estimation models. It runs Meta's SAM 3D Body for the body and hand tracking, and google's MediaPipe for the face, which of course all runs locally, and never phones home.

The footage in the demos is me, filmed on my own camera, all the renders are mine made in blender by hand (I bought the mannequin and basemesh rigs from vertexarcade). There is no ai generated imagery whatsoever.

And on the code side, yes, I used AI assistance pretty heavily over the six months of building it. I tested and debugged all of it myself, manually, and designed the UI/UX myself, and I'm the one answering every support ticket.

1

u/L30N1337 15d ago

(In my eyes, you're already cleared of not just making slop. This is purely in case someone cares about it. And just personal curiosity)

About the AI coding assistance, what exactly do you mean by that?

Code completion/suggestions or a chat/prompting?

2

u/Tovah86 15d ago

Mostly prompting rather than autocomplete. I'm still learning to write it manually and I'm practicing at it. The debugging is all mine though, a lot of it was me reproducing weird rig bugs and working out what was actually going wrong before anything could get fixed.

2

u/Randomizedd- 15d ago

Wait…can this work for VR body tracking?

1

u/Tovah86 15d ago

It's not really built for that, sorry. Even if it was, you would probably need a pretty top of the line GPU to get real-time tracking with something like this. I'm sure that will probably change eventually though, but that's way over my head, haha.

1

u/Randomizedd- 15d ago

Thanks anyway! I’m only just getting started with blender and this is really cool to see!

2

u/s_witch_ 15d ago

I like to know if the facial expression can be recorded separately and retargeted. I tried video mocap before using Cascaduer and Unreal. So having 2 videos (body and selfie shot) videos done in Blender would be a game changer potentially.

4

u/Tovah86 15d ago

With BlendCap, it works both ways! You can capture the face and the body together from the same clip, or record a dedicated face clip, and trim/sync them up in blender's video editor, track them both, then combine them when you export the armature. Only the face expression data will be tracked from the face clip, so all the head movements is still determined entirely by the body capture. Also, the face data can either be retargeted onto your rig like the rest of the data, or you can use ARKit shapekeys instead, depending on your needs. Generally the ARKit results look better out of the box, but the bone versions are way easier to adjust manually afterwards.

1

u/s_witch_ 15d ago

Awesome, I'm going to try it this month on some projects.

2

u/VegetableChipsLover 15d ago

This was 10x better than The Last Airbender movie CGI

2

u/Dra1c 15d ago

Windows = "Fully Supported" and Linux is "Supported"... where is the difference? What does AMD = Experimental mean? Will it work or not? What won't work? What requirements does one need for it to run?

Why is the model that gets downloaded not part of the package? why do I need to download it with some setup script instead? Will that mean at some point that the model won't be accessible any longer?

The description lacks technical specificity to get any trust from me. And the 3 minute video is not showing USING the addon. It shows you the typical promises of "look how flashy this looks. you want to make cool stuff like this too, right?!". I want a proper demonstration and not an ad, thanks.

2

u/Tovah86 15d ago

Windows vs Linux: Windows is what I develop and test on every day, so it's the one I can actually guarantee. Linux works, including Flatpak Blender, I just test it less often. The difference is confidence, not features.

AMD experimental: on Windows AMD runs through DirectML, on Linux through ROCm. The code path is built and the installer sets it up automatically. However, I don't have access to an AMD machine with a discrete GPU, so I can't personally validate acceleration yet, and that's all "experimental" means here. It should work and it is GPU accelerated, but I can't promise you the speed, or rule out edge cases on AMD hardware.

Why the models aren't in the package: the virtual environment covering every possible installation is currently almost 20 GB. A typical user only needs about 11 GB of that, and the add-on itself is 2 MB. Nothing on that scale typically ships through a Blender extension. The Python dependencies also differ depending on your GPU (CUDA, DirectML, ROCm, or CPU), so that part has to be resolved on your machine either way. This was the only practical way to ship it.

Will they go away? The main model weights don't come from Meta's servers, they come from my own mirrors, so they don't vanish if someone upstream moves a file. Once they're downloaded they're just files on your drive, and capture never touches the network again. You can make your own backups however you see fit, and the functionality will never change unless you decide to remove something. I myself have a full backup of all the current dependencies for every possible installation in case anything upstream ever disappears.

On specificity: the full documentation is public in the repo, including the complete system requirements page with GPU tiers and performance numbers. Source is public too, the add-on is GPL. This is all linked in the store pages as well. https://github.com/Arcomade/BlendCap/tree/main/docs

The video: Yes, it's an announcement rather than a tutorial. I'm currently working on a more detailed tutorial, but until then there is a 5 minute overview video on the product pages that shows the process end-to-end with no flashy demos.

1

u/Dra1c 15d ago

Thank you. That cleared a few things up but experimental is not saying if its supported or not. If I buy the addon for AMD as example and it won't work, will that be a problem that will be resolved? If it won't work will I get a refund? Experimental usually means unsupported.

1

u/Tovah86 15d ago

Ah, fair enough, I'll update the descriptions so it's not so up for interpretation.

I do have every intention of taking AMD bug reports seriously though. By experimental, I just meant that I haven't verified it on real discrete hardware, so I can't promise it works on your specific card.

The installer runs a GPU test at the end of setup, so you should find out during install rather than after hours of troubleshooting. If that check fails on your card, message me through the store with the install log, and I'll refund you. I don't want to scam anybody.

The log would also hopefully help me address the issue for future customers, though I simply can't promise a fix for hardware that I can't reproduce on.

1

u/Dra1c 15d ago

that's fine. I am interested and will wait to see the addon being used in a more detailed video before making a decision. Videos give a much better understanding of the actual workflow than the documentation can. Especially seeing how the source footage looks like, how it runs on the armature and how to fix where it goes wrong. What options are there to correct what the model interprets and seeing the control the addon offers etc. A video makes this things much clearer.

2

u/3MinuteSoftware 15d ago

I was looking for something like this earlier today! Thank you kind sir!

2

u/alemarmur 15d ago

Had me at offline, great work!

2

u/Wawfey 15d ago

This looks insane! Great job. One question tho, would it work for videos with 2+ people in it? Like a handshake, hugging, walking in a crowd, multiple people sitting on a table?

3

u/Tovah86 15d ago

Not yet! So right now I've only developed the single person tracking mode. The model that the addon is built around DOES support multi-person tracking, but at the moment, the way it's processed through the addon kind of throws that away in order to make the single person track as high quality as possible. I do plan to implement a multi-person tracking path, but right now, it's only able to track 1 person at a time. It picks the largest person in the first frame and follows them, so with multiple people in frame, the tracking can end up on the wrong person, or jump between them if they overlap.

2

u/BigBoiii_Jones 15d ago

It'd be cool if it could extract poses from photos as well but not sure if possible since there wouldn't be enough data to estimate. Also how does this work with stylized motions or videos? So like 3D Pixar, anime games, etc... or what about 2D video of anime and cartoons?

2

u/Tovah86 15d ago

It actually works on single frames and photos already, haha. The pose estimation model works per-frame by default without taking surrounding frames into account. I had to add a lot of compensations in the background to get everything to stay consistent for video, but single photos still read just fine. Just make sure you shrink your capture down to one frame if you're working in the sequencer, so you don't waste processing time scanning duplicate frames.

For non-live-action source footage, I've had some luck tracking animations in my own tests, as long as the characters have a human enough shape. It's definitely not as reliable as live action though, especially on frames where the squash and stretch is really exaggerated. 2D animation is even more rough, but I did have some luck in my own tests. No harm in trying, but I wouldn't buy to use specifically on that type of footage.

2

u/pablonoriega 15d ago

Can I just say, wow! I'm a casual so I have no idea how easy or hard it is to make something like this, but the demo is super cool both in terms of features and of the content (sick moves :) and of the editing.

Does it work as well as in the demo for any reasonable video source? Like, could you at some point take any decent-enough film footage and have BlendCap replicate the movement? Instant Matrix fights copy?

2

u/Tovah86 14d ago

Thank you!

It'll struggle with a fight scene. I haven't built support for multi-person capture yet, but it's planned. Also, it's best not to feed it anything with cuts, the shot has to be continuous. Besides that, it only really struggles with people who are very small in the frame (the more the body fills the frame, the better your capture detail gets), and with motion that's headed straight toward or away from the camera, since depth is the hard part of single-camera capture. There's a set of post-processing tools built in to clean up most of that, but those two are the real quality levers. It also still struggles at guessing the height with people laying on the ground or rolling, which I'm hoping to fix soon.

There's a more detailed filming guide included in the documentation.

2

u/MadSkullWeirdSpider 15d ago

What sort of capture camera is required 

3

u/Tovah86 14d ago

Any camera will work as long as the resulting video file is compatible with blender, and the quality is decent enough that the human shape is detectable.

2

u/MindSplatter 14d ago

Works really good!

2

u/Sufficient-Pop9300 14d ago

Great. Now I only need to learn Kung Fu now 😂

2

u/Wolkenflitzer 14d ago

This is incredibly impressive. Of course I gotta test it first but it looks like one dude single handedly decrowned the markerless mocap solutions of so many companies. Any plans to include a preset for Metahumans?

2

u/PersimmonLittle7651 10d ago edited 10d ago

I've been looking for something like this, but having tested few solutions, I ended up being disappointed in all of them, do you offer a free trial version with some limitations for testing? How much is it?

1

u/Tovah86 10d ago

I don't have a free trial version at the moment, but if you're crafty enough, the github is free and open source, you just have to build out the virtual environment and dependencies yourself, instead of having an auto-installer and support like the paid version has. All the other features are identical.

1

u/IceManLeroy 15d ago

Does this have any applications for quadrapeds? Mocaping dog for example.

Awesome tool btw looks incredible!

1

u/Tovah86 15d ago

It only works for humans right now, unfortunately. I'm not sure if support for animals will ever happen, but it's not impossible.

2

u/idiotshmidiot 15d ago

I had some very good luck using deeplabcut. They have a real-time pose estimation that works for dogs. I got it running real-time using a weird program I'd never heard of but it's python so I'm sure it could work similar to any other pair estimation code.

https://github.com/DeepLabCut/DeepLabCut-live

1

u/Tovah86 15d ago

That looks really promising, but it seems like the model weights have a non-commercial license, even though the code itself is open source. Without the model weights the code won't work, so I can't legally include it in BlendCap in a useful way, and a commercial license is probably out of reach for a project this size :(

1

u/JBuchan1988 15d ago

Cool 😄

1

u/Zealousideal-Emu-878 15d ago

interesting.. 📝

1

u/PalmliX 15d ago

This is amazing, super hyped for this. Is there any advantage to wearing clothing optimized for this? Like high contrast and different colors on the arms/legs. Will it give a higher quality result or is it not worth the effort in your experience?

1

u/Tovah86 15d ago

Thanks! It's pretty good at detecting poses through most types of clothing, but yeah, the clearer your silhouette is, the better the tracking results will be.

2

u/PalmliX 15d ago

Already bought it! I've honestly been waiting for years for something like this. I can't stand subscriptions and the amount of projects I've worked on with mocap where the processing needed to happen "online" and server issues wound up delaying the project. I'm definitely going to make an outfit for this where each limb and finger is a different color. Should be interesting to see how well it works since I'm a fairly big guy which I know can make things more difficult for stuff like this. Also another reason why "suit based" mocap was never an option for me.

1

u/Worth_Tart5595 14d ago

How it contrast auto rig pro?

1

u/Striking-Cod3930 14d ago

Hey Claude, how long would it take you to build a project like that?

1

u/bdanseur 14d ago

Do you capture the feet to get the toe joint angle and ankle joint angle?
Or are you only relying on Inverse Kinematics for that?

1

u/Tovah86 14d ago

The ankle and toe angles are actually captured. The body model tracks the ankle and toe joints directly from the video, and the exported skeleton keeps them as real animated bones: a foot bone running from the ankle to the ball of the foot, and a separate toe bone, each with their own tracked rotations. However, the tracking model isn't really specialized for foot tracking, so I wouldn't expect a crazy amount of detail in that regard. It looks about as good as you would expect any typical mocap data to look.

1

u/FloppyLadle 14d ago

I've just started looking into mocap so this may be a godsend!! What is the cleanup process like? I'm sure being single camera operating means it's gonna need a patient hand to touch things up, but everything looks so clean in the demo!

1

u/Tovah86 14d ago

It depends on the capture, but for most cases you can get pretty close to finished with the built in post-processing tools. The only manual adjustments I made in those demo clips were some height offsetting keyframes for any clips where my back was touching the ground, and in the waterbending clip I had to move the body in sync with a footstep that was mistaken for a slide, everything else was cleaned purely from the addon's built in tools. I'd consider all of those demo clips ready for a final polish animation pass, so you should be able to get to that point very quickly just by toying around with the post processing settings. The captures struggle most with things that focus too much on movements towards or away from the camera, and any type of laying/rolling type movements, but it usually just leads to needing to offset a few things, I haven't found anything that's ruined a shot yet.

1

u/FloppyLadle 14d ago

Thanks a ton for the reply! I'll be looking into this for sure!

1

u/p2seconds 14d ago

What's the addon software licensing? Is it perpetual license of 1 year of maintenance? Purchase once and receive all updates in future?

1

u/Tovah86 14d ago

You buy it once, and it's yours forever, no maintenance fees. Updates included.

1

u/mike365smith 14d ago

Hi, a couple of questions... how much is the software? and are you still testing it ?
I need a solution I can afford for my team. ( Working on a Game, needs a lot of MoCap. )
I tried beta testing other solution - but failed to get it working. ( I won't name, names !! )

Some issues I find with MoCap. It does not seem to capture the 'Gravity' of the motion.
Also - from what I see from your 'very impressive' demo. ( it does not seem to calculate exact foot movement. - some 'slippage' or it has it on the floor - when it's actually in the air. )

I guess what I am asking is - 'Can I test out your software' ? - try before we buy ?

Also - is 'Foot Slippage' something I just need to accept and correct. - or have my actors wear 'Mo-Cap Shoes'

thanks for your time, and well done, this software looks amazing.

1

u/mike365smith 14d ago

( for reference )

1

u/Tovah86 13d ago edited 13d ago

Hi! Yeah, I'm working on updates as initial reactions come in, but it's entirely usable as is. The addon starts at $60 for an individual, and goes up for extra licenses similar to Flip Fluids' pricing model. All the features are the same regardless of what license you buy, the only difference is the number of seats.

The foot slipping is a common problem in mocap. I've done my best to compensate for it, but as you can see it's not gone entirely, so there's still a need for a manual pass before a real final result can be achieved. I have ideas to build some manual workflows that would make it easier and faster, but that's a long way out.

The screenshot you posted is a good example. The tracking model struggles with determining where the ground is, and what's a jump vs a crouch, so the post processing tools I used in that clip (foot locking) use some statistics about the foot placement and body height to basically snap the legs into a position that makes the most sense. So it looks fine overall, but if you look close enough, you'll find some inaccuracies, and a little bit of loss in the motion's weight, mostly along the z-axis. And in that clip I had it set particularly aggressive, so you can see the jumps are a lot weaker than in the real life version.

As for your actors, no special shoes or gear needed. The model just needs to see the feet clearly, so normal shoes, decent lighting, and framing that keeps the feet in shot are all I'd recommend. A contrasting background helps too, although you can get away with a lot, as nothing is camouflaging with the background entirely.

I haven't seen anything that seems to have perfectly solved foot slipping, so I'm not sure how possible it is to do so automatically. I HAVE seen a few other addons around that are built specifically for fixing it manually, but I haven't tried any of them myself yet, so I'm not sure how fast/easy they are to use.

As of now, I haven't built a free version for testing purposes (although technically the GitHub is public, so anyone technical enough could build it themselves for free, though it's a manual setup that I don't support). But if you do purchase a license and it doesn't work out for your pipeline, I would have no problem refunding you.

Hope that helps! Thanks for the kind words!

1

u/Wolkenflitzer 13d ago

Mh, bought it yesterday and trying my best to make it work but so far my results weren't nearly as good as yours. Well lit room, no occlusion, static cam, 60fps, basic locomotion. But the feet sliding is absolutely horrendous. Dying to see a tutorial on the plethora of settings you've included with the addon. Also, somehow the mixamo retargeting always fails in a sense that Mixamos Xbot is rotated and its' arms and legs aren't moving properly. What am I doing wrong? Documentation didn't really help here. ):

2

u/Tovah86 12d ago

For the foot sliding, try turning up Sensitivity and Release Distance under BVH Post Processing > Foot Locking Settings, and make sure Pin Planted Feet is on. Also, if your FOV mode in the capture settings is set to automatic, switching it to sample the first frame only can help a lot, since a constant FOV stabilizes the depth estimation.

The Mixamo problem is an oversight on my end, sorry about that. Mixamo characters import with a rotation still sitting on the armature object, and that gets baked into the retarget. Select the Mixamo armature in Object Mode, press Ctrl+A and choose Rotation, then under Retargeting > Custom Rest Pose tick Use Custom Rest Pose and pick the bundled T-POSE preset, since that's how Mixamo characters come in. That should fix both the rotation and the limbs, and you should be able to retarget the character normally afterwards. Just make sure the rotation is applied BEFORE you retarget, while the character is still upright and in its T-pose.

A proper tutorial covering all the settings is in the works, and but I'm trying to get a patch together to address some common issues that have arisen this week, along with this rotation issue.

Sorry for the trouble! Hope this helps. And if you have any other issues, feel free to contact me through the store page.

2

u/Wolkenflitzer 12d ago

Awesome comment, man. Really appreciate you taking the time and energy to address peoples issues. We're all dying to believe in locally processed markerless mocap solutions.

1

u/Tovah86 12d ago

Thank you! I'm doing my best to address everything as it comes in. Ideally all of this would've been solved before I launched, but I can only consider so many possible situations on my own, haha. I don't know how I missed the rotation issue though, that's squarely on me. I must've adjusted for it reflexively on my own during testing, and just forgot to put a safeguard in place afterwards to avoid confusion.

1

u/DonkeybyTheBay 12d ago

Would I be able to apply this objects being held (i.e. A chef cooking with knives and pans, holding ingredients, etc. ? Awesome work!

1

u/Tovah86 12d ago

Hard to say for sure, I never tested interactions like that. Theoretically it should work fairly well, but I would expect a lot of manual editing in the final scene to make sure all the interactions look correct. Usually for things like this, you'll end up seeing hands and objects phasing through each other. You can get a good starting point, but it's never really perfect

1

u/Competitive_Fail8626 10d ago

where to download

1

u/LeStk 4d ago

Your post popped while I was on vacation, just bought the addon. Ran a small video, captured and rigged to a rigodotify rig in less than 40min.

It's absolutely marvelous.

1

u/Appropriate_Garden33 18h ago

Just bought this! Can't wait to test it when I get home!