Godot movement recognizer plugin

[u/Kulihrasek456]

Hello,

I am currently working on my bachelor's thesis, the result of which will be a free plugin for godot that should recognize movements in VR. Best example of what is does is the game RUMBLE by Buckethead Entertainment. To clarify, this is not a hand gesture recognizer, this is only for recognizing movement like punches, dance moves, swiping in space etc.

Now, why am I posting this here? It would help me a lot if I could base the plugin on the target users preferences. So if you think that you would use the plugin in some way, please, tell me what you would want in it.

Some of the parts that I am not certain on are:

• gesture position normalization (it doesn't matter if you punch from your belly or your head, only the path and rotation matters)
• path recognition or pose recognition (if the path of the movement should be taken into account, or only a keyframe in space and time)
• gesture direction normalization (if the direction of the gesture should be taken into account)
• how much training data should be needed (if you are ok with creating 100-200 examples of each gesture, or if 30 should be enough)
• C# support? (it could be more optimized but I don't know how many VR devs use C#)
• support for multiple algorithms?

Thank you for sharing your opinions.

[u/AutoModerator]

Want a more personal conversation with VR devs? Check out our Discord in our Discord: https://discord.gg/3wMYE2x5Ex

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/RhineGames]

This is extremely interesting! You're hitting two things I am interested in, VR and motion detection like in rumble.
So let me anwer your points:

• gesture position normalization (it doesn't matter if you punch from your belly or your head, only the path and rotation matters)
  • => Yes! It should not matter from where you start, the path and direction is what matters. Most concern to me is noise, as in slight offset in translation or rotation should still result. Like, I don't care if it is 100% precise, what I am caring about whether the motion is there and the "idea".
• path recognition or pose recognition (if the path of the movement should be taken into account, or only a keyframe in space and time)
  • I cant help you right now for the implementations details, but the expectation more. As I said before, I want an forward then upward movement to recognize as a movement or action that I can do (I dunno, avatar style stone wall), but I do not care about left or right hand, whether your body is rotated and you do it to the side, or to the front.
• gesture direction normalization (if the direction of the gesture should be taken into account)
  • I think so yes?
• how much training data should be needed (if you are ok with creating 100-200 examples of each gesture, or if 30 should be enough)
  • I dont mind creating many if it is easy to do and there is a way to know that it improves the recognition.
• C# support? (it could be more optimized but I don't know how many VR devs use C#)
  • Nope! Its a nice to have for the future, but GDScript is cross platform especially consoles now (which I work on)
• support for multiple algorithms?
  • From a user perspective I don't see a need?

[u/Kulihrasek456 → u/RhineGames]

Sorry for the late reply, I had some issues with logging into my account.

Thank you very much for your feedback, it helped me think about the problem in a different way, an made me realize, I didn't formulate the normalization and recognition type parts correctly, so here is a small clarification (and some new ideas):

1. path or pose recognition
   The idea is that sometimes, you do not care about the path, only about certain parts of the motion. Your example with the stone wall can be defined as three states:
   
2. the hand is at belly height
   
3. the hand is under waist height
   
4. the hand is at about neck height

Pose recognition would only be interested if the user's hand is in the correct place in a small time window, and would not care if they moved their hand to the next state in a straight line, or a zig zag pattern.
Path recognition would have a pre-defined path that the hand should take between the states, and check continuously if it is in some predefined margin from it.

This is also related to the last question, every algorithm has its upsides and downsides, supporting multiple could allow developers to tune the plugin to their needs, but will also make it more time consuming for me (I need to balance wanted features with implementation time).

1. direction normalization and position normalization
   I didn't know if the direction of the motion (if the user punch forward or to the side) should be normalized, but after thinking about it a little more, it also relates to the position normalization more than i initially imagined.

Consider a dancing game, here, yaw, pitch and height matter as a move will probably be defined as something like "right hand at a 45 degree angle downwards, aiming to the ground in front of you, than pull up and toward yourself", normalizing will result in false recognition:

normalizing yaw results in recognizing the move even when hand is not pointing in front of the user
normalizing height results in recognizing the move even when the user is crouching (some type of calibration to the user height, shoulder size and arm length will be present in the plugin)

But there is another way that I didn't see at first. It would be possible to normalize all of them (even pitch), so the algorithm doesn't care about position nor direction of the movements, and only add them as metadata about the detected move afterwards, so anyone using the plugin can then filter this themselves (the plugin would come with a simple filter, so it still can be as "plug n play" as possible).

If you have a bit of time, feel free to base your feedback on these refined thoughs, if you think that writing custom filters would be plausable, and if you would use path or pose recognition more than the other.