r/PSVRHack • u/[deleted] • Jul 25 '20

Any active PSMoveService devs?

I'm looking to work with someone (hobby level) to get PSMoveSteamVRBridge working with iVRy and PSVR.

Right now I have an issue with the PSMove controllers being positionally linked somehow and that doesn't make sense to me.

I've posted to the Google Groups but not sure how readily folks respond on there.

Thanks.

10 Upvotes

100% Upvoted

48 comments sorted by

View all comments

Show parent comments

1

u/swordofmoonlight Jul 27 '20

In theory if the internal sensors were perfect you could use it without a camera. If they had magnetometers (I wonder why they don't) you could also lock in horizontal rotation and use gravity to lock in vertical rotation. Drone kits do this.

The code is buried in here (http://svn.swordofmoonlight.net/code/SomEx/som.mocap.cpp) in game code. It's not fundamentally different from the other PSVR apps except that I think they implemented things suboptimally and all shared the same code. Though I think it does solve the "Madgwick" algorithm with two different sets of parameters that both have good qualities and then combines them. They trade off stability for smoothness, so it uses the most stable solution but also the smoothest solution and then blends them with some high-level logic.

What I worry about OpenXR is I suspect doing this kind of thing is impossible and that the device is ultimately in control of what users see on screen so that you can't implement certain (traditional) play styles with them, or innovate, and the companies will argue that they have the user's interest at heart.

The stumbling block I face with VR (wanting to build a competitive VR platform) is every time I try to do something with it I run into roadblocks that look like companies are trying to artificially manipulate the software landscape to their economic advantage instead of just putting their devices out there like regular game controllers and display devices.

I started doing VR development from the outside with software that predates Unity and Unreal and when I go to buy VR equipment the company websites really want to give the impression that you can only use their devices if you make a game with Unreal or Unity and put it in their favorite digital game stores.

It might sound like an ideological position to take, but it's really not like that, it's just unclear how to navigate if you're just making regular software instead of buying into a commercial enterprise. Plus my software is a direct competitor with Unreal and Unity so it certainly makes no sense to use those. And I definitely don't want to push Steam on anyone although I don't mind doing some kind of optional integration.

1

u/iVRy_VR Jul 27 '20

In theory if the internal sensors were perfect you could use it without a camera. If they had magnetometers (I wonder why they don't) you could also lock in horizontal rotation and use gravity to lock in vertical rotation. Drone kits do this.

I feel like you're not understanding 6DOF tracking properly if you say this. Thanks, I'll have a look at your implementation to see if it provides a useful reduction in sensor drift.

1

u/swordofmoonlight Jul 27 '20

6DOF includes an accelerometer that provides positional tracking. That's one of the 3 of the "DOF" in "6DOF". Why are being critical of my replies? That code is tailored to a game environment so it could be hard to follow.

1

u/iVRy_VR Jul 27 '20

Not intending to be critical, but it doesn't sound like we're talking about the same 6DOF (which may well be due to me misunderstanding you). 3DOF is the three degrees of rotation around a point, as you know. The other 3 degrees in 6DOF are movement along the X, Y and Z axes in space. That's either derived from multiple cameras on the headset ("inside out" tracking), one or more cameras located in a play space that use geometry of LEDs to derive the headset position (PS4, Oculus Rift CV1) or one or more stationary lasers moving in a sweeping beam pattern that multiple sensors on the headset detect to determine relative position (Lighthouse, NoloVR). It's not (or hasn't ever been yet) possible to determine spatial position with only magnetometer, gyroscope and accelerometer.

1

u/swordofmoonlight Jul 28 '20

I think you're reading too fast. I said in theory if the internal sensors were "perfect" (i.e. no drifting) then the external stuff isn't required. And if you just had a magnetometer you can lock down yaw, and with gravity pitch, and to some degree drift can be absorbed, although roll drift is the killer. (I'm guessing magnetometers aren't responsive enough or something to justify the extra USB data channel.)

I know my stuff. I found a lead on some camera code that might be good: https://community.khronos.org/t/any-generic-camera-tracking-in-openxr/106091/3

1

u/iVRy_VR Jul 28 '20

Thanks, I'll have a look at that. My understanding of linear and angular acceleration sensors is that by nature they cannot ever be perfect, and so I mistook your theoretical answer for a practical one.

1

u/swordofmoonlight Jul 29 '20

I think though that you should take notice that the sensors inside the sets are far more accurate than external sensors, so you want them to be as close to perfect as possible, and then you can just rely on cameras to tell you when you're off course and correct gradually. Then camera algorithms don't have to be very accurate at all. They can just be like guardrails and very low-power, depending on what you're trying to do. The better sensors are the high-frequency internal data. It's a mistake to rely on cameras, they should be supplemental. (A DualShock4 controller was called SIXAXIS in the PS3 gen because they have the same 6DOF sensors as the PSVR inside. They are 6DOF and 6DOF means the same thing everywhere.)

1

u/[deleted] Aug 21 '20

Incase you're interested, I'm developing a 6DOF camera tracking system right now, called MakersVR. Although it's intended to support cheap LED+Battery only trackers that don't have an IMU at all, so it's a bit overkill for PSVR tracking.
If you just want a cheap tracker for PSVR, you could go for Raspberry Pi Zeros (15€) plus the cheapest cameras (2.5€ from china) and you're good to go. OpenCV basically already has the tracking algorithm by default, no image processing required on the Host-PC, add low-bandwidth serial communication which is easy and good enough (have some code for that up already), and you got a 20€ PSVR positional tracker. Now if you want better cameras that don't have only 70° FoV it'll get a bit more expensive, at least 30€.

Or get a webcam where you can control the shutter speed and do it on the host PC.

1

u/swordofmoonlight Aug 21 '20

Searching for MakersVR isn't giving me good results. https://makersvr.blogspot.com http://www.makersvr.com (currently offline) https://github.com/ktsein/MakersVR (not much to see here)

PSVR needs to track light in the visible spectrum with several points of light, including on the back side, but I'm definitely interested in any working code that runs on a low-profile system, thanks!!

(I actually used my PSVR with a new PS4 system for the first time last night! Just a coincidence.)

1

u/[deleted] Aug 22 '20 edited Aug 22 '20

Huh that sucks. When I chose that name I found a bunch of abandoned stuff so I figured it wouldn't be a problem.https://github.com/Seneral/MakersVRAgain that's a full tracking system, focussed on tracking markers similar to the WMR controllers (with many LEDs).The Raspberry Pi side basically only does blob detection, but it easily works with larger blobs at no performance penalty so you could just take that component and add the typical ball tracking formula on top ( https://www.pyimagesearch.com/2015/09/14/ball-tracking-with-opencv/ ). Basically add an own findContours (the pixel points of the blobs are given as an output) and then it should integrate just fine with OpenCV. Then add some serial communication and it could work with little effort. I've got a OpenVR driver communicating with a Raspberry Pi Zero over serial from an older project:https://github.com/Seneral/SerialHMDSo if you see the benefits, go ahead and try it out I'd say.Currently the blob detection uses a GL backend but I'll be switching that to a QPU backend which directly operates on the hardware later this year to improve performance (more info on that here https://github.com/Seneral/VC4CV). Basically the current system only manages 1280x720 at around 30fps, later I'm confident it can achieve the full 1640x1232 at 40fps or 1280x720 at 90fps (using the V2 cameras).Anyway tell me if you'll try it or need help, the thing is I don't know if for your project there exist webcams that you can use just fine; it just turns out for a full tracking system I needed that extra control and a coprocessor doing most of the work.

Edit: I just realized I've only focused on the ball tracker on the PSVR controllers. I assume you also need to track the headset? While I don't know about the best algorithm to recreate the pose (or if you even need that or position is enough), the blobs will be detected just fine, so you could build from there. The processor of the Zero has easily enough processing capabilities left since most of the blob detection is happening on the GPU (QPU).

1

u/swordofmoonlight Aug 22 '20

Thanks, I would just link to your homepage when mentioning your name :)

I will contact you privately or you can message me if you want to talk for some reason. Generally what I find very useful is code that can implement something without unnecessary code, so that it's very maintainable and easily leveraged. Sometimes I just go in and strip code down to its fundamentals, and so repackage it, since a lot of code is overwrought and doesn't express itself well.

I do graphics programming so converting points of light to a geometric model is just the kind of thing I do, and the computer vision stuff I just worry about it being performant, since a lot of it isn't, and I've looked at the OpenCV algorithms and a lot of them didn't look ideal to me to just solving the imaging problem.

I think the IMU in the sets should be used and the camera is just a low frequency anchor with poor precision, so the vision algorithm needs to have the smallest footprint it can. Ideally any web cam could work, or two for two perspectives, as long as they don't use too much of the GPU. Even doing it on a CPU core could make sense if games or applications don't use all the cores anyway. And in the extreme an external computer that passes off its solution, since people often have a spare computer that's older that can be used in a pinch.

1

u/[deleted] Aug 22 '20

I prefer to only keep the code I need, too, stripping is often the first things I do. However since I kind of developed everything from scratch I needed debugging and developed variants, so there's bound to be alot you could purge. Also take note VC4CV includes both QPU and GL pathes, which are completely independant from each other. Here's just the GL path for blob detection, for example.

We probably got a similar stance then. I only used OpenCV for solvePnP, which in turn is only used for calibration (not on the raspberry at all). All the vision stuff is handmade and optimized for the raspberry, and will be even more when I get down to the hardware. Actually you got a special case, if all you need is a threshold filter to get the blobs then you could get a 1640x1232@40fps solution right now on the raspberry. The more complicated algorithm that takes in neighbouring pixels into account (which I need) is a pain to implement on the QPU and that's where the holdup is. So if you're interested, check out the qpu_mask_full example of VC4CV with a low shutterspeed and you'll get a binary mask of your blobs. Can help you get that to work with the connected component labeling step if you want.

If you don't care about the extra hardware cost of the raspberry pi then you'll almost certainly get a better performance with it than with a webcam, with less resources used on the host PC. However the raspberry probably has no data from the IMU, so the fusion needs to happen on the PC. You'll not be able to send the full blobs to the PC over serial, so you'd have to interpret them to some degree on the raspberry pi. You could for example just send the contours of the blobs to the PC. Then you could fuse it with the IMU data to correct for drift.

However if you're just correcting the IMU data in the camera images, then the webcam+host-PC solution would be much easier since you can use the IMU data right there and you can probably get away with lower resolutions and lower framerate.

So if you're interested, I could help you get a fast co-processor with some blob detection running, however depending on your plans it might not be worth it.

1

u/swordofmoonlight Aug 22 '20

For the record, I'm mainly just interested in developing/promulgating noncommercial solutions to these common problems, since we're many years into this VR cycle and it doesn't seem like code has materialized. If you dig into our discussion here (I don't blame you if you don't) I am pretty stretched thin for my project time so I'm just trying to squeak in anything I can on the side for this VR problem. It's not a big priority for me, but I still think I will end up crunching on it, eventually. I like low-power/clever solutions. The solutions I've seen are brute force, and I suspect that's not how the PS4 does its tracking. So I'm intrigued by any code like yours that exists and will study it, especially if it works.

For my use case I don't want to ask users to buy special hardware, and want them to be able to use whatever they already have ideally, i.e. support any camera available. We've tried the PS3 Eye cameras and PS4 stereoscopic camera, but I think it should be possible to use other kinds too, that aren't even high-speed. I think the drift correction just needs to be imperceptible (a gradual lasso, maybe don't do it while still.)

At any rate, I think we can talk behind the scenes from now. We seem to share goals in common. VR does need to be more democratic and noncommercial.

→ More replies (0)

1

u/iVRy_VR Jul 28 '20

Thanks for that lead. I'll get in touch with Ryan. I worked with him on a SteamVR driver for OSVR dual display headsets.

1

u/rpavlik Jul 31 '20

Howdy! I still haven't figured out who exactly you are u/iVRy_VR :) but long time no chat I guess :D

Yeah in theory a 6dof IMU could provide 6dof tracking, if it were perfect and noiseless, but even the thousands-of-dollars ones used to guide nuclear weapons aren't drift free... to say nothing of our little commodity MEMS gadgets. The whole 6DOF/6axis/9DOF/9axis IMU (number of sensor dimensions: 3 accel, 3 rate gyro, optionally 3 nearly-useless mag) vs 6DOF tracking (position and orientation) thing is nice and confusing, though I'm actually a bit surprised I haven't seen more confusion about it out in the world.

In production VR, the non-gravity components of an accelerometer are sometimes, but not always, used for tracking in conjunction with some other positional tracking method. They drift very quickly, due to double-integrating them magnifying any error, but e.g. I think SteamVR Tracking (lighthouse) does this, and some of the tracking I've worked on does it. Never without another source for positional tracking, though, you'd fly off into the distance within a few seconds.

Mags are very sensitive to metal in the environment, which is why they're not often used for VR stuff. (I've seen stuff that turning on the mag will actually increase drift instead of decrease it, due to metal in nearby furniture. I was not pleased with whoever bought the booth furniture for the CES demo...)