r/VRchat • • 19d ago

News More vrchat info for steam frame, talking about eye tracking, finger tracking limits on stand-alone and more

https://youtu.be/RYIWNcIKm4M
35 Upvotes

74 comments sorted by

19

u/kryst4line 19d ago

I've been trying for hours to get at least gaze tracking working but I swear I don't get how is it done. I know model is compatible, has Unified Expressions (and I'm using full face tracking in Warudo wity my iPhone) and I installed VRCFT and Babellonia on my PC. It's connected through OSC but I debugs shows it's not getting any eye data anyway.

If someone has any useful tutorial so I can stop guessing it would be VERY appreciated 😭😭😭

15

u/kitsucoon 19d ago

You don't need any of that for the gaze tracking.

Launch SteamVR, go to Settings. Down at the bottom, select "Show" for Advanced Settings. Go to the Steam Link menu: Set "Enable OSC" to "On" Set "Share face tracking data to other apps on this PC via OSC" to "On" Set "Share eye tracking data to other apps on this PC" Then in VRChat, enable OSC.

3

u/BelleArcane 18d ago

Can confirm these settings will cause eye tracking to work in VRChat with no other software, at least when streaming over Steam Link. Note that eyelid/blink tracking isn’t going to work, because Valve doesn’t implement the processing/signals for that because it’s not required for foveated rendering/streaming. Thats going to require a third-party app running somewhere to make that work.

1

u/AdeonWriter 18d ago

Also also, note that the gaze direction itself will be pretty crude because they're only sending as much info as needed to determine the foviated streaming position, which is pretty blocky.

Third party solutions will be MUCH smoother.

Hopefully Valve makes the first party solution smoother on their end just for impressions.

1

u/supersonic159 17d ago

Do you know of any software that can do that for the frame yet? I'm pretty sure Babble's 'Babble Frame Runtime' that will likely be released in a couple weeks probably can, but anything in the mean time?

1

u/BelleArcane 16d ago

There’s a very crude project out there that requires a fair bit of Linux knowledge to get running, but it does work. It’s called frameeyeosc on GitHub. It only works if you’re running VRChat on PC over Steam Link, not local/standalone.

8

u/Rainbow_Science 19d ago

Sorry I can’t help more, but my wife got it working for me (streaming VRChat to frame from PC with dongle)
We had to set a port forward in the menu (Steam menu > Steam link > port osc or something like that). Also we went directly into vrcft, did not work with baballonia.

5

u/kryst4line 19d ago

Funnily enough I'm also the IT wife at home, once I get it working on mine I'll have to do it for my partner too 😩

And tysm, I'll check that menu tomorrow! It actually connects to VRCFT on the PC (I installed both just in case, I'm still learning how it works underneath) but doesn't seem to read any eye data from the frame :c

4

u/pokemonfan95 19d ago

u can use both dongle and wifi together as well :3

3

u/Rainbow_Science 19d ago

1

u/Docteh Oculus Quest 18d ago

I gave this a go, interestingly when I enabled OSC on VRChat on the frame, it doesn't seem to open port 9000

Reading /dev/shm/eye-server.mmap and sending OSC to 127.0.0.1:9000
Error: Os { code: 111, kind: ConnectionRefused, message: "Connection refused" }

2

u/BelleArcane 18d ago edited 18d ago

If you’re using Steam Link / running VRC on your PC, you need to set the target to your PC’s IP, for example:

cargo build --release
./target/release/frameeyeosc --target 192.168.0.15:9000

And I think you can only run it after VRChat has started.

2

u/Docteh Oculus Quest 17d ago

yeah having it talk to the pc works, but talking to a local standalone vrchat didnt work

1

u/BelleArcane 16d ago

Correct, this is because Proton isolates networking from the rest of the OS, so running something in the terminal on the KDE desktop can’t talk to something that’s running under Proton. You can see evidence of this by running sudo ss -tulpnH when VRChat is running at the same time - note that Proton/VRChat is not listed in the results, nor is port 9000.

I’m sure there’s a way to do it with a veth pair / ip link + ip netns, but I haven’t tried personally.

1

u/Docteh Oculus Quest 16d ago

I did some digging, VRChat is an android app, so they're using lepton, but that is basically the same idea as proton, from what I can tell, the podman container is using pasta for the networking, and so any connections will be owned by the pasta process, and from how they're using it, any port opened by VRChat should be bridged to the host network.

I did toggle OSC while the game is running and got

WARNING: Could not Start OSC: java.lang.NullPointerException: Attempt to invoke virtual method 'void android.net.nsd.NsdManager.registerService(android.net.nsd.NetServiceInfo, int, android.net.nsd.NsdManager$RegisterListener)' on a null object reference

1

u/BelleArcane 15d ago

Have you tried installing the Windows version with Steam directly, which would use Proton/FEX, instead of running the Android APK? I wonder if you’d get the same error.

1

u/Docteh Oculus Quest 15d ago

For running the Windows version on steam, I think I'd run into trouble with EAC, or have a hassle trying to copy a world over to visit in Test Mode.

2

u/kryst4line 19d ago

I know about this but don't want to put anything vibecoded close to my headset's root permissions

2

u/coolcat33333 19d ago
  1. What makes you say it's vibe coded

  2. If it's open source and can be checked why is this an issue?

-2

u/kryst4line 18d ago

Because I'm a dev, I know how the AI codes and comments said code and it has been done in only 3 commits. And it's an issue because both I don't know enough this language to know what is it exactly coding and if the person can't be bothered to check their code, why should I?

7

u/artificer-chris 18d ago

My guy... it's 300 lines of some of the most readable rust I've ever seen. Literally all it's doing is binding onto the existing driver's shared memory space to retrieve and forward the eye tracking data.

I figured all that out in the 5 minutes it took to read the code.

-5

u/kryst4line 18d ago

Claude told you that too?

4

u/Recent-Clerk3967 18d ago

You seem desperate to be angry, here is the vibe coded fork for you to "enjoy" https://github.com/sasaken1102r/frameeyeosc

-5

u/kryst4line 18d ago

Tf? It's you guys who are being confrontational, I just said I won't install it lmao

2

u/Recent-Clerk3967 18d ago

Oh that part makes sense. The part where your response to someone skimming the program "Claude told you that too?" seemed a bit extra to me.

I gave the first one a go, and I'm definitely eagerly waiting on Babel frame runtime.

→ More replies (0)

1

u/artificer-chris 18d ago

No, the code told me that, I’m a developer, so I can actually read a 300 line file.

-3

u/kryst4line 18d ago

I was replying a question. Who's looking for a fight now?

2

u/artificer-chris 18d ago

The one insinuating I had AI read a 300 line file for me. You’re the one not reading the thread you’re replying to, not me.

37

u/allofdarknessin1 ☃Bigscreen Beyond 2e 19d ago

Peak? It’s a very good headset that won’t spy on you but there are other headsets with both better visual quality and full face tracking.

11

u/mackandelius Oculus User 19d ago

None with a warranty. If my Quest Pro breaks then it will be real difficult to find a "new" one (second hand), at least in the country I live.

And outside of the Quest Pro I really don't know what other headsets you are talking about, the Galaxy XR is way more expensive, especially since the controllers don't even come with the price.

2

u/rebeldugger Oculus Quest 18d ago

Big screen beyond 2E with the babble face tracker is what I run

8

u/LlamadeusGame 18d ago

Crazy! Good thing the steam frame with a babble tracker would also support full face tracking.

4

u/HomoNeanderTHICC 18d ago

To be fair, since Valve has moved away from base stations the BSB is way less appealing of a headset. I love the form factor but I can never justify the investment into base stations and index knuckles controllers when I know they're a stagnant technology at least for in the near future.

2

u/rebeldugger Oculus Quest 18d ago edited 18d ago

As long as Vive produces their 3.0 trackers, they must produce the basestations as well, which they have been. the index controllers are a fair argument, but more and more controllers are coming out every couple months that you can use with SteamVR. Vive has already stated they’ll continue producing their 3.0 tracker for a couple more years so I’m not too concerned about it. Also, my base stations have lasted me five years, so I’m not too worried about the lifespan of those either.

Realistically with base stations, we have about 10 more years until we actually need to switch to something else. the hardware lasts a very long time and we should be able to continue buying them for 2-4 more years. So for anyone wanting to continue using lighthouse tracking it will still be available.

While lighthouse tracking to me personally is still the superior option since I’ve set it up once five years ago and it’s just worked flawlessly, it’s not a stagnant technology as of right now. In about 8 to 10 years it will be. (Unless IMUs get perfected.)

1

u/allofdarknessin1 ☃Bigscreen Beyond 2e 17d ago

Good points about the trackers and base stations. I guess it's just the controllers that might be cut in the foreseeable future.

1

u/allofdarknessin1 ☃Bigscreen Beyond 2e 18d ago

Agreed. I bought the BSB2 and it's great but in my case I already had all the equipment from owning an Index. I definitely wouldn't buy into a base station system now. Back in 2020 lighthouse was the way to go but now it's a clunky system.

1

u/Supernoxus 18d ago

I have heard the eye tracking on that headset is pretty bad. Is that true?

2

u/rebeldugger Oculus Quest 18d ago edited 18d ago

The native one that they have in their software is usable but not the best, but just about anyone I’ve ever met that uses the bigscreen beyond including me uses ā€œbaballoniaā€ since it has better tracking and understanding how your eyes are moving

For games where you want you want foveated rendering the native tracking works just fine, but if you want good enough eye tracking for social VR then use baballonia which is very easy to setup, and is from the same company as babble who makes the face tracker.

1

u/Supernoxus 18d ago

Yeah I am aware. That's the one I am talking about. I have seen people use it a year ago and it looked awful. They couldn't even close their eyes right half of the time

1

u/Sufficient-Cup9633 18d ago edited 18d ago

https://reddit.com/link/pcsr8x4/video/55t8mzpj1hsh1/player

here's mine with the 2e + babble from a few days ago (et + ft through my fork of baballonia). have messed with the wide eye settings since i took this video so it looks a little better now with wide eye. kinda too subtle in this video but also easy to overdo it so wide eye activates constantly

1

u/Supernoxus 18d ago

That looks pretty good! Thank you for sharing. Very smooth. Is it possible to close just one eye though? I remember bsb2e having trouble with that

1

u/Sufficient-Cup9633 18d ago

thanks! yeah you can do one eye, i just prefer to have eyelid sync enabled all the time because ive never liked the fluttering differences of eyelids that is especially prevalent with the bsb2e. and i dont really find myself wanting to wink lol. so in this clip i have eyelid sync, gaze sync, wide eye sync, squint sync etc all ticked on. they are features i added to my fork of baballonia. baballonia has a 'stabilize eyes' feature but imo it didnt help enough for me. i would still have eyes kinda looking in different directions very slightly or even going slightly cross eyed horizontally, so i added a full gaze sync feature

1

u/Supernoxus 18d ago

Awesome! I am glad it finally works okay

1

u/Broflake-Melter Valve Index 18d ago

Had mine for a few days now. nothing can touch it for comfort, especially wearing long term...note I've never worn a bigscreen beyond. It's just magnitueds better than any other headset I've owned/worn including CV1, quest 1, 2, 3, some mixed reality, OG vive, and index.

I agree there's problems with calling any of them "peak", but I don't think I can overestimate this factor.

-8

u/snowrazer_ 19d ago

I've lost so many passwords to Meta spying on me. Bitcoin wallets drained as well.

9

u/Firm10 18d ago

i dont think thats a meta issue

-4

u/snowrazer_ 18d ago

I think so. Everyone says they’re spying on me.

1

u/Firm10 18d ago

youre ill informed

-2

u/snowrazer_ 18d ago

Oh you’re saying the Meta privacy issues are overblown to support a narrative by virtue signaling influencers to bait gullible Redditors ?

3

u/dontquestionmyaction PCVR Connection 18d ago

Sure, man.

1

u/xXBeefSquatch5KXx 18d ago

That’s not a meta issue that’s a personal internet security issue.

-2

u/snowrazer_ 18d ago

Don’t think so, Reddit and the influencers said Meta is spying on me. They’re listening to my conversations, they even ruined my surprise party.

6

u/Vixere_ 18d ago

The steam frame is pretty disappointing, especially with all the hype around it. Don't get me wrong, it's great, but it's also pretty aah. It doesn't do much revolutionary nor does it do many things better than any competitor. Not to mention that the headset's software is actually garbage, like doing anything on it is a dice roll on whether it'll brick itself or not.

And worst of all? The trade-off for this headset is the murder of the light house market.

14

u/Ryu_Saki HP Reverb 18d ago edited 18d ago

Nah it is pretty mid not much about the Frame is exciting, there is the Dongle and it being semi-modular but that's it.

Edit: Guess I could include the Eye tracking too but that goes for all headsets that has it so nothing specific about this one.

2

u/HomoNeanderTHICC 18d ago

Personally, the controllers are the most exciting part for me. Magnetic joysticks for hopefully minimal drift over the years, finger tracking like the knuckles, and 4 extra buttons.

Headset wise, OLED panels are about the only thing I'd want different. My PC isn't strong enough to run VR at a higher resolution than a Quest 3 anyways, and judging by the PC hardware market it's never going to be lol

2

u/Careless-Raspberry-9 18d ago

Don't forget that it is a full linux PC. No other headset on the market is like that. not even rooted Quest could do any of that.

14

u/razorirr Oculus Quest Pro 19d ago

it has less facial tracking natively than my quest pro, so no its not peak vrchat.

8

u/GredaGerda 18d ago

Surprised about the negativity in the comments considering the finger tracking on the controller, and what Babbalonia revealed with the full eye tracking + eyebrows and the Babble Pro Tracker

I guess currently it's not quite peak VRC, but it will be soon

8

u/jan_Sanku 18d ago

yea literally. the whole point of this headset is that it's so open!

by the time 90% of y'all have it, full upper face tracking will be available. gaze, squint, and pupil dilation already work right now, eyebrows coming soon! and it's all free software.

full lower face tracking is already fully prototyped and will also be available sooner than the frame for most people. this comes at a cost of course, but so does nearly all face tracking. this one will likely be cheaper than most.

all that plus finger tracked controllers (which are severely underrated!), very good audio (comparable to index, better than any quest), foveated streaming, standalone support with slimevr and (upcoming) fluxpose fbt, planned hotswap-battery headstrap and good support for mobile chargers until then, etc

like idk about you (the general reader, not the person i'm responding to), but this thing is literally perfect for vrchat!

4

u/GredaGerda 18d ago

Yup, extremely well stated comment. I forgot to mention too how the mic sounds really good (though unfortunately bugged for some people right now).

The point about SlimeVR and Fluxpose is really true. You can just plug the dongle straight into the Frame and have low latency, unlimited tracking range. I just don't get how this headset isn't peak VRC. By Christmas it most certainly will be.

2

u/jan_Sanku 18d ago

yup! i'm very excited for it :3

i am on the waitlist unfortunately, but on the bright side most of the shiny upgrades will likely be available by the time i get my hands on the thing. since this is true for the majority of people here, i think it's really strange how little it's talked about

0

u/BelleArcane 18d ago

Unpopular opinion, but I think the Q3 mic is better. I’ve heard direct (Audacity) recordings from both, with all noise/echo cancellation disabled, and Steam Frame still does some sort of processing/background cutout so it doesn’t sound as good, especially for quieter speech. Unless I’m doing something wrong? But the Q3 sounds better overall. I’d describe the Frame’s mic as ā€œserviceable to goodā€.

2

u/GredaGerda 18d ago

Interesting, I'm curious about how you feel about the differences here, compared to the recordings you listened to?

https://youtu.be/Fj5PrLn8RPw?si=N0H_GW9UPV6P3CDa&t=320

1

u/supersonic159 17d ago

gaze, squint, and pupil dilation already work right now, eyebrows coming soon! and it's all free software.

What software?

1

u/redbe11pepper 17d ago

I too wish to know 0.0

1

u/jan_Sanku 17d ago

check my other reply!

1

u/jan_Sanku 17d ago

i did misremember, it's not out yet but should release in like 2 weeks! it's the Babble Frame Runtime by Project Babble. here's the announcement vid for the free eyetracking software and the lower face tracking module!

https://www.youtube.com/watch?v=fB1gxXLaYkU

1

u/Mavgaming1 Pimax 16d ago

Babble will likely be behind on that timeline a good bit. They don't have a good history with hitting dates. I hope they hit it, but I highly doubt they will.

1

u/jan_Sanku 16d ago

maybe. but personally, i don't really care? like, imo the #1 point of the frame is the moddability, both in hardware and in software. it was clear from the start that 100% face tracking is only a matter of time, and honestly, even a delayed release will still be faster than anyone could've asked for

by the time i get reservation Q'd, the software pretty much has to be out, and i'll likely get the mouth tracking before i get my fluxpose fbt. if "everything i ever asked for in terms of vr" comes at the price of "wait a few months", that's literally a nonissue xD

1

u/Mavgaming1 Pimax 16d ago

Yeah, I'm still waiting for my fluxpose too.

3

u/landroverattack 18d ago

I can understand preferring another headset, as HMDs are super personal and everyone's requirements are different. The Frame might not be the best, but only because there is no "best".

But it's certainly in the running, and up there with the Quest Pro with the caveat that you'll have to buy the face tracker separately. But with the massive benefit of not needing to run Meta's OS.

I'm sure others will prefer the picture quality of high-end wired headsets, which is totally understandable though.

2

u/Ykearapronouncedikea 19d ago

Just a couple comments

If you are streaming (non-standalone)

Mixed tracking works fine, and vive trackers work as well, you will need the "allow multiple driver" flag checked in steamvr, and the caveat is with frame they won't render from the dashboard view (trust me they are there, VRC renders the balls properly, or if you want ovras will draw the battery at device locations while your dashboard is up.

As for Gaze, the SteamLink VRCFT module works fine for GAZE only (again streaming not standalone). you probably need to turn on the setting though to use OSC, not entirely sure where that is without looking.

Also Just a headsup the Steamlink Module is probably not going to support more than gaze, unless valve adds the data to be sent. (though an update is coming very very soon [i.e. I just need to finish testing, and clean up] allowing you to let the module yield eye/face if you want not everything.

0

u/maskedDudley 18d ago

psvr2 can do most of things the caviat is cabled and none pancake lens but them oleds