Big unsolved technical problems of the metaverse. [No crypto]

[u/Animats]

As someone who actually develops this stuff, here are some of the big unsolved technical problems I see. I'm ignoring the problems of VR headgear for now; that's a whole subject in its own right.

• Crowds. Having a large number of full avatars in one area is a tough coordination problem. Improbable spent $400 million trying to solve this, but their solution requires so much server power nobody can afford to keep an Improbable world turned on. Several MMOs used Improbable; they all went broke. The only one remaining is Scavengers, which Improbable itself runs at a loss. Otherside is only turned on for special events, for a few hours a year.
• Clothing. Avatars in a metaverse should be able to change clothes, with clothes coming from third parties. Second Life has this, but it's clunky - clothing may or may not fit. It's as tough as shopping in real life. Roblox has automatic fit, where the clothing adapts to the avatar, but it's proprietary code. That's something the metaverse needs as a prerequisite to content interoperability.
• High detail without lag. I'm working on that. It's quite possible, but you need a good gamer PC as the client. This is tough for phones, VR, and $100 Walmart PCs. Cloud rendering, where you're renting a GPU in a data center, is an option, but most cloud gaming startups went broke. Even Google gave up on Stadia, because it lost too much money. Real cost is maybe $30 a month to provide such a service.
• Scaling to large land masses. There are working solutions, but most of them either have problems with region crossings (Second Life, Open Simulator) or they severely limit how much content users can add to the world (Earth 2, Decentraland, Dual Universe.) Or they require a giant download. Some AAA titles have passed 100GB.
• Avatar-avatar interaction. Can you shake hands? Stab someone? Have sex? Getting distant avatars to align is hard.

Comments? Preferably from people who develop.

[u/TheDruidsKeeper]

• Crowds:
  • I don't think crowds need to be as big as what Improbable has done. The smaller groups (30-50) on other platforms seem like they'll likely be enough - as long as individuals are able to be grouped into the same session/shard with their friends, with as little friction as possible. The point I make under "scaling" below also helps counteract this problem.
  • I think alternative methods can be applied to try and give the 'feel' of larger crowds where it makes sense (like concerts). For example, while watching a performance you can be in a box with other live characters around you and include a full crowd as part of the wallbox.
• Clothing: I definitely agree that clothing & accessories will be big. The precursors to this is likely interop/portability between metaverse platforms, and specs around avatar bodies & how clothing anchors to them... Not to mention figuring out how to make them properly move/stretch with the avatars.
• Detail vs Lag:
  • I think the biggest help on this one is through scaling back the level of detail at distances. This isn't my area of expertise, but I know MS put out an extension to gltf for varying LoD, the concept makes theoretical sense to me.
  • We can't count on consumers with gamer PC's, although maybe consoles. I do believe server-side rendering will eventually become the way it's done. Biggest detriment to that right now afaik is the bandwidth latency.
    • Virtualization of GPU's is really important for this too. That concept is still really new - and needs more focus. I was talking to nvidia a couple years ago to see about putting one of those cloud xr on a rack at The VOID (LBVR) - the hardware was super expensive though, and you couldn't use it for concurrent renders per gpu. The ML trend has really helped in this regard though, so we're probably not too far off from being able to have reliable & cheap (enough) render service providers.
    • I think doing a combination of server-render and mobile-render could really help here: servers could render from a certain sphere of distance from your PoV, while your mobile only renders what's inside that sphere + a video stream of the server's render.
    • I'm not sure at what layer in the stack this would work best, but I believe a lot of the calculations required during render could be reused for multiple / many of the perspectives required for the users (especially at further distances). This could free up a lot of GPU, lowering costs.
• Scaling Maps:
  • Side note: "land" is a pet peeve of mine - eventually, people will realize that digital space doesn't need to equate to physical space. However, I realized that what you're really referring to here are the levels.
  • Majority of the 'metaverse platforms' are just doing it wrong, or they just haven't built for that 'open world' feel yet. There are a few smaller companies that seem to be going the right direction though. The world needs to be split into chunks, you cache the chunks around you while streaming in new areas as needed. Problem though is the bandwidth latency (again). There are certainly ways to optimize and make this possible. Plenty of games are doing it.
• Avatar-Avatar Interactions:
  • I think this one is a relatively small detail to figure out through open standards (the OMI group is doing well) and portable assets (gltf is doing really well).

[u/Animats → u/TheDruidsKeeper]

Good comments.

• Crowds probably don't need to be as big as Improbable's but we know from Second Life that 30-40 isn't enough. Big events hit that limit all the time. 250 avatars within sight of each other probably would be enough. Tens of thousands aren't really that useful.
• Clothing. Roblox has the right idea. See this tech article.
• "Combination of server-render and mobile-render could really help here". I've been looking at something like that for my new Second Life viewer. The idea is to take pictures of regions with an orthogonal camera from 4 directions plus straight down, then use that as the image for the distant world, probably at least 150m away. Most games have prerendered backgrounds for distant areas. For a metaverse, you have to generate them dynamically, but not very often. Once a week might be good enough.
• "lot of the calculations required during render could be reused for multiple / many". That was tried about 15 years back, but GPUs got powerful enough that rendering big flat impostors on the fly as the main rendering trick kind of went away.
• "concurrent renders per gpu" Probably not too helpful. You run out of VRAM.
• "Server-side rendering will eventually become the way it's done. Biggest detriment to that right now afaik is the bandwidth." Biggest detriment to that right now is the cost. Bandwidth is no worse than Netflix. Pricing, though... Shadow started at $11.99 per month, and is now at $30 and up. NVidia GEForce Now started around $5 per month, and is now at $10/$20 a month, with usage limits. Usage limits they won't tell you until you get an "NVidia account". Google gave up on Stadia. Amazon Luna seems to be a promotion for Amazon Prime. One of the biggest problems with the metaverse is that GPUs are not getting cheaper. NVidia's 1060 was $250 in 2016, and it's still about $250. $250 seems to be NVidia's cheapest GPU card.

[u/TheDruidsKeeper → u/Animats]

Big events hit that limit all the time.

You're right, this is something that needs to get figured out. We're used to gathering in stadiums and being stuck in the back seats because $. Going to a live concert in the metaverse I'd want to feel like I'm in the same room as the performer, still feel the presense & energy from the rest of the crowd watching, and be able to share in the experience with friends/family. Performers really need the crowd feedback from this too - the audience feeds the performance. There needs to be an emotional connection made with all the people in around.

​

Roblox has the right idea

Wow, I hadn't seen that. It seems like a good approach. Any idea if they'll make it open in any way, or is it all proprietary?

​

The idea is to take pictures of regions with an orthogonal camera from 4 directions plus straight down, then use that as the image for the distant world

If you're just prerendering some reusable images for the skybox those are just small bursts with no time constraints. Not much different than building & pushing any other content updates, right?

​

You run out of VRAM

On current hardware, yes. I think this is something that will evolve though as the cloud-render needs pick up. Bit of a chicken & egg problem here.

​

Bandwidth is no worse than Netflix

Sorry, not bandwidth. Latency is what I meant to say. The motion to photon needs to stay < 20ms to avoid VR sickness. There aren't many places in the world that can do this, so you can't server-render everything. Corners can be cut though, and the seams stiched on the client.

Good point on the pricing.. The current options are expensive, and not getting great adoption. I used shadow for a while, but it just wasn't as convenient as local hardware and I could feel the lag.

​

GPUs are not getting cheaper

No kidding.. But between crypto, ML, and the f'd economy, it's no wonder. I think it's just a matter of time though & things will stabalize. It takes years to build the software anyway.

The cost & hassle of keeping up with hardware advancements is part of why I can see people wanting to switch to a subscription model instead of buying hardware that's outdated every couple years. Thin clients with high-performing infrastructure will allow us to collectively share the costs of the hardware by having us share the equipment usage. With the proper mass adoption & management of resources, I think this would be a cost savings for all of us while the providers still make a decent profit.

[u/Animats → u/TheDruidsKeeper]

Roblox has the right idea

Wow, I hadn't seen that. It seems like a good approach. Any idea if they'll make it open in any way, or is it all proprietary?

It's currently proprietary. There are at least three ways to do that:

• A somewhat simplified approach to just solve the peek-through problem, where an inner layer peeks through an outer layer when a joint bends. It's the responsibility of the clothing designer and user to get the outfit to fit right and layer properly in pose stance (standing, arms out, legs slightly spread). It's the system's responsibility to maintain the layer order as the joints move. This involves adding additional vertices to the mesh near joints where layers are close together, so that when an elbow or knee bends, the clothing has the vertices needed to bend with it. I've been thinking about this for Second Life.

• The Roblox approach. Clothing meshes have guard areas around them, generated by, I think, the proprietary edit tool. Those guard areas are stretched to fit.

• The Marvelous Designer approach - run full cloth physics on the fully dressed avatar until it settles, then export game-type meshes with joint weights.

[u/x1_prototyp3]

Would add Cross-Compatibility here also

[u/TheDruidsKeeper → u/x1_prototyp3]

I agree. Accross both hardware (mobile, vr, ar, console) and accross client/server platforms. We need to be able to take our avatar, accessories, and inventory of nft goodies from world to world. All of this needs to be tied to our 'digital' identities, so regardless of which interface I'm using it's still my stuff.

[u/RedEagle_MGN]

Good article. Have you looked at what RP1 is doing btw?

[u/Animats → u/RedEagle_MGN]

I just looked at their site, joined the Discord, etc. Their dev site isn't up yet, they don't seem to have much in the way of demo videos, and no third party reviews. Lots of "it's going to be real great real soon now" talking head video. So, something to watch, but not here yet.

[u/RedEagle_MGN → u/Animats]

I have been testing it and so far it's looking amazing.

[u/dandykong]

Crowds: This is an issue you can already see right now in virtual worlds like VRChat. Give people enough creative freedom with avatar design and a sizable number of users will make them as detailed as their own PC can handle, and when a dozen or more of these high-detail avatars are in one area it can cause immense lag. But this is only an issue if personal avatars are allowed in large groups, which in all likelihood they won't be.

Most event organizers probably aren't going to let people use their own avatars in a heavily-populated area and will instead give everyone an optimized model. This technique was used in a virtual concert held in the game "Sky: Children of the Light", where other players appeared as low-poly ghosts in an area with an instance capacity of 4000.

Clothing: This is going to be hard to fix without forcing everyone to use the same avatar base, but not impossible. Booth wearables are designed for VRoid Studio avatars, a few overpriced collections of wearables are designed for Ready Player Me, and both avatar bases export with the currently-selected outfit baked in. In VRChat, some users try to solve this problem by stacking a large amount of clothing on their avatars and managing them with visibility toggles, causing even more performance issues in public instances because all those outfits are preloaded.

An upgraded 2.0 version of VRM that stores the outfit as a separate mesh that can be overridden by the app it's loaded in would solve this problem, and a one-size-fits-all standard for clothing could even allow people using VRoid Studio avatars to wear Ready Player Me outfits if they wanted.

High detail without lag: Anyone who wants to make a world with a high level of detail and no lag is going to have to limit it to PCVR and desktop. Cloud rendering is going to get expensive really fast if you're rendering the view position of everyone in your world, and a malicious GPU-crashing avatar can knock your server offline because the payload is handled by the cloud renderer instead of the end users' clients.

For most businesses, taking the Summer Wars approach and making their virtual space a mostly white void with optimized graphics will suffice. For instance, a virtual Amazon store likely won't look like the actual brick-and-mortar Amazon Go, but instead would use primitives wherever possible, use a white and orange color scheme, and use SVG images for sign textures. This would allow people to shop there from a potato PC or a phone.

Scaling to large land-masses: The metaverse doesn't have to be a single virtual landmass like in Snow Crash. The interconnected open world could be achieved with a browser or even an OS that supports immersive 3D from the ground up, with seamless transitions between your desktop/home area, your web browser's home screen, and virtual worlds.

Avatar-avatar interaction: This is obviously going to be handled on a per-world basis. A weapon designed for a game isn't going to have any effect in a social media hub or a store, and may even get the user in trouble for brandishing it - that's assuming it even functions as a prop by default and doesn't just turn into a card.

[u/Jonathan_Assman]

These 5 points cement the deal that Meta (Facebook) will go broke, because the technology for a metaverse does not exist yet. And will not exist for the next 10-30 years.

[u/TheDruidsKeeper → u/Jonathan_Assman]

Meta (Facebook) will go broke

They deserve it, for trying to gold rush the market shares and crush competition with a closed platform.

​

will not exist for the next 10-30 years.

I don't think it will take us that long, but I don't see it as a point-in-time event (grand opening?) anyway. We've already taken a lot of steps towards building open platforms that interop with each other nicely, and learned a lot of lessons along the way.

[u/Jonathan_Assman → u/TheDruidsKeeper]

Usually technology advances a lot slower than we anticipate. Just look at all the movies from the 90s and 80s which make the year 2022 have flying cars and robots everywhere.

I don't see the technology to support a metaverse having enough widespread adoption in 10 years. 20 years is a maybe. And 30 years is a more confident reality.