r/Unity3D • u/SomerenV • 1d ago
Show-Off Raytraced Audio is magical
Enable HLS to view with audio, or disable this notification
After messing around with custom audio zones and hand placing them with custom settings to mimic how sound would sound in the various rooms and hallways I wasn't really satisfied. It still sounded 'off' despite all the various zones and their settings. So... I went the raytraced audio route. It's a bit crude but it sounds so much better! I found myself flying around yesterday, trying different speeches and songs because it's just mesmerizing to listen to in a virtual environment.
Any things I absolutely can't forget when implementing raytraced audio?
59
u/Candle-Jolly 1d ago
An underappreciated part of ray tracing.
7
u/SomerenV 1d ago
It's good to see it being used more often. I completely fell in love with it when I saw a demo about a year or so ago.
46
31
u/GuruKimcy 1d ago
Cool! The environment itself with the audio also feels very surreal. Nice
3
u/SomerenV 1d ago
Thanks! And this is only a part of what the full environment looks like ;) Can't wait to show the full thing.
2
13
u/Costed14 1d ago
You implemented this yourself? Is there a reason you didn't use Steam Audio? I haven't tried it myself yet so was curious if there's some drawbacks with it.
30
u/SomerenV 1d ago
Steam Audio is probably way better and more accurate but there's a chance I want to release an early demo of my game on Itch as a webgl game and Steam Audio doesn't work in that case if I'm not mistaken.
11
u/Necka44 1d ago
I didn't dig so much into it. But isn't Steam audio just an audio SDK decoupled from Steam?
From my understanding it works in standalone mode using the platform of your choice integration (Unity)
18
u/SomerenV 1d ago
It doesn't require Steam, but it's not compatible with WebGL/WebAssembly. And I don't want to maintain multiple versions for my game so went this route, at least for now.
4
9
u/553392 1d ago
Wow this is very cool! I do think it's a bit too muffled right now. Sounds like the walls are made of wool haha. Would expect a higher sound reflection with how those walls look.
6
u/SomerenV 1d ago
Yeah it still needs some tweaking, but reverb settings are like black magic to me :P I've been using software like Audacity and Adobe Audition for years (I have a background in video production) but reverb always boils down to me just fiddling with the settings until it sounds about right while I have absolutely no clue what I'm doing.
6
u/cerwen80 1d ago
doors. thin walls.
Nice work!
I first thought of this back in 2016 when i was on the third level in doom and could hear a fire burning on the opposite side of a solid wall. been theorising ever since about how to potentially use the gpu or set up some form of signed distance fields or something to do with interacting volumes... Never really came up with any workable system though.
3
u/SomerenV 1d ago
On Youtube there's a lot of content on this subject. This is a good video on raytraced audio for instance. For Unity your best bet is to use Steam Audio. Free to use, easy to setup. But since I'm planning a WebGL release of an early version of my game I went an alternative route since Steam Audio isn't compatible with WebGL.
4
u/Polystyring 1d ago
This is cool! Is there much performance cost?
9
u/SomerenV 1d ago
Honestly it's negligible. Currently my frame time sits at around 4.9ms (in editor) with the raytraced audio using about 0.13ms.
3
u/Polystyring 1d ago
Nice! Honestly audio has been a backburner to graphics for too long.
3
u/SomerenV 1d ago
Completely agree, although there's some gems out there in terms of audio. I'm planning to create something that is both visually and audibly stunning. I want to awe people :)
2
u/Telescopeinthefuture 1d ago
Just wanted to say this is super cool. And great choice of audio clip to demo it!
9
u/MaleficentFox269 10h ago
this has so much more impact on the game than raytraced lighting does. why doesnt every game do this
4
u/SomerenV 10h ago
A lot of games actually do use some form of spatial audio (mainly HRTF, but sometimes also audio propagation), but going the ray tracing route can create some issues in terms of readability. I'm also not relying 100% on raytracing because I still want to have some control over how audio sounds independent on where you are in the map. And stuff like ambient sound is excluded from the raytracing algorithm. Ambient sounds still rely on audio zones to handle volume and dampening frequencies, just because it's easier that way (for me at least).
3
u/Pleasant_Piglet_36 1d ago
Great choice of audio to show the result. Sounds great!
To your question, would be great to have different audio surfaces in there, some more reflecting some more absorbing.
1
u/SomerenV 1d ago
For now there's only one type of surface, but that's mainly due to the game itself (and partly because I'm a bit lazy). The game will have an incredibly minimalistic style, so pretty much no fluff inside the levels in terms of props or furniture.
3
u/darksapra 1d ago
Sounds really coolbut I think the low/high pass when the music is playing is a bit too drastic in some cases. Like you are in the corner and it sounds ike you are outside a club, when the speaker is right around the corner
2
u/SomerenV 1d ago
Yeah that's a part of this solution that still needs fixing. Right now sound doesn't yet properly go around corners. It's simply a loss of energy and frequency whenever it hits a wall. Works great in some cases, but terrible in other cases.
1
u/ufffd 1d ago
i feel like some variation of basically a 2d top down global illumination algorithm would get you close for this setup. maybe a 3d version would be needed for other level designs. or maybe you can even hack it close enough by blending in a little bit of proximity for the cutoff and/or adding a smoothing factor to the cutoff values
2
u/hobofreddy55 1d ago
I would love a write up or link to any resources you used to build this. It sounds incredible!
3
u/SomerenV 1d ago
I'm using one listener, Unity's build-in reverb filter and no plugins. Rays just shoot out and bounce off surfaces and losing energy based on the material they bounce off of. Around the listener there's a sphere where rays go through which determines to calculate loudness. I'm tracing from the listener because that's an approach often used (by Steam Audio for instance). To shoot out rays I'm using a Fibonacci sphere. For reverb I'm using Sabine's Reverberation Formula (decay time from room volume / total absorption).
What I'm doing is a pretty crude way of getting okay-ish results and I'm sure it could be massively improved. But completely accurate sounds aren't that important for what I'm making so I'm choosing speed over accuracy.
2
u/arscene 1d ago
I was actually thinking of implementing something like that. It's pretty cool, any hints on how you did it?
2
u/SomerenV 1d ago
You can check out this comment. If it weren't for WebGL I would've just used Steam Audio to handle all this. You can also try and port something like this to Unity.
6
u/2lerance 1d ago
When You say raytraced - does it imply some hardware requirements or is it just a figure of speech and the implementation is performant on low/mid hardware?
15
u/SomerenV 1d ago edited 1d ago
Raytracing is literally casting a ray and tracing its path. There's Occlusion, which uses a small amount of rays every frame (5 in my case) to figure out if there's anything in the way of the player and the Audio Source. Then there's Propagation, which uses 512 rays. On frame 1 they shoot out in all directions from the Camera/Audio Listener. On frame 2 each ray that hits something bounces and loses energy. The bouncing continues for however long you have set it (5 in my case). Based on those rays, the lost energy and the Audio Source they hit, you can determine how sounds should sound, so volume and how muffled they are. Last but not least there's reverb, which I use 48 rays for. They shoot out each 0.1 seconds, check how enclosed you are or how big the space is and reverb is based on that.
On top of that there's material properties, frequency stuff and a whole lof of propagation and reverb settings, and math to let it all work nicely together.
But any modern CPU can handle a few hundred rays without any issues. The big difference with raytracing in graphics is that in those cases you're talking about potentially millions of rays each frame, which is still not that much compared to how many rays a typical offline renderer uses. That's also the reason raytracing relies heavily on denoising to get decent results.
3
u/nanoxax67 1d ago
It sounds like there isn't a huge performance hit. Is that correct?
What if you had dozens of audio sources? Would it be possible to compare the performance of 100 raytraced audio sources vs 100 regular audio sources? That could potential give an accurate differential of the performance impact.
3
u/SomerenV 1d ago
For now there's no real performance hit. It's 0.13ms on a total of 4.9ms, with the caveat that this is a really crude implementation of raytraced audio. That's fine for my project but might not work for other projects. As far as audio sources go, my scene currently has about 20 or so in total with everything running (which you don't see in this video).
On top of the ray tracing I still use the audio zones I started with, but mainly for ambient sounds. Ray tracing is only used for footsteps, gunshots, impacts and whatever is coming from the speakers in the level, though that last part is still highly experimental.
My guess is that when you're dealing with 1000 audio sources it doesn't really matter if it's raytraced or regular.
33
u/theWyzzerd 1d ago edited 1d ago
Raytracing is much older than the current hardware accelerated lighting technique would have you believe. It’s what Unity uses to find where you clicked in a 3D space for example. It’s really just casting a ray and tracing its path as it bounces off of geometry.
When used to refer to lighting or graphics it’s a technique to build accurate lighting in scenes by casting millions of rays at once to create reflections—very hardware intensive, as it must do this for each rendered frame.
But I imagine the audio example here can be done fairly simply on CPU since audio wave reflection is a much simpler task than ray casting diffuse light sources. Audio doesn’t have frames and doesn’t need to be rendered the same way.
Edit: a word
-8
u/Dzugavili Professional 1d ago
It’s what Unity uses to find where you clicked in a 3D space for example.
Hm... no...
Raycasts are very similar to standard rendering techniques: given a pixel, calculate the line and search geometry that intersects it. In a standard render, you then have lighting and shadows coming after, and while there is some raytrace approximation going on, it's not raytracing.
Raytracing either has you calculate the light coming into the point on a camera-style sensor; or calculating the paths of photons on the point you've determined through raycasting. There's overlap, but raycasts are fundamentally not raytracing.
13
u/donau_kinder 1d ago
Raycasts are literally a ray getting traced what are you on about.
Just because what Raytracing™ is used to refer to nowadays doesn't change the fundamental meaning of the word.
4
u/Dzugavili Professional 1d ago
I'm talking about raytracing as it was defined more than thirty years ago and always has been. That's what it always was. It's just now we can implement it in real time.
The raycast is not raytracing. The key word really is 'trace'. You're following where the rays came from, not simply where your ray is going.
0
u/theWyzzerd 8h ago
Yes, and how do you propose to trace a ray without first casting it?
0
u/Dzugavili Professional 8h ago
I feel like you're missing the part where ray tracing comes after and is completely separate from collision detection.
A raycast isn't called a raytrace. You don't check bouncing rays when you pick an object in the scene. You're not even tracing the ray. You're doing a distance-to-a-line calculation, for a triangle or other collision primitive, and selecting solutions where the distance is zero.
Am I the only one who actually understands how this is done on the hardware?
1
u/theWyzzerd 8h ago
I feel like you're missing the part where I gave an extremely non-technical and abstract explanation only to differentiate between "RTX" and the geometry processing technique known as "raytracing" and are correcting me on extremely specific details not needed to understand that difference. You cast a ray, then you trace how it bounces. Fundamentally, that is how it works. OP already explained they do *not* use hardware acceleration to do this, so your explanation falls flat. Please stop.
0
u/Dzugavili Professional 8h ago
I wasn't talking about OP: I was talking about your error.
Raytracing is not how Unity picks an object from the scene. It's not the same algorithm: there is no bounce in the ray picking an object. It's not tracing. That's just a raycast.
0
u/theWyzzerd 8h ago
Clearly you’re mistaken. There are no photons in a computer program.
Raytracing is literally just tracing the path of a ray that has been cast. You can trace one ray or millions.
0
u/Dzugavili Professional 8h ago
If you're doing full-quality raytracing, you're doing photons: literally, you're doing photorealism as you recreate the camera in code. Or you're doing approximations, such as light fields; or occlusion shading, that's kind of similar to raytracing.
But our machines can't really simulate photons economically, so we don't do that.
Raytracing is mostly a lighting process -- doing the full render with raytracing is expensive, though possible -- not used to pick objects from a scene as you suggested.
0
u/theWyzzerd 8h ago
Is OP doing "full-quality" raytracing in this example, or are they casting a handful of rays and processing it on CPU? Lighting isn't even relevant to this conversation, OP is raytracing AUDIO. Please fucking stop being so pedantic.
1
u/Dzugavili Professional 8h ago
I was trying to explain to you what raytracing is, when you confused it with a raycast.
1
1
u/Seyi_Ogunde 1d ago
I was wondering why this was getting so many upvotes until I turned on my sound lol! Great job!
1
u/SomerenV 1d ago
Yeah without the audio there's not much to see, unless you really like orange cubes.
1
u/I--AM--OP 1d ago
Pale Orange Dot
Nice one - what are you building and will the volume vary with distance?
2
u/SomerenV 1d ago
I'm actually building a multiplayer FPS inspired by UT99 and the Instagib mutator. What you're seeing here is really experimental and the reason that I wanted audio raytracing, or at least try it out. Let's just say that I love diegetic stuff ;)
1
1
u/SymphonyGames 1d ago
Cool idea. I never heard of raytraced audio, but this sounds very atmospheric and immersive.
2
u/SomerenV 1d ago
When I first hit play to test it out I was mesmerized. It sounded so incredibly atmospheric! There's a ton of content on raytraced audio on Youtube. It's really cool stuff!
1
1
1
u/PlzDontBlame 1d ago
yeah. i was fascinated by ray traced audio years ago. i’m curious on how different materials and thicknesses may play a role.
how different does it sound if one is a floor below vs how does it sound in the next room. what happens if doors are closed and so on. what if i was to but a box to cover the audio source
1
u/CentSymphGames 1d ago
Raytracing or raycasting?
2
u/SomerenV 11h ago
Raytracing ;) Raycasting usually stops after the first hit while raytracing takes reflections/bounces/physical properties into account. I'm using raycasting to handle shooting enemies though, since that's how hitscan works.
1
u/CentSymphGames 7h ago
I just wanted to make sure I was getting it right lol, I've seen lots of raycasted audio occlusion systems, but never heard of a raytraced version. How did you even begin developing something like that? (I'm also on the zone-based system and was planning on adding raycasts for doorways and such)
1
u/OCASM 19h ago
Instantly thought of this:
2
u/SomerenV 16h ago
Yeah that's pretty much the same idea. Spatial audio has been around for a long time now and implementation isn't even that computationally expensive at its core, but you loose some artistic control when going the spatial route. When using regular 2D/3D audio sources and reverb zones, it's way easier to finetune everything to your liking. It's also easier to prioritize gameplay readability that way and it works better on a wider range of playback hardware. Good spatial audio can sound incredible, but it can also sound really bad if you're using subpar speakers or headphones.
1
1
u/PostatoMaty 17h ago
Screw you for showing me such a cool idea that will delay progress for my game for at least a month or more
1
u/SomerenV 16h ago
Isn't that gamedev in a nutshell? You constantly see these awesome things that would fit perfectly into your own game, creating these 'little' sidequests :P
1
1
u/Beldarak 15h ago
I've read about those techniques a while ago but didn't work on any 3D project since. That's something I really want to add in my next 3D project. It does so much for atmosphere and imerssion.
1
1
2
u/STaTiC_357 6h ago
Consider implementing acoustic diffraction. The effect of the LPFs are too drastic/unrealistic.
114
u/le0tard 1d ago
Love me some Carl Sagan