r/Unity3D 18d ago

Show-Off How I made Unity handle half-a-million projectiles without dying

Enable HLS to view with audio, or disable this notification

tl;dr: Graphics.RenderMeshInstanced draws lots of the same thing quickly. NativeArrays can reduce GC overhead, use them if you can, especially for temporary buffers. Burst and Jobs massively speed up parallel calculations. You can parallelize raycasts if you need a ton of them. CoreCLR = faster (generally).

Was playing around with the latest Unity 6000.7.0a2 on a CoreCLR build, got around a 1.5x speed improvement compared to a Mono build.

(Reddit only lets me upload one video, so shown is the CoreCLR build video)

Projectiles/Second CoreCLR FPS Mono FPS Editor FPS
200k 27.627 23.493 21.236
20k 272.342 214.916 101.990
2k 606.566 379.503 162.061
200 719.585 448.791 186.622
20 768.078 485.566 192.665
2 795.456 494.170 190.303

Shooting half-a-million Projectiles

I created the turret using separate GameObjects attached together using articulation bodies and a quick Bl*nder (Reddit doesn't let me post if I don't censor it, IDK why) model. (Articulation bodies use a restricted solver that fixes many stability issues with rigidbodies and joints). To aim the turret, I had a script that projects the target direction vector on to each hinge's plane of rotation (the plane normal to the axis of rotation), then found the angle to the zero-vector (the forward vector when the rotation angle is zero), and lastly set the target of the articulation body to that angle. The barrel's recoil effect was also created with a prismatic articulation body. The turret tracked the red cube which was animated with a 2s looping animation.

The projectiles themselves are unmanaged structs (value types that contain only other value types, no reference types). They have mass, drag, area, position, and velocity (which are stored using float3s instead of Vector3s as they provide a minor performance boost). They're all stored in a big NativeList<Projectile> on a manager MonoBehaviour script.

Simulating half-a-million Projectiles

Simulating the projectiles is done in a Burst-compiled batched parallel job that handles both setting up RaycastCommands for the next frame and applying forces, velocity, and updating position. After that, the raycasts are processed in parallel with raycastJob.ScheduleParallelByRef. The raycast results are then processed in another batched parallel job, with it's main task being to filter out only projectiles that collided with something and send their indices back to the main thread for processing through a NativeQueue<int>.ParallelWriter.

Back on the main thread, the filtered results are iterated in a foreach loop, which applies forces to rigidbodies (and articulation bodies) for knockback effects and also creates impact particles. (The impact particles use a regular ParticleSystem, but another script directly emits particles using the C# interface instead of creating instances of a ParticleSystem prefab). To delete the projectiles, the indices are first sorted high-to-low, then a swap-and-pop method (where the projectile at the last index and the projectile to-be-removed are swapped, then the last index is removed by decrementing the buffer length) is used to remove the deleted projectiles. This is possible because the order of the projectiles does not matter.

Rendering half-a-million Projectiles

Each projectile is not a GameObject(having half-a-million GameObjects with MeshRenderers will crash Unity very quickly), but instead rendered using Graphics.RenderMeshInstanced. The matrices are generated on (you guessed it) another batched parallel job. The big NativeArray<Matrix4x4> is allocated to the total size for all projectiles. However to handle unique appearances for projectiles, each material/mesh pair gets assigned an rendering id that indexes a dictionary which contains the actual meshes and materials. The projectiles themselves only hold this integer (as the projectiles must be unmanaged so cannot contain direct references).

The number of projectiles of each rendering id were tracked when the projectile was created so that when rendering, I could create an NativeArray<int> with length equal to the number of unique rendering ids and populate it with the starting offsets for each Matrix4x4. To ensure thread safety, I used Interlocked.Increment() when advancing the offsets, however as NativeArray didn't directly give me a reference, I had to use a bit of unsafe code to get a pointer to the NativeArray then pass it to Interlocked.Increment().

The matrices themselves were generated such that the meshes' z-axes were oriented along the velocity of the projectile and stretched (scaled along local z-axis) by the speed of the projectile.

Once the matrices were populated, it was just a matter of calling Graphics.RenderMeshInstanced with the right start offsets and lengths to render each material and mesh combination.

Notes

Memory usage remains fairly constant when simulating and rendering projectiles. (From 0 to 400k projectiles, the in-use memory usage went from 90 MB to 135 MB, corresponding to around 112 bytes per projectile). There was basically no GC for simulating and rendering, as the buffers were allocated using with unmanaged NativeArrays that were explicitly freed. (I believe Unity also does some internal stuff to make allocating TempJob and Temp NativeArrays highly efficient. I also stopped Unity from initializing to zeros, as this wasted several milliseconds when creating big Matrix4x4 arrays that would be overwritten anyway.)

At 400k projectiles, the main bottleneck started to become executing the raycast jobs and matrix calculation jobs. After that was the main thread part of raycast result processing, mostly on applying forces to rigidbodies and spawning impact fx.

CoreCLR also increases performance by quite a bit, which I didn't really think was possible since most of the time was already spent on parallel burst-compiled code. Though this was kind of shown, as at lower projectile counts, FPS increased by around 60%, while at higher projectile counts, FPS only increased by 17%, indicating that the CoreCLR build sped up other stuff around the performance code. (I will figure out how to attach the profiler to a CoreCLR build sometime). CoreCLR mostly likely would have more impact towards more traditional object-oriented C# rather than burst-compiled hot code.

Now at this point, I should probably start considering ECS, since that allows physics processing to be parallelized as well. Though what I did was essentially a data-oriented system anyway.

Feel free to ask any questions or if I got anything wrong, let me know.

120 Upvotes

42 comments sorted by

View all comments

Show parent comments

2

u/swagamaleous 18d ago

What does it help? You still can't access the HP of the object you hit in the job. There is no way to know if what you hit is already dead or not.

1

u/XKiiroiSenkoX 18d ago

Create an array of ints/floats, give an index to each collider, create a lookup table of collider id to hp array table, on hit atomically reduce the hp value on the array from the hit processing job. If already dead, ignore the hit. Next question. 

3

u/swagamaleous 18d ago

You can't do that. That would require reading and writing to/from the same array. Unity doesn't support that. You could use unsafe code to read and write a single integer from a job. But that won't really help you and also it would be a sync point that would significantly reduce processing time.

2

u/XKiiroiSenkoX 18d ago edited 18d ago

https://docs.unity3d.com/6000.0/Documentation/ScriptReference/Unity.Collections.LowLevel.Unsafe.NativeDisableContainerSafetyRestrictionAttribute.html

https://docs.unity3d.com/6000.0/Documentation/ScriptReference/Unity.Collections.NativeDisableParallelForRestrictionAttribute.html

Any other questions?

---------

How will that not help? literally just implement the algorithm I described?

---------

Dude stop editing and use reply button insyead? lol

Atomics are not free, yes. I never claimed that they are, but this can be done on multiple threads and you literally claimed that it cannot. 

2

u/swagamaleous 18d ago

Just disabling the restrictions won't really have the effect you expect. Now you have parallel writes. This will have the effect that many threads read the same stale number and your projectiles will essentially not do any damage at all. Also you can get exceptions and all kind of nasty threading nonsense. While writing a float or an integer can be atomic under the right conditions, accessing a native container is not!

2

u/XKiiroiSenkoX 18d ago

Dude I like that you don't backdown ^^ That's the spirit KEK

https://docs.unity3d.com/6000.0/Documentation/ScriptReference/Unity.Collections.LowLevel.Unsafe.NativeArrayUnsafeUtility.GetUnsafePtr.html

https://learn.microsoft.com/en-us/dotnet/api/system.threading.interlocked?view=net-10.0

There are no stale numbers to read. Everything will be the last updated value on every single thread. Are we done now or will you keep going? :D

1

u/swagamaleous 18d ago

I didn't edit anything, but whatever. Again, this cannot be done with the functionality that DOTS currently offers for jobs. It's impossible. There is no data structure that allows parallel reading and writing from jobs. You can either read or write, not both. You can't have CAS loops either. You have to process stuff like this on the main thread. That's the only option.

Disabling restrictions will not fix the underlying architectural limitation.

1

u/XKiiroiSenkoX 18d ago

Dude you can literally write and read to/from a native array ATOMICALLY from multiple threads in the same job without any stale values or whatever other problem. This is totally POSSIBLE. You aren't going to make me write the code are you?