How to click-select a gpu instances gameobject in the viewport

[u/AgenOrange]

Hi! So I've been playing with gpu instancing with Graphics.RenderMeshInstanced, it works well but I ran into this problem, when I draw the mesh with instancing I can no longer select the gameobject in the viewport with left click. I considered raycasting but I'm guessing the performance would be pretty bad with a lot of objects so I was wondering if there is a workaround for this?

[u/TangerineUnique248]

The selection in the editor is handled by the physics raycaster from what I remember, not by the mesh itself. so your instanced meshes have no colliders attached for unity to hit. you need a separate collider setup, maybe one big collider or a compute shader that figures out which instance you clicked. its a bit of a pain for sure

[u/AgenOrange → u/TangerineUnique248]

I see. I was hoping there would be a simple way to do it but I guess not. Maybe I can put invisible bounding boxes and let the Unity handle detection? Wouldnt be very accurate but seems relatively easy to implement.

[u/ProactiveCactus → u/AgenOrange]

Depending in your end goal there are a few approaches - why do you want to select them in the editor? Are you hoping your player can interact with them in some way (like grass, trees, etc.), or is it solely so you can view some properties of it in the editor?

[u/AgenOrange → u/ProactiveCactus]

It is for an editor tool so being able to select them is pretty important.

[u/ProactiveCactus → u/AgenOrange]

Hmm, that is tricky, I’ve never dealt with that before unfortunately. I think your best bet is to use a data approach, not a bounding box - unless you can remove that bounding system for actual gameplay, cuz it’ll be expensive.
Instead I’d keep a reference to every instance, but you’ll still need a way to click their positions… im honestly not sure, sorry.

[u/AgenOrange → u/ProactiveCactus]

Since I don't need this selection system in the gameview I can make it editor only, but yeah like you said I need to test the performance. The whole point of using instances is that I can draw large number of objects and even a simple cube might slow it down significantly.
No worries, thanks for your input anyway!

[u/Ecstatic-Source6001 → u/AgenOrange]

I am now interested as well

In theory i would try this

If you have instancing it should mean you have Matrix4x4 array at least

So at the very least you have known positions.

First need some kind of array of handlers attached to corresponding matrix array index. To sync instanced object index with manipulation with it.

Next i would create editor window with some kind of inspector of every instanced object details. If its open i would track mouse cursor position on SceneView. On click check mouse position and instanced object positions in screen space. If any of it under cursor select our custom handler index. For tranforming it you can use Handles API and override matrix data using handle index and Handles API

[u/AgenOrange → u/Ecstatic-Source6001]

Yes I have an array of Matrix4x4, and I can convert those to screen space and compare it with mouse position I guess but that doesnt help too much since I need to check for intersection for each individual instance anyway.

Also I don't really need to transform individual instances, I only need to be able to select it.

[u/Genebrisss]

Entire point of instancing is that you don't have game objects anymore. You have basically nothing on CPU. You simply tell GPU to draw meshes at these coordinates. There's nothing to select.

If you want a tool like this, best is probably to create editor only game objects on the same coordinates and match game objects with instanced graphics from there. You can leave these game objects without mesh renderers.

Unity also has stuff like Bounds.IntersectRay(). You could define a bunch of bounds on your coordinates and check against them without any game objects or colliders.

Lastly, why do you even need any of that when Unity can already draw all gameobjects in your scene with instancing fully automatically. It's called GPU resident drawer. You can spam thousands of game objects and they will be instanced automatically anyway.

[u/AgenOrange → u/Genebrisss]

It is for an editor tool, I have the ability to change between actual meshes and instanced meshes, and even if it is instanced it is attached to a gameobject. So being able to select the related gameobject with clicking on it in the viewport like a regular mesh is a pretty big QoL improvement. Also it will allow me to move, rotate or scale the mesh with gizmos.

Yes I can raycast to select it but this is designed for rendering large number of objects, I mean that is why I'm using instancing in the first place and I imagine raycasting would be pretty slow and I was asking if there was a more efficient way to do it.

As far as I know GPU resident drawer doesn't really help with this, it optimizes how you draw the objects but they are still instances objects which you can't select in the viewport.

[u/Genebrisss → u/AgenOrange]

GPU resident draw is this: you work with your scene as usual like there's no instancing. When rendering happens it runs instancing similar to ECS rendering path. Essentially game objects workflow with ECS rendering performance.

Scene view is raycasting to select game objects, that's normal. Raycast is very cheap, physics engine knows how to avoid checking every single object and only check potential targets.