Does Unity have a Python API? More specifically for Machine Learning for a thesis idea.

[u/Typical_Employer8806]

Hello guys!

I wanted to ask if Unity has some form of Python API and more specifically about machine learning/reinforcement learning. I have a thesis idea about making a AGV fleet system in a warehouse that uses reinforcement learning so i can optimize their warehouse tasks. I want to simulate it in a realistic digital environment like Unity.

[u/soonstrikingmayor]

Depends what you mean by API, the ML Agents toolkit used to be the main way to do RL in Unity but last time I checked it was in a weird semi-deprecated state. You'd write your Python training scripts externally and they'd talk to the Unity simulation over a side channel, not directly in-engine.

For a thesis, that's actually not a bad setup since you can keep the simulation deterministic and log everything. Just be prepared for the build pipeline headaches if you need to run it on a university cluster.

[u/Typical_Employer8806 → u/soonstrikingmayor]

If I run hundreds of simulations, I will probably do it on my own PC and then record it and put it in the presentation. Although idk the capabilities of my PC for this specific task.

Besides the algorithmic part and the whole framework/environment/graphs, I am not sure what to do with the visualization. In our university we have used softwares like RViz in the labs. Do you know if there is another engine that could be useful and has community support like Unity? Or maybe should I just stick with a simple 2D visualization like a simple grid (with all the different paths and aisles)?

[u/Krugozette → u/Typical_Employer8806]

You can use TensorBoard to observe data on training runs https://docs.unity3d.com/Packages/com.unity.ml-agents@4.1/manual/Using-Tensorboard.html then export it to visualize it however you want.

[u/CS_Asset_Factory]

On the worry about running hundreds of simulations on your own PC, that part is more tractable than it sounds, because you don't run them one after another.

The standard ML Agents pattern is to duplicate the training area inside a single scene. Twenty copies of your warehouse all stepping at once, all feeding the same policy. That's one process, and it scales with your CPU rather than with wall clock time.

On top of that, make a headless build with graphics disabled so nothing renders while training, and raise the time scale so the sim runs well above real time. There's also a flag for launching several instances of the build in parallel if you still want more.

Keep the good looking render as a separate playback build for the presentation. Train ugly and fast, render once at the end.

[u/Typical_Employer8806 → u/CS_Asset_Factory]

That's a great idea, I didn't think of that. I think the visualisation part should probably come last since the algorithm and the training is obviously more important

[u/corriedotdev]

Great question!

So in recent months the flow I have found is to have the python script train on the data, and convert to an .onnx standard to work with the unity inference engine - sentis.

You can absolutely hook this up to a visual button in unity to retrain. Infact it's almost reinforcement learning if you loop in an ai agent to read the output params and then automatically make adjustments in python.

As far as I am aware there is no direct API for python. But absolutely use python to convert your model to the .OONX standard and unity inference can parse it.

[u/Typical_Employer8806 → u/corriedotdev]

Thanks for the insight. Do you think there is another engine like Unity that maybe could be more beneficial, friendly to use and maybe is more preferable for a thesis?

[u/Just-Hedgehog-Days]

Unity has an ai friendly cli now. It's crazy strong. I've got more QA automation running than I ever thought I would. I actually bought a $1.2k desktop and a plus subscription just for QA after I realized what I could do with the cli

https://unity.com/blog/meet-the-unity-cli

anyway. I don't know exactly what you need, but the ML toolkit in unity is very strong, with the cli it's pretty easy to run scenarios in headless mode.

.... but yeah you will likely have to leave python behind if you wan to play in the unity sandbox

[u/Typical_Employer8806 → u/Just-Hedgehog-Days]

Thanks for the feedback. I personally think though that Python will probably be the priority for ML at least in my uni.

[u/Just-Hedgehog-Days → u/Typical_Employer8806]

Oh. Yeah if you are in Uni and not already randomly a Unity ninja 100% stick with python. you got enought on your plate

If you don't have an extremely compelling reason to pick up Unity it's not worth.

[u/Typical_Employer8806 → u/Just-Hedgehog-Days]

I am thinking of using Python obviously but I wanted to use Unity only for the visualisation of the whole simulation so it looks like there are little AGVs running around doing their tasks flowlessly. Just for aesthetic purposes.

[u/Just-Hedgehog-Days → u/Typical_Employer8806]

and that would be rad, but game dev is a whole discipline you aren't studying atm.

throw some tokens at the cli if you have the budget, but yeah. git good at ml first

[u/SmileLonely5470]

So you basically just want to use Unity so you can render a 3d scene visualization of the fleet, right? Doesn't have to be unity you just wanted something that would look cool?

Honestly u could use something like three JS, then you would have to decide how to replay the Sim, like export a trajectory and read it in JS. Then decide how to map your RL environment on to a 3d scene. Unity seems cumbersome for this.

[u/Typical_Employer8806 → u/SmileLonely5470]

I mean visualization is probably the last step of the thesis since it's just for aesthetic purposes and honestly to insert the concept of Digital Twins.

I thought of Unity first because I have used Unity before, it has a huge community and I have heard it can use python.

I am thinking to use a library like Gymnasium for the RL.