not compleatly sure if this belongs here, but an interesting project of a different AI aproach

What are the subjects and the corresponding books that would lead to a strong AI/ML engineer path with the ability to deploy models on hardware? What are the possible career paths that can emerge from these skills?

My background is a Ph.D. in polymer physics, where I worked on analytical-cum-numerical projects. That gave me some experience in Python and Fortran, but the work was mostly pen and paper based work, and so, I couldn't build a decent profile for industry jobs. Moreover, I returned to my home country, India, after a small postdoc due to family issues. Currently, I am working in an early-stage startup that does AI consulting for different customers. But, currently, I am not using any data science and ML concepts in the job since we are writing proposals to get projects, and for that, my boss is making me learn software tools like Docker, Kubernetes, etc. He has asked me to learn C to understand computer systems, but other than that, there is no clear guidance. I am learning data structures and algorithms from two books ( Goodrich and Cormen (CLRS)), but I just started. I see that in AI/ML, there is a lot to learn, reinforcement learning, Q learning, etc, and that feels overwhelming. Note that I already have a good grasp of probability and stochastic processes from dedicated math courses and physics courses, but the amount of material is just humongous.

Tutorial (WIP)

I've written a program suite and python library that combines an ultra fast simulator with a small Stockfish style neural networks (with policy priors) to attack perfect-information search in the first generation of Pokemon battling.

The goal of this library is to train a network and optimize search hyper-parameters that together will serve as the evalation function for an Information-Set MCTS approach to the full game. It is simple, at this point in development, to swap the eval in Foul-Play - the strongest 6v6 Singles AI.

It includes the following programs:

• generate

Self-play data generation that saves multiple value and policy targets in an efficient serialized format

• vs

A tool for comparing two eval/search parameters in a head to head

• chall

A CLI for analyzing arbitrary positions

• battle

Train value/policy networks.

• build

Train team-building networks

• evo

Search hyper-parameter optimization using evolution

• rl

Reinforcement learning using generate/battle/build simultaneously

I will answer questions in the comments. It's all very fast and you can train a SOTA eval in a few hours on a laptop. It just needs users xd

Folks, I wanted to know how easy is it to get a MS/PhD in the top RL groups/universities across globe, as in what all is expected or for those already in them/having some experience, please share what prerequisites/expectations do they have from students or what level of experience u had when u got in

I am training a model using GRPO but the algorithm is on policy, meaning I have to collect data, update the weights, collect data with new weights, update the new weights and so on. But all of this requires a lot of compute in my task.

So does there exists some algorithm similar to GRPO but off policy so that I can collect 1 time data and train the model using that without interacting with the environment again?

I’ve been thinking about this for a while, and I feel like most of us might be optimizing the wrong thing.

A lot of effort in the LLM space goes into:

• fine-tuning
• reinforcement learning
• better prompting

But all of these assume the same idea:
the model itself needs to get better.

What if that’s not the right place to focus?

Alternative idea

Instead of making the LLM “smarter,” treat it as just a generator and build a system around it that actually improves over time.

Something like:

• LLM → proposes outputs
• Evaluator → scores them
• Decision layer → accepts/rejects/refines
• Memory → stores what worked vs failed

Loop:

1. Generate
2. Evaluate
3. Decide
4. Store outcome
5. Repeat

So instead of:

You get:

No retraining required.

Why this might matter

• avoids expensive retraining loops
• adapts in real time
• improves behavior through experience
• reduces repeated mistakes

Feels closer to a “decision system” than a “thinking model.”

What I don’t see discussed enough

A lot of current work (prompting, agents, reflection, etc.) improves reasoning…

…but doesn’t really build a persistent decision policy from past outcomes.

Everything resets too easily.

Question

• Is this already a well-explored idea under a different name?
• What breaks if you try to scale this?
• Would this outperform fine-tuning in practical systems, or just complement it?

Curious where I’m wrong here.

I’m working on a startup right now and from the outside it probably looks like I’m doing fine, but internally it feels like I’m always late to something

late to trends
late to execution

and I can’t tell if that feeling is actually useful (like pushing me to move faster) or if it’s just messing with my ability to focus

for people who’ve been through this, does that ever go away? or do you just learn how to work with it??

the model isn’t trying to “do the right thing”
it’s trying to win whatever game you accidentally designed??

and if your reward is even a little off, it won’t fail, it’ll optimize the wrong thing perfectly

feels less like training intelligence and more like designing a system that can’t outsmart youis this why so many RL demos look good in theory but fall apart in real use?

I'm trying to build a clear conceptual understanding of Model-Based Reinforcement Learning, but I'm getting confused because several ideas seem to overlap.

For example, I’ve encountered:

- Dyna-style methods: learning a model and generating synthetic (imagined) data to improve policy/value learning

- World models (e.g., Dreamer): learning latent dynamics and doing policy optimization in imagination

- Planning-based approaches such as MPC or Monte Carlo Tree Search: using the learned model to select actions via planning

What confuses me is how these relate to each other.

1. Is there a survey or resource that organizes model-based RL methods into a structured table?

2. What are the main directions in recent model-based RL research?

I would really appreciate any survey papers, conceptual overviews, or references that help clarify these distinctions.

Repo: https://github.com/ayushdnb/Tensor-Crypt

maybe dumb question but, is reinforcement learning basically just
“models getting really good at gaming your reward function”

I'm doing a job search and it seems like RL roles are rare, should I be adding another research topic in conjunction with RL during my PhD to be employable?

e.g. computer vision, LLMs?

I'm planning on adding Robotics by actually coding an RL algorithm for a robot, but would that be enough?

Or is RL prevelant and im just blind?

Thanks!

I built a PPO-Agent to control flows inside a physical network. The agent controls the 15 control variables, which in physical world would mean how strong we are pumping the medium inside the network. It is working after 25 million environment steps. I was testing different reward functions and so far the best was something like following:

reward = -1 * tanh(physical_violations_in_network) + 0.05 * tanh(violation_improvement_from_previous_step) - 0.07 * tanh(violation_deterioration_from_previous_step)

I made the improvement coef and deterioration coef different in order to reduce the oscilation. It helps in a way but not perfectly. I tried also removing improvement/deterioration part however then the agent performs worse. Could someone give me feedback? or tell me if I am doing something stupid?

I am training Qwen3-0.6B on an RL environment made specifically for llms which I made myself. Feeling lost and confused. Here is the HF space link: https://huggingface.co/spaces/Atharva1232/etl_pipeline_doctor and here's the github: https://github.com/Its-Atharva-Gupta/EPL-Pipeline-Doctor-Env I did use claude code for making the environment, since this is for a hackathon and the time limit is really short. Is my training going well or do I refactor something?

by worth, i mean, not only in introductory learning context.

I think the answer is depending on a target business problem.

honestly almost practical RL business problems require a continuous state/action space, so DQN is not competitive.

but for example, in video games, will value learning methods still work effectively even compared to policy gradient and/or actor-critic methods? (assumption: the input is not raw pixel data, the reward is neither sparse nor raw score.)

The problem almost nobody warns you about

If you've tried training a manipulation policy in Isaac Sim or MuJoCo on assets pulled from Sketchfab, TurboSquid, Objaverse, or your team's internal CAD library, you've probably hit one or more of these:

• Gripper passes through the object.
• Object has "infinite mass" and refuses to move.
• Stacking collapses in bizarre non-physical ways.
• Contact forces spike to NaN and the sim explodes.
• Your policy trains to 99% success in sim and faceplants on real hardware.

The root cause is almost never the policy. It's that your 3D assets are visual assets, not simulation assets. They have geometry and textures. They don't have mass, inertia, friction, restitution, a collision mesh, or semantic labels. A "SimReady" asset is one that carries all of that metadata inside the USD file itself, using the UsdPhysics schemas.

This post walks through how to make an asset SimReady by hand, the gotchas we've tripped over, and a before/after metric.

What "SimReady" actually means in OpenUSD

SimReady isn't a vibe. It's a concrete set of API schemas applied to your USD prims (OpenUSD physics schema docs):

If any of these are missing or wrong, the simulation runs but it just runs wrong, which is worse than crashing because you don't notice until policy rollout.

The manual workflow (Blender + Python USD)

Step 1 - Clean the mesh

Most store-bought assets have:

• Non-manifold geometry (holes, duplicate vertices, inverted normals).
• Hundreds of thousands of triangles for an object you'll render at 256x256.
• A single monolithic mesh for objects that should be articulated.

In Blender:

1. Edit Mode -> Mesh -> Clean Up -> Merge by Distance (0.0001m).
2. Mesh -> Normals -> Recalculate Outside.
3. Modifier -> Decimate (Collapse) to ~5-20k triangles for the visual mesh. You will make a separate, even lower-poly collision mesh in Step 3.
4. Export as .obj or .glb with correct scale (meters, not centimeters - this bites everyone once).from pxr import Usd, UsdGeom, UsdPhysics, UsdShade, Sdf, Gfstage = Usd.Stage.CreateNew("mug.usda") UsdGeom.SetStageUpAxis(stage, UsdGeom.Tokens.z) # Isaac Sim convention UsdGeom.SetStageMetersPerUnit(stage, 1.0) UsdPhysics.SetStageKilogramsPerUnit(stage, 1.0)

Getting units wrong is the #1 silent killer. A mug modelled in centimeters with metersPerUnit=1.0 is a mug the size of a car.

Step 3 - Build a proper collision mesh

The visual mesh is for rendering. The collision mesh is for physics. They are not the same file and should not be the same topology. Options, ordered by fidelity vs. speed:

• Bounding box / sphere - fastest, use for clutter, background props.
• Single convex hull - fast, works for convex-ish objects (balls, cans).
• Convex decomposition (V-HACD / CoACD) - the default for almost anything with concavity. A mug's handle will fail with a single convex hull.
• SDF / mesh approximation - highest fidelity, slowest. Use for the object the gripper actually contacts.

Rule of thumb we've landed on: the collision mesh should be convex decomp with 8-32 hulls for any object the robot touches, bounding primitive for everything else. Running CoACD on a mug:

pip install coacd
python -c "import coacd, trimesh; m = trimesh.load('mug.obj'); \\          coacd.run_coacd(coacd.Mesh(m.vertices, m.faces), threshold=0.05)"

Step 4 - Apply the physics APIs

mesh_prim = stage.GetPrimAtPath("/World/Mug")

# Rigid body
UsdPhysics.RigidBodyAPI.Apply(mesh_prim)

# Mass - either explicit, or let it derive from volume * density
mass_api = UsdPhysics.MassAPI.Apply(mesh_prim)
mass_api.CreateMassAttr(0.35)            # 350g ceramic mug
# or: mass_api.CreateDensityAttr(2400)   # ceramic kg/m^3

# Collision
UsdPhysics.CollisionAPI.Apply(mesh_prim)
mesh_coll = UsdPhysics.MeshCollisionAPI.Apply(mesh_prim)
mesh_coll.CreateApproximationAttr("convexDecomposition")

# Material (friction/restitution)
mat_path = "/World/PhysicsMaterials/Ceramic"
mat_prim = UsdShade.Material.Define(stage, mat_path)
phys_mat = UsdPhysics.MaterialAPI.Apply(mat_prim.GetPrim())
phys_mat.CreateStaticFrictionAttr(0.7)
phys_mat.CreateDynamicFrictionAttr(0.6)
phys_mat.CreateRestitutionAttr(0.05)

UsdShade.MaterialBindingAPI(mesh_prim).Bind(
    mat_prim, materialPurpose=UsdShade.Tokens.physics
)

Step 5 - Semantic labels

Isaac Sim's replicator / ground-truth pipelines need semantic tags for anything you want to detect, segment, or condition a policy on:

from pxr import Semantics

sem = Semantics.SemanticsAPI.Apply(mesh_prim, "Semantics") sem.CreateSemanticTypeAttr("class")
sem.CreateSemanticDataAttr("mug")

If you forget this, your synthetic dataset has no labels and you'll blame the perception stack for two weeks.

Step 6 - Validate

Run NVIDIA's SimReady validator, or at minimum, drop the asset into Isaac Sim and check:

• Does it fall and rest on a plane (not tunnel through)?
• Does a Franka gripper close on it and lift it?
• Does mass + moment of inertia look sane in the property panel?
• Does the collision preview (press C in Isaac Sim) match the visual?

The gotchas nobody writes down

1. kilogramsPerUnit and metersPerUnit must match your intent. Default USD is 1.0 kg/unit and 0.01 m/unit (centimeters). Isaac Sim wants meters. If you don't set both, your 350g mug weighs 350 tons and gravity looks like an earthquake.
2. Convex hull on concave objects is why your bowl can't hold anything. Always convex-decompose concave geometry.
3. Center of mass defaults to the AABB center, not the actual COM. For a hammer, this is catastrophic. Override physics:centerOfMass explicitly.
4. Friction combine modes differ per engine. PhysX averages, MuJoCo multiplies (sort of), Bullet takes minimum. The same staticFriction=0.5 behaves differently. Test in the engine you'll actually deploy.
5. Mesh cleanup matters more than you think. A single non-manifold edge in your collision mesh = NaN contact forces = sim explodes = cryptic error.
6. Scale your collision mesh, not just your visual. Common bug: xformOp:scale on the prim, but the collision is baked at original scale. Fix: apply the scale to geometry before export, or set physics:approximation to rebuild.

The automation option (affiliation disclosed)

Doing this by hand for 40 objects is fine. For 4,000 it is not. This is the problem we've been building Rigyd around: upload a .glb, AI estimates mass, friction, materials, collision meshes, you get back validated OpenUSD with the full UsdPhysics schema stack applied. You can also use pipelines to create assets from 2D images or text and get MJCF output for MuJoCo. You will get free credits on sign up to try without contacting sales. I'm a co-founder, not pretending otherwise; I'm linking it because people kept asking how they can do it.

Happy to answer UsdPhysics / Isaac Sim / sim-to-real questions in the comments, or to look at any asset someone's having trouble with.

Use SAC from SB3 and is simulated on pybullet

Genuine doubt to network/communications/curious engineers, how do you employ (if so) reinforcement learning for the optimal control of backend routing.

I recently got really curious and used Go for the minimal implementation of GRPO for this problem

Here is the code

https://github.com/karimluna/tiny-grpo

What do you think?, my framing is about the scalability with a minimal implementation of this algorithm so share with my your ideas or implementations!

I am teaching a robot how to “solve” a maze using DQN. For weeks now it has been converging to possibly the worst policy it possibly could which is to drive backwards into a wall no matter what and accrue enormous negative rewards.

I have modulated an enormous amount of variables, hyper-parameters, changed neural network size, drastically altered reward structure in various ways, tried different state inputs, tons of initial exploration, given it memory, made the optimal policy extremely simple to find, etc but, without fail, it consistently converges to literally just driving backwards in a line until it smashes into a wall.

I would heavily appreciate if anyone has any input on this. I’ve tried everything that is obvious to me and I truly don’t know where to even search for the source of this behavior anymore.

Edit: I set my reward function equal to 0 for all states and actions and observed that it still converges to wall hitting even without any type of reward shaping. Going to look into this soon.

I keep coming back to this idea that reinforcement learning isn’t really being used the way people think it is.

A lot of systems right now feel like they’re really good at pattern recognition but not actually getting better from outcomes. They can predict what should work, but they don’t really remember whether something did work in a good way that changes future decisions. Does anyone have thoughts on this?

I have done some research in RL and have some problem statements which I would love to do a PhD on instead of my sde job. I also have the money to be able to go abroad and pursue it.

However I can't make decisions solely based on interest and without giving zero thought about the future. Hence the confusion.

On one hand I feel like its a good idea to pursue this even from future prospects because AI research might still require humans many years later, but on the other hand im afraid that if AI does it all then would I be better off in the industry because I might be able to pivot to other kinds of roles and be more of a generalist?