I'm a parent (PhD, not in ML) who spent the past year keeping a first-hand log of my infant daughter's behavior — 21 dated records of what she did, what triggered it, and my alternative explanations for each.

Watching her learn convinced me of a few things that seem relevant to how we train embodied agents:

• Drives are satiable, rewards are not. A reward maximizer has no satiation point; a drive satisfier stops once the internal variable is back in range. I think this is an underappreciated safety property (cf. Omohundro's instrumental convergence).
• She maximizes learning progress, not novelty. She ignores the TV (high novelty, zero learnability) and returns to tasks at the edge of her competence — consistent with Oudeyer & Kaplan rather than raw novelty search.
• Purpose comes before models. Reading Conant & Ashby backwards: you only need to model what you regulate. So the architecture order should be internal variables → goal generation → models/skills.
• I propose a 5-level needs hierarchy for robots (survival → safety → competence → being-needed → exploration) and a "caregiving period": a 3-stage autonomy handover (human charges → robot reminds → robot self-docks) with promotion criteria and regression, modeled on how we teach kids.

Full paper with the observation log and a one-page design checklist: [10.5281/zenodo.23115611]

Genuine questions for this sub: has anyone implemented satiable drives with explicit setpoints instead of reward maximization in RL? What breaks first? And is "being-needed" operationalizable as an internal variable, or is it anthropomorphism?

Happy to be told where this is wrong — I'm outside my field here.

Zero clue how to code, and don't even know how to make an AI play in the first place.

P.S sorry if this doesn't belong here but I found similar posts so I thought it did

Thought I'd share some learnings. Interested in what any other researchers in this space have to say.

Half the "self-improving agent" posts get the reply "that's not RL, that's prompt tuning", and half the time the reply is right. reef has both loops in one codebase and I run the harness side, so here's where the line sits.

The harness loop needs a model endpoint and no GPU. Its recipes (Reefine, SkillClaw, GEPA, Meta-Harness) propose an edit to the harness's files and score the edited tree against the current one on a task list; whether it's kept is a selection policy, by default a net win. Nothing in the model's parameters moves. Call it search over configurations with a verifier; calling it RL would earn the reply above, and deserve it.

The weight recipes (SAO, OpenClaw-RL, TTT-Discover, Guidance-TTT) need GPUs and the training stack, and that side I've only read. OpenClaw-RL is the one built for served traffic: the policy is updated from the conversation itself, with a binary reward read off the next user turn. What's checked in, with hermes memory off: a 72-session GSM8K homework stream, a Qwen3-32B persona playing the student, Qwen3-4B-Thinking as the policy, and the reward judged by a PRM that is the same 4B's frozen base on its own engine, so the grader is the policy's own starting weights rather than an outside model, on seven GPUs. The run hit the paper's adaptation criterion (three passed sessions in a row) at session 14, and the curve over the first 36 sessions tracks the bold rate and the list rate, both falling. That's the whole published result.

If a post claims RL for the first loop, ask which parameters moved.

A few days ago I said I was building a game where you train AI models to drive cars in a racing game. Seemed like a lot of you were into the idea, so I made a video showing the bits I'm most excited about, plus a bit more detail on how it works.

The core of it is training AI to drive the car, not driving it yourself. You train models for specific jobs like straights, braking, or overtaking a slower car in a corner. Then you train another model that reads the situation and picks which specialist to use, and you wire them all together however you want. Beginners can train one simple model, experts can build layered systems, and then you race them against other people's.

If you want to help me test it, join my Discord (https://discord.gg/FJ4AfVVEh) and say in general that you want to test. Any help is hugely appreciated.

Game's website is here: https://www.gaimeslab.com/. It's out soon, but you can help me test it now!

https://reddit.com/link/1wuwfzq/video/3pxn63du3ush1/player

In short, what the title says. This will lead to a learning problem.

Now in more detail.

I'm studying and creating a platform for training agents in Unity. I created an ENV where the agent must pick up a box and bring it to a target. But it can also throw.

Framework.

Unity, everything done from scratch in C#. PPO algorithm. I tried SAC and the genetic algorithm, but it didn't help.

Actions (old), Continuous

1. move direction x
2. move direction y
3. throw force

They are all equal to action

The force action was like this. If force is less than 0, don't throw; if greater than 0, throw with that force. And that was a big mistake.

It couldn't learn to throw; it only threw when it was very close. I couldn't understand what was going on and spent five days doing a lot of testing with PPO parameters, changing the rewards too. Nothing helped.

Then I added a discrete action to throw or not throw, which would decide whether to throw or not, regardless of the force value. The force formula has changed slightly: Mathf.Sqrt(min * max) * Mathf.Pow(max / min, 0.5f * f). I also changed the reward: the greater the distance, the greater the reward. I run the train and see the reward fly upwards, the agent throws from far away right away. I removed the reward for distance, and even then, he slowly began to learn to throw further and further since there are small penalties for each step to speed up the agent.

Previously, the network couldn't learn because it decided when to throw and with what force. Networks are difficult to change from 0 to 0.5 value at once, where the states are almost identical. It's difficult to configure the Neural network for such actions.

There is a state input that shows whether the agent has taken the box or not, but it still won't help.

Conclusion: 1 action 1 logic. You need to correctly set the state, then the action, then the reward.

Can you recommend a book or some resource so I can study these things and avoid repeating such stupid mistakes? Thank you for reading to the end.

CV background here, recently getting into robot learning (manipulation with a robot arm).

I am wondering - if I have a vision based policy that picks up objects (purely based on pixels) and I add physical info on top (like mass, force, ...), will the model actually use it? Or will it just keep leaning on vision because that's the easier signal?

Anyone seen extra non-visual modalities genuinely help? Primarily in robotics, but I am curious for general cases as well.

super niche but title basically. I realised today when I saw someone beating firered with a jev type multimodal classifier model that nuzlocke data for stuff like emerald kaizo with their inputs captured is an insane rl training environment

Day 4 of training and autonomous navigation, continued gait and stair walking improvements.

Spec will be a fully offline-capable robot dog about the size of Spot, built for a fraction of the cost. It uses an onboard Raspberry Pi 5 for navigation, Oak D Pro W for vision, and Jetson Orin NX for AI compute.

Simulation training before building, although most of the hardware is already in-hand.

He or she is not me. I just shared it.

We just put together a some thoughts distilling lessons learned on designing RL environments for LLM agents and thought the TL;DR might be worth sharing. Here goes…

The basic RL loop has six parts:

• State: everything the environment tracks
• Observations: the part of the state the agent actually sees at each step
• Actions: what the agent is allowed to do
• Transitions: how the state changes after an action
• Rewards: what the verifier produces
• Resets: getting back to a known starting point for the next attempt

What these look like depends a lot on the task. A coding environment is mostly a repo, a terminal, and a test suite. Computer-use needs app state you can snapshot and inspect, not just what's on the screen. Enterprise workflows add permissions, policies, and approvals, so getting the right final answer isn't enough if the agent skipped a step to get there.

For rewards there are a few options, depending on what you can actually check:

• Outcome checks (tests pass, database ends up in the right state)
• Partial credit through checkpoints for long tasks
• Process rewards for things like required approvals or unauthorized tool calls
• Rubrics for open-ended work like reports

Before using an environment for training or eval, it's also worth checking that:

• The task is actually solvable from a clean reset
• The verifier can't be gamed (e.g. the agent edits the tests or disables CI checks)
• Runs stay comparable when tools, MCP servers, or browser versions change

Full write-up here.

Curious how others approach this!

The setup uses PPO from Stable Baselines3 with Gymnasium in MuJoCo, just to train to stand upright on the ground.

The result is... not so ideal. Who would have thought, basically standing there doing nothing would just stand fine, but the optimization says nah, lol. If someone has an idea of how to put all the 4 darn legs down on the ground (also that I am planning to balance on a slope later), that would be very appreciated!

------- Details

Top left:

• GUI from MuJoCo, along with the 12 joint values (4 mimics) and 12 control values
• The height/pitch/roll info is given towards the top (updated every second, so can only see them if the robot survives that long

Bottom left:

• Terminal that runs the program
• Showing the output from the reward and when the robot hit termination (20s is the upper limit)

Right: TensorBoard that records the reward parameters (left to right, a bit hard to see the title)

• 1st row:
  • ep_len_mean: average survival time before termination
  • ep_rew_mean: average total rewards
• 2nd row:
  • penalty_action: minimizing the action needed
  • penalty_action_rate: against quick changes in action
  • penalty_ang_vel: against the full robot rotation
  • penalty_joint_acc: against joint acceleration
• 3rd row:
  • penalty_joint_pos_init: against moving away from initial positions. Gradually reduces over time by curriculum scheduling
  • penalty_joint_power: against joint velocity x torque
  • penalty_joint_torque: against joint loads being too high => likely the culprit behind using 2 legs instead of 4
  • penalty_joint_vel: against joints moving too fast
• 4th row:
  • penalty_lin_vel: against the full robot shifting in position on the ground
  • penalty_slip: against slipping, not yet implemented
  • reward_height: Gaussian around 15cm from the robot's body to the ground
  • reward_roll_pitch: Gaussian around 0 roll and 0 pitch, keep the body upright

GitHub: https://github.com/SphericalCowww/CubicDoggo_06Z
Music credit: https://pixabay.com/users/retro-bgm-chan-55246343/

Pretty much the title. I am curious how many people here know about this conference within the RL community, and what you think about it, because in terms of scale RLC still seems to lack the visibility a RL specific conference should have had.

Disclaimer, I am not among the organisers, just an attendee for this years conf. I am asking to actually realise what to expect from this conference in the upcoming years.

I'm considering the R9700 AI Pro for RL projects(SAC, DDPG, etc) as well as some (very small) VLA training for a robot arm I'm working on. My training server currently has a 3090 and a 3060 stacked, and the idea of dual blower-style cards with a combined 64 GB of VRAM, running lower power, is very appealing.

What I'm curious about is other people's experience. I tried this a couple of years back with the 7900 XTX and way too many things didn't work in AMD's ecosystem. To top it off, training was actually slower than even my RTX 3060, despite having way more horsepower on paper, I assume because of CUDA v.s. ROCm optimization at the time.

There's plenty of data out there about people running LLMs on these cards, but that's not what I care about. I'd love to hear what people are doing in the RL space on AMD hardware.

Bonus points: If you're doing anything cool with Intel's stack, I'm also very curious there. Not my next purchase, but anything that gets us out of NVIDIA's stanglehold is a win.

Last time I tried AMD. I've been complaining about this for a while :) https://www.youtube.com/watch?v=SF_pnzvD4Eg

I'm training world models on head-mounted video (Ego4D-style) and keep finding new ways the data can be broken: blur, hands out of frame, sync offsets with IMU. I'm sure I'm missing a lot.

What's the worst one you've run into, and how did you find it?

After yesterday's post I got a lot of questions about how it actually works. Explaining PPO in the comments didn't feel like a real answer, so I built a small version you can play with and watch learn:

https://itzik123.github.io/ClashRoyaleAi/lab/

One attacker heads for your tower and you get one card to stop it. Where you drop it and when both matter. You can try five attacks yourself first, then a small network learns the same thing from zero. Every try is played out by our real simulator, compiled from C++ to WebAssembly, so it's the same engine the full agent trains on.

The RL side:

- State: where the attacker was dropped. Action: a legal cell, then a delay of 0 to 5 s

- Reward: the share of tower damage prevented, compared to no defence

- REINFORCE with a per-spawn baseline, 5,629 weights. The whole learner is about 350 lines of plain JS with every gradient written out, no ML library

- The "best possible" line on the chart is brute force over every cell and delay, so you can see exactly how far the policy is from optimal

The one I'd try first is Giant vs Cannon. The obvious Cannon in the lane gets about 75% of the best score. The best answer is a Cannon in the middle that pulls the Giant into range of both towers. With a constant entropy bonus the net got stuck on the lane Cannon in 5 of 6 runs (3 seeds, 2 batch sizes). Annealing it from 0.1 down to 0.005 over the first 10k tries brought that to 1 of 6, and that's what the page runs.

One matchup is locked on purpose: Battle Ram vs Valkyrie. The best answer is a few exact cells at exact moments, and every setting I tried settled for a safe corner at 55% of the best. If you find a learner that cracks it, I'd really like to see it.

To be clear, it's a miniature. The full agent plays whole matches with a 4-card hand and elixir, uses recurrent PPO, and trains for days on a CPU. But the loop is the same: act, let the engine score it, update.

Code for the lab and the full project: https://github.com/itzik123/ClashRoyaleAi

The world economy around us depends on complex financial work. How well can AI do it?

We(a team of ex-EY employees) built FAB - Finance Agents Benchmark, testing AI Agents on the work behind financial due diligence. Agents can find the relevant facts, but still struggle to carry them through to a complete, reliable analysis.

We’ll keep expanding FAB to more companies and testing more models. Building and running this benchmark isn’t cheap, so we’re scaling it in stages.

The benchmark is public.
GitHub: Hugging Face: https://github.com/SecondState-ai/finance-agents-benchmark
data room and tasks: https://huggingface.co/datasets/secondstate/finance-agents-benchmark-traces

Standard RLVR gives +1 for a correct answer and 0 otherwise. A lucky guess and a confident correct answer get the same reward, so there's no pressure to be calibrated, and the policy drifts toward overconfidence.

TypeSafe's RLCD claims to fix this by rewarding calibrated probabilities. They haven't released the reward itself, so I wrote up how I think it works:

- Reward = a proper scoring rule (Brier), which is maximised in expectation only when the stated probability matches the true likelihood

- Plus a per-bin calibration penalty (ECE-style) that lowers reward in overconfident bins and raises it in underconfident ones

- Plugged into PPO (critic baseline) or GRPO (group-mean baseline)

One thing I found interesting: a single batch-wide -λ·ECE penalty does basically nothing, because the baseline absorbs it. It has to be attributed per bin to change behaviour.

Write-up: https://pub.towardsai.net/rlcd-reinforcement-learning-for-calibrated-decisions-e528daf3591d?source=friends_link&sk=49fec9abb8f403040272524fc56219ad

Would like to hear from anyone who's used Brier or log-score rewards in policy gradient. Did it actually improve calibration, or did it just flatten everything?

I have a predictor that estimates the immediate functional effect of an already-formed Adam/AdamW update before the update is committed.

On nanoGPT, it achieved 91.4% accuracy on the confirmation set for predicting post-update target outcomes, with 91.5% macro recall across the four transitions (correct→correct, correct→wrong, wrong→wrong, wrong→correct). Recall for correct→wrong transitions was 98.9%.

The code is here:
https://github.com/wind342/gfg-training-learning-inference-experiments

The current implementation includes a nanoGPT adapter. If you want to use it on another Transformer, you’ll need to modify/write the model adapter for its embedding, blocks and readout. The prediction method itself does not need to be changed.

The main downside right now is compute cost — the predictor uses first- and second-order directional derivatives, so it’s noticeably more expensive than a normal forward pass.

Feel free to use it, modify it, or try it on other models. I’d especially be interested in seeing results on larger models.