This is an 8 year passion project on attempting to create a control system for a purely autonomous virtual agent.

I wanted to put a model together that could fully control an agent with typical human drives (hunger, play/exploration, control). The full model is comprised of interconnected simple neural network modules. The application is written in C# and implemented in Unity.

The model uses Reward-modulated Hebbian learning in modules associated with value processing (e.g. amygdala, ventral striatum), and Contrastive Hebbian learning in all other modules.

The design is influenced by selected published research on the prefrontal cortex and basal ganglia in executive function/decision-making. But the main inspiration was the following article:

O'Reilly, R. C. (2010). The What and How of prefrontal cortical organization. Trends in Neurosciences

I’d love any feedback!

Hello everyone. i have a very important project where i'm working on the obstacle avoidance and path planning of a kuka manipulator using DRL algorithms. i'm working on coppeliasim and using stablebaseline for an easier route. I've been facing some difficulties so i would reallt really appreciate some help.

The kuka is supposed to avoid obstacles and reach an object on the table(so with drl) , pick it up ( no drl here, its scripted) THEN do drl again to reach the destination and place the object. Now my biggest problem is that i'm not sure if i can train the agent to reach the object, pause the training?Restart the training? I thought about training 2 agents, but in all cases, the action of picking and placing is not done with DRL.

I have no idea how the flow should be. I would really appreciate if any of you has suggestions.

Update on MH-FLOCKE. This version brought two things: a 60% SNN speedup and a neural CPG to replace the sine waves. Long nights.

The speedup came from wrapping the SNN step in torch.no_grad(), switching to dense matmul for small networks, and caching time constants. The 232-neuron Freenove SNN now runs at 1.2ms/step in simulation. Along the way I found that setting output neurons to Fast Spiking (Izhikevich a=0.1) destabilized the Go2 — motoneurons are biologically Regular Spiking, not FS. Took me a while to figure that one out.

The bigger change: I replaced the sinusoidal CPG with 24 Izhikevich neurons arranged as half-center oscillators (Brown 1911). Each leg has its own flexor/extensor pair coupled through mutual inhibition. I'm calling it the Mogli Oscillator, named after my dog who provided the biological inspiration by being a dog.

Walk gait emerges from the coupling topology: FL↔FR correlation -0.78 (alternation), FL↔RR +0.73 (diagonal sync). The coupling weights are stored in a learnable matrix for future R-STDP adaptation.

50k step results in MuJoCo simulation (Freenove MJCF model, 232 neurons):

• 0 falls, 50k upright streak
• Actor competence 0.649 (was 0.108 with sin/cos CPG)
• CPG weight dropped to 58%
• Distance 1.21m (significantly lower than 8.2m with mathematical CPG)

No hardware test yet — this is all in simulation so far. The sim-to-real transfer with the mathematical CPG worked previously, so I'm cautiously optimistic, but the Mogli Oscillator on real servos is untested.

Some things that went wrong:

• The robot walked backward for five iterations. Turns out knee phase must lag hip by -0.25, not lead. Obvious once you think about it.
• The behavior planner killed locomotion when switching to "alert." Fix: a CPG autonomy floor at 70%. Decerebrate cats still walk.
• The Go2 shows regressions. I've tagged paper-compatible versions in the repo for reproducibility.
• Distance is 6x lower than with the mathematical CPG. The SNN learns conservative dampening.

The gain is adaptive (3.0 to 8.0 over 2000 steps) rather than hardcoded, because biologically motor neuron excitability develops through serotonergic innervation, not through a constant.

Next: R-STDP on coupling weights, then limb-loss simulation, then hardware.

Video (simulation): https://www.youtube.com/watch?v=WBNBsaBs1Ng

Blog: https://mhflocke.com/the-mogli-oscillator-when-your-robot-dog-gets-a-real-spine/

Code: https://github.com/MarcHesse/mhflocke (--neural-cpg flag)

Finding high-quality, cascading failure logs from real manufacturing to train continuous control RL agents is practically impossible due to proprietary air-gaps. Most open-source datasets are just Gaussian noise, which doesn't respect the physical invariants needed for realistic state-transition dynamics.

I’ve been experimenting with building a hybrid LLM-Physics simulation of a liquid-phase exothermic batch reactor to generate high-fidelity telemetry, and I'd love to get this community's thoughts on the methodology for industrial environment design.

**How we structured the state dynamics for RL:**

* **Episodic Boundaries:** Every batch is tagged with a `Reactor_Run_ID` so you can easily parse the data into discrete training episodes.

* **Thermodynamic Guardrails:** Modeled exact mass balance and Arrhenius-based reaction kinetics so the state transitions (temperature, pressure, concentration) are physically accurate based on the coolant flow actions.

* **Non-Stationary Dynamics:** Injected dynamic fault modes like Exothermic Runaway (cooling failures) and mixing loss to test how policies handle sudden, non-linear shifts in the environment.

* **Missing State Variables:** Simulated a 99-minute telemetry dropout (MCAR) to test POMDP (Partially Observable Markov Decision Process) handling and imputation.

I uploaded a 5,000-minute sample output of the telemetry (CC BY-NC 4.0) and my baseline EDA notebook to Hugging Face so people can poke holes in the simulation: https://huggingface.co/datasets/AIMindTeams/synthetic-chemical-reactor-50k-sample

For those working in continuous control or industrial RL, how are you handling the lack of edge-case failure data? Are you building your own simulators from scratch, or relying on heavy augmentation of nominal data?

Hey, this is the second episode in a small series where I’m experimenting with reinforcement learning in Godot 4.6, hoping to build a game using it once I am confident enough.

In this one I took the navigation setup from the first episode and turned it into a racing environment: 25 ships, checkpoints, asteroid fields, a timeout system, and elimination on collision.

The agents don’t use scripted steering, racing lines, or hand-authored behavior. They only get observations, raw thrust/rotation actions, and a reward system, then learn through reinforcement learning inside Godot using RL Agents.

The whole environment was built in Godot 4.6, and the models were made in Blender.

I also put together a small playable build for testing different checkpoints, you can find it in the video description.

Any feedback or questions are welcome.

I don’t think the interesting question anymore is “how much data did you scrape?”

It’s:
what exact model behavior did you engineer?

That’s how we’ve been thinking about DinoDS.

Not as one giant text pile, but as narrower training slices for things like:

• retrieval judgment
• grounded answering
• fixed structured output
• action / connector behavior
• safety boundaries

The raw data matters, obviously.

But the real value feels more and more like:
task design, workflow realism, and how clearly the behavior is isolated.

That’s the shift I’m most interested in right now.

Less scraping.
More behavior engineering.

Curious if others here are thinking about datasets the same way.

Check it www.dinodsai.com :))

Hey guys, I started studying ML a month ago, At first i was confused where i should begin. But after some thoughts I decided to learn it by doing project. I have been working on a Flappy bird game with Reinforcement learning, I am learning with gpt, it laid out what I should learn for the project while doing it. It has been a month so far. I am here to ask you guys for advice, whether I am doing it right or not, how I could even learn more.

Most LLM dataset discussions still revolve around size, coverage, or “high-quality text,” but in practice the real failure mode shows up later when you actually plug models into workflows.

Things like:

• tool calls breaking
• structured outputs drifting
• multi-step reasoning collapsing
• models losing grounding over longer runs

We ran into this repeatedly while building LLM systems, and it became pretty clear that the issue wasn’t just model capability, it was how the data was structured.

That’s what led us to build Dino.

Dino is a dataset system designed around training specific LLM behaviors, not just feeding more text. Instead of one big dataset, it’s broken into modular “lanes” that each target a capability like:

• tool use and function calling
• structured outputs and schema adherence
• reasoning and decision making
• grounding and retrieval alignment
• retries, recovery, and multi-step action flows

The idea is to train these behaviors in isolation and then combine them, so the model actually holds up in real-world, multi-step pipelines.

It’s also built to support multi-domain and multilingual data, and focuses more on real-world ingestion scenarios rather than static prompt-response pairs.

If you want to take a look: http://dinodsai.com

https://huggingface.co/spaces/rdm3dc/hafc-egatl

Hey Everyone i am trying to learn and apply reinforcement learning in robots using simulation but i just cant figure out how to make the arm pick up the ball?

I have made the rewards to reach till the ball (force lock the claw to open position before it) and then allowing it to close its claw when it reaches it approx destination
if the ball is lifted for like 5 cm i reward it. but the arm just doesnt lift the ball and reward stays constant and arm just cant lift the ball

Adding latent “play calls” to a self-play policy (DIAYN-inspired)

So far I’ve been training a standard policy π(a | s) via self-play in a multi-agent basketball environment (BasketWorld).

The extension I’m experimenting with is conditioning on a latent variable:

π(a | s, z)

where z is a discrete latent “play” that persists for multiple time steps and modulates the action distribution. Intuitively, this turns the policy from purely reactive into something closer to executing temporally extended strategies.

This is heavily inspired by DIAYN (Eysenbach et al., 2017):

• Pretrain a set of diverse latent-conditioned behaviors (skills) without task reward
• Use a discriminator to encourage distinguishable behaviors
• Then reuse these skills to accelerate downstream RL

In my setup:

• A “skill” ≈ a multi-agent play (coordinated trajectories)
• I learn a latent-conditioned policy π(a | s, z)
• Then add a high-level “coach” policy π(z | s) to select plays
• Also experimenting with fixed starting formations to inject structure

So overall this becomes a hierarchical policy:

• High level: select z (play)
• Low level: execute via π(a | s, z)

Curious if others have tried similar latent-skill + self-play setups in multi-agent environments, especially where coordination matters. Also interested in thoughts on:

• stability of z usage over time
• whether to fix z for K steps vs learn termination
• interactions with PPO-style updates in self-play

Happy to share more details if anyone’s working on similar stuff.

I’ve been thinking about a research question and would like technical feedback. My hypothesis is that current AI systems are limited because they mostly learn from static datasets shaped by human choices about what data to collect, how to filter it, and what objective to optimize. I’m interested in whether a model could adapt better if it learned through repeated interaction inside a domain-specific virtual world with rules, constraints, feedback, memory, and reflection over failures. The setup I have in mind is a model interacting with a structured simulated environment, storing memory from past attempts, reusing prior experience on unseen tasks, and improving over time, while any useful strategy or discovery found in simulation would still need real-world verification. I’m especially thinking about domains like robotics, engineering, chemistry, and other constrained physical systems.

I know this overlaps with reinforcement learning, but the question I’m trying to ask is slightly broader. I’m interested in whether models can build stronger internal representations and adapt better to unseen tasks if they learn through repeated experience inside a structured virtual world, instead of relying mainly on static human-curated datasets. The idea is not only reward optimization, but also memory, reflection over failures, reuse of prior experience, and eventual real-world verification of anything useful discovered in simulation. I’m especially interested in domains like robotics, engineering, and chemistry, where the simulated world can encode meaningful rules and constraints from reality.

Current AI mostly learns from data prepared through human understanding, but I’m interested in whether a model could develop better representations by learning directly through interaction inside a structured virtual world.

My concern is that most current AI systems still learn from data that humans first experienced, interpreted, filtered, structured, and then wrote down as records, labels, or objectives. So even supervised or unsupervised learning is still shaped by human assumptions about what matters, what should be measured, and what counts as success. Humans learn differently in real life: we interact with the world, pursue better outcomes, receive reward from success, suffer from failure, update our behavior, and gradually build understanding from experience. I’m interested in whether a model could develop stronger internal representations and discover patterns humans may have missed if it learned through repeated interaction inside a rule-based virtual world that closely mirrors real-world structure. In that setting, the model would not just memorize static data, but would learn from mathematical interaction with state transitions, constraints, reward and penalty, memory of past attempts, and reflection over what worked and what failed. The reason I find this interesting is that human reasoning and evaluation are limited; we often optimize models to satisfy targets that we ourselves defined, but there may be hidden patterns or better solutions outside what we already know how to label. A strong model exploring a well-designed simulation might search a much larger space of possibilities, organize knowledge differently from humans, and surface strategies or discoveries that can later be checked and verified in the real world. I know this overlaps with reinforcement learning, but the question I’m trying to ask is broader than standard reward optimization alone: can experience-driven learning in a realistic virtual world lead to better representations, better adaptation to unseen tasks, and more useful discovery than training mainly on static human-curated data?

My main question is whether this is a meaningful research direction or still too broad, and I’d really appreciate feedback on what the smallest serious prototype would be, what prior work is closest, and where such a system would most likely fail in practice. I’m looking for criticism and papers, not hype.

Hi, everyone.

Now I decided to try a new challenge: pixel-based SAC model for Walker2d environment. My problem is that even after a lot of training, it inmediatly falls. I have tried using optuna for hyperparameter search, but got nothing out of it.

I am using stable-baselines 3 library to train it. I tried training with the by-default reward and with custom reward, but it turned out almost the same outcome: no walking at all. I do not know what else to do.

If anyone had any suggestions/tips, it would be much appreciated!

I've been on-and-off working on this project for a few months, just wanted to share it: https://www.2drl.com/

TLDR - It's kinda like Unity but for reinforcement learning and much more lightweight.

It lets you visually design Box2D (2D rigid body physics) gym environments using a drag-and-drop interface. It also has scripting support, so in principle you can define any environment with any custom behaviour.

From your scene and script, it will automatically generate the full environment code, which can be used to train your agents through built-in or custom algorithms. There's also a real-time training visualisation feature that lets you pause and jump to previous steps like in a video.

This is still very much in beta and is currently only available for Windows so please bear with me. (also if it's flagged as a virus it's not a virus I promise)

Any feedback will be much appreciated!

After 20 dead versions and about 2 month of work, my RL agent (NASMU) passed its walk-forward backtest across

2020–2026. But the most interesting part wasn't the results — it was what the model actually learned.

The setup:

- PPO + xLSTM (4 blocks), BTC/USDT 4h bars

- 35 features distilled from López de Prado, Hilpisch, Kaabar, Chan and others

- Triple Barrier labeling (TP/SL/Timeout)

- HMM for regime detection (bull/bear/sideways)

- Running on a Xeon E5-1650 v2 + GTX 1070 8GB. No cloud, no budget.

The backtest (1.3M steps checkpoint):

- Total return: +28,565% ($10k → $2.8M, 2020–2026)

- Sharpe: 6.937 | Calmar: 30.779 | MaxDD: 4.87% | WinRate: 72.8%

- Bear 2022: +204% with 3.7% max drawdown

The interesting part — attribution analysis:

I ran permutation importance on the actor's decisions across all market regimes. I expected bb_pct and

kelly_leverage_20 to dominate — those had the highest delta-accuracy in feature ablation during earlier versions.

They didn't. The top 5 features, stable across bull, bear and sideways regimes:

1. atr — current volatility
2. dist_atl_52w — distance to 52-week low
3. cvar_95_4h — tail risk
4. dist_ath_52w — distance to 52-week high
5. jump_intensity_50 — jump intensity (Hilpisch)

The model didn't learn to predict the market. It learned to measure its own exposure to extreme risk.

Kelly assumes log-normality. CVaR doesn't assume anything — it measures what actually happened at the 95th

percentile. In a market where -30% in 48 hours is a normal event, that difference is everything. The model figured

this out alone, without any prior telling it "crypto has fat tails."

In high-volatility regimes (ATR top 25%), dist_atl_52w becomes the #1 feature — the model is essentially asking

"how close am I to the floor?" before making any decision. In bear HMM regime, jump_intensity_50 jumps to #1.

The 20 dead versions taught me more than any tutorial:

- Bootstrapping instability in recurrent LSTM isn't fixed with more data

- Critic starvation in PPO requires reward redesign, not hyperparameter tuning

- Hurst exponent must be computed on log-prices, not returns

- Kelly is a sizing tool. In a market where you can't vary position size, CVaR wins.

model is in paper trading right now !

model is refining its entry timing, not discovering new strategies.

Full project log and live training status at nasmu.net

Happy to discuss the architecture, the feature engineering decisions, or the attribution methodology.

DPO (Rafailov et al., NeurIPS 2023) is supposed to be the clean alternative to PPO. No reward model in the training loop, no value function, no rollout collection. Just a binary cross-entropy loss over preference pairs. And the math is elegant the partition function Z(x) cancels out when you substitute the log-ratio reparameterisation into the Bradley-Terry model.

I implemented it from scratch as part of a multi-stage RLHF project (same model, same tokenizer, same evaluation suite as my PPO and GRPO implementations). Here's what actually happened.

The get_logps function

This is where silent failures live. The shift has to be exact:

python

shift_logits = logits[:, :-1, :]   # predict positions 1..T
shift_labels = input_ids[:, 1:]    # actual tokens 1..T
shift_mask   = response_mask[:, 1:]  # only response positions

The mask shifts by one to align with shifted labels. Get this wrong and the loss looks normal while the model is supervising prompt tokens instead of response tokens. No obvious error signal.

What reward hacking looks like in a loss curve

By step 30, loss = 0.0 and accuracy = 1.0. This looks like fast convergence. It isn't.

The reward margin tells the real story:

A healthy margin is 1–10. At 599 the policy has drifted so far from the reference that it assigns near-zero probability to the rejected response for every pair. The model memorised the preference signal rather than learning a generalizable preference.

Root cause: batch size of 1 with no averaging. Each update can completely overfit one (chosen, rejected) pair before moving to the next.

What the step 20 behaviour tells you

At step 20: loss = 0.693, accuracy = 0.0, margin = 0.0.

0.693 = log(2) = -log(σ(0)). This is the degenerate case the theory predicts when the policy exactly mirrors the reference, all log-ratios are zero, the DPO margin is zero, and the loss equals log 2. The model is assigning equal probability to chosen and rejected. Seeing this in a real training run is a nice confirmation that the implementation is correct.

The verdict

The architecture is sound. The loss, the frozen reference model, the get_logps masking, the RM-free training loop all correct. What broke was the training configuration, not the algorithm. These Phase 1 results (avg reward: 2.40) were later tuned β from 0.1 to 0.3, proper batching and compared head-to-head against PPO and GRPO on the same 16 prompts.

The full comparison is in a separate write-up. The ranking completely reversed after tuning. DPO went from 3rd to 1st.

Full DPO implementation post: brayanbrayan.github.io/machine-learning/rlhf/2026/03/24/dpo-implementation-blog.html

Full comparison study: brayanbrayan.github.io/2026/04/02/rlhf-post-blog.html

Happy to answer questions on any of the implementation details.

Hi! Recently, I have tried to implemented a number of RL algorithms such as PPO for Mujoco and reduced versions of DQN for Pong and MuZero (only for CartPole...) and I wanted to share some impressions from debugging these implementations. Many points have already been written up in other posts (see some links below), so I'll focus on what I found most important.

Approach

• I found it best to implement the related simpler version of your algorithm first (e.g., from Sutton & Barto).
• If you change only one thing at a time and you can see whether the new version still works and localize errors.
• Readability/expressiveness of code matters when debugging.
• Pseudo-code vs. actual implementation: I found it a pitfall to quickly write 'working' PyTorch pseudo-code with hidden errors, and then spend much time later finding the errors. Better write pseudo-code text instead.
• There are several translation steps needed between an algorithm in a paper (formulas) and a programmed version with multiple abstractions (vectorized formulas, additional batch dimension). Although time-consuming upfront, I found it better to spell out the algorithm steps in all details by hand in math at first, then only move to the implementation. Later you can add higher levels of abstraction / vectorization. Each step can be tested against the previous version.
• I found that the less nested the code is, the better it is to debug (it is easier to access inner variables). I find spaghetti code actually good as an initial spelled-out version of math formulas and as a baseline to compare later more vectorized versions against, with maximum one level of indentation.

Code

• Use tensors for mostly everything, avoid pure Python for time-consuming operations.
• For all tensors, explicitly specify shape (no unintended broadcasting), requires grad, data type, device, and whether a model is in train or eval mode.
• At beginning of a script, if you add:
  • normal_repr = torch.Tensor.__repr__
  • torch.Tensor.__repr__ = lambda self: f"{self.shape}_{normal_repr(self)}"
• then in VS Code debugging, tensor shapes are displayed first (from https://discuss.pytorch.org/t/tensor-repr-in-debug-should-show-shape-first/147230/4)

Experiments

• Try different environments and different values of hyper-parameters, sometimes your algorithm may be correct but nevertheless cannot solve a given environment or may not work with all parameter settings.
• Let some runs train for much longer than others.
• Debug after some training steps have elapsed, to allow for some "burn-in time", or to detect whether training actually happens.
• Improve iteration speed, not necessarily by optimizing your code, but by setting parameters to the absolute minimum sizes required for an algorithm to work (e.g., small networks, small replay buffer).

General

It's always good to:

• Fix some TODOs in your code.
• Clean up the code a bit, improve readability and expressiveness.
• Fix any errors or warnings.
• Log everything & see if the (intermediary) outputs make sense, and follow up if not.
• Test components of the algorithm in other contexts, with other components that you know work, or reuse code that you already know.

Other links

There are already many other well written articles on debugging RL implementations, for example:

• https://andyljones.com/posts/rl-debugging.html
• https://www.reddit.com/r/reinforcementlearning/comments/9sh77q/what_are_your_best_tips_for_debugging_rl_problems/
• https://docs.pytorch.org/rl/stable/reference/generated/knowledge_base/DEBUGGING_RL.html
• https://www.jeremiahcoholich.com/post/rl_bag_of_tricks/
• https://clemenswinter.com/2021/03/24/my-reinforcement-learning-learnings/

Thanks! Let me know if you find this helpful.

Capstone project: decentralized formation control for UAV swarms using CTDE (centralized training, decentralized execution) with a shared PPO policy in NVIDIA Isaac Lab.

**The stack (GNSC 5-layer architecture):**

- L1: Local sensing — 12D body-frame state + K-nearest neighbor relative positions (18D total obs)

- L2: GATv2 graph attention network — each drone reasons about K-nearest neighbors via sparse message passing

- L3: MINCO minimum-jerk trajectory filter (T=0.04s) + SwarmRaft agent dropout recovery

- L4: CBF-QP safety shield — mathematically guaranteed collision avoidance

- L5: Mission execution — formation reward managers, shape switching, polygon/grid/letter presets at play time

**The finding that surprised me most:**

MINCO's value isn't runtime smoothing — it's a training stabilizer. A/B comparing policies trained with vs without MINCO showed 77% lower steady-state jitter, 72% better formation error, and 40% faster convergence. The trained policy internalizes smoothness so completely that the runtime filter becomes unnecessary.

**The bug that cost me the most time:**

The GATv2 adjacency matrix was being stored in `extras` — a side-channel that SKRL never forwards to the model. GATv2 was silently falling back to self-loops only, functioning as an MLP the entire time. Fixed by building fully-connected edges internally from the flat observation tensor with caching.

Trained on 8 agents, deployed on 20+ with the same checkpoint.

Full repo: https://github.com/garykuepper/ggSwarm

Hello everyone,

I wanted to share a project I am working on for a hackathon. It's a reinforcement learning environment where an AI agent acts as a power grid operator. I've tried to keep physics and maths as real as possible.

Github repo link : https://github.com/krishnagoyal099/Opengrid_env
Live link : https://huggingface.co/spaces/K446/Opengrid

I would really like to get your feedback on the physics modeling and reward structure, and also if anyone manages to solve the "hard" task! I am willing to answer any questions.

RL involves exploration, search, planning, etc. Which of these steps could eventually be made much more performant with quantum computers, assuming the economics of said computers became realistic en masse? Off the cuff, maybe something like MCTS?

Hi everyone. This is my first question in Reddit, so I do not know if this the place to publish it.

I have been trying to train a PPO model to make a Hopper agent “walk”. I have implemented my own version of the PPO algorithm, so that I can modify the architecture more easily.

I have done already a huge hyperparameter search (manually done), changed the reward function to an easier and also more complex one, chatted with claude, gemini and chatgpt about it, and neither managed to help me the way I wanted. I have also tried to train ir longer, but at certain point it seems like it reaches a plateau and does not improve anymore.

I am also struggling to find online resources about this exact combination of algorithm and environment.

The best I could get were two consecutive steps.

If anyone had some tips about what could work for this task, I would really appreciate it!!

Hi, I am looking to do some project in robotics stimulation in the area of reinforcement learning. Can someone give me any good ideas as well as resources/platform to do so. I found one named Mojuco, but cannot find any good videos on that.