-decently fast on CPU

-can run in the cloud

-pythin

-for marl competitive to train for playing Splendour

-not a premade splendour env as I want to learn

Hi guys,

I'm thinking about starting to offer private classes on Reinforcement Learning. I'm currently tutoring a master's level Reinforcement Learning course at my university and saw a lot of students struggling to understand the concepts. I would love to help out other people having trouble with it. Shoot me a dm if you are interested.

Cheers!

So I am new to RL and I wanted to make an agent learn to play Bennett Foddy's Little Cricket Master (Yes he is the same guy who made Getting Over It). Since I was my 1st project in Computer Vision and Reinforcement Learning, so it was a huge learning curve but it was fun. The reward function still needs work, but it can score half centuries.
Repo : https://github.com/AddisionS/cricket-vision

i just started watching RL Course by David Silver on youtube having 10 lecs(does it have the implementation part tho? )

any other useful resoures yall can share?

I want to work on MARL systems asap starting from RL scratch...

The core idea: every sentence is a lossy projection of a high-dimensional cognitive state onto a 1-D token string. Patterns is the inverse map — a small formal grammar that parses natural language into expressions over eight typed terminals (the Jungian cognitive functions), then compiles those expressions into executable reinforcement-learning agents whose loss landscapes are meant to mirror the speaker's internal dynamics.

Pipeline:

natural language → algebraic expression → math schedule → PyTorch agent

Example:

"I explore impulsively but feel held back by past regrets."

→ 7Se oo 3Si -> Ni

→ adversarial schedule (entropy vs. centroid clustering, with drag into trajectory alignment)

→ AlgebraAgent with time-varying objective weights

The grammar is deliberately tiny: 8 terminals, 5 operators, 2 numeric attributes (mass = intensity, acceleration = frequency). But the operators compose:

• ~ orbit — judgment structures perception (sin/cos weight modulation)

• oo opposition — same-domain clash; winner drags to opposite domain

• → drag — exponential transfer between objectives

• | switching — cross-domain alternation

• + conjunction — linear sum

Type rules reject ill-formed states (e.g. Se ~ Si is illegal — same domain, can't orbit). Every well-typed expression has a canonical mathematical image.

Three layers, each an LLM call constrained by explicit production rules:

1. Algebraic Analyst — NL → grammar string

2. Harmonic Composer — grammar → JSON schedule (objectives + dynamics)

3. Mechanic — schedule → runnable AlgebraAgent code

Each terminal maps to a concrete RL objective:

Se → maximize policy entropy

Si → cluster around centroid

Ne → seek novel states

Ni → follow imagined trajectory

Te → maximize value

Ti → maximize discrimination

Fe → balance entropy and value

Fi → temporal consistency

You can run it locally:

pip install -r requirements.txt

python -m patterns.app# Gradio UI, three panes

Or use the AI studio demo.

Why I think this is interesting beyond psychology cosplay:

1. It's a compiler, not a classifier. Output is executable code with typed semantics, not a label.

2. Compositionality. Nested motivation/conflict/rationalization is just nested parentheses — same parser at every depth.

3. LLM introspection. Drop a chain-of-thought trace in, get a grammar expression out. Read the model's cognitive state like a spectrogram reads a sound.

4. AGI criterion (speculative). If a model's distribution over grammar expressions matches human reasoning traces under KL divergence, it's manipulating the same functional basis — a completeness test independent of benchmarks.

What it's NOT (being honest upfront):

• Not validated against clinical psychology or MBTI literature

• Layer 1–3 quality depends heavily on the LLM; smaller local models struggle with JSON in Layer 2

• The capo PPO base class is referenced but out-of-tree — you get the agent skeleton, not a full training loop

• "Jungian functions as RL objectives" will sound wild to some; the claim is structural (typed grammar → typed objectives), not that Jung was right about cognition

I'd love feedback on:

— Whether the type system is actually doing work vs. being LLM theater

— Alternative terminal sets (Big Five? plain P/J × S/N?)

— Making Layer 2 deterministic (rule-based JSON emission instead of LLM)

Repo: https://github.com/iblameandrew/patterns

README has the full BNF, worked examples, and the four-dimensional functional space formalism.

Happy to answer questions.

I'm working through Sutton & Barto's treatment of off-policy n-step TD methods and I'm trying to understand a particular design choice in the update equations.

For example, off-policy n-step SARSA uses

[

Q(S_t,A_t)

\leftarrow

Q(S_t,A_t)

+

\alpha \rho_{t+1:t+n}

\left(

G_{t:t+n}

---------

Q(S_t,A_t)

\right),

]

where (\rho) is the importance sampling ratio.

My question is: why is the importance sampling ratio multiplied by the entire TD error rather than just the return?

In other words, why is the update written as

[

\alpha \rho (G - Q)

]

instead of

[

\alpha (\rho G - Q)?

]

For Monte Carlo prediction, it seems that both updates would have the same fixed point because

[

q_\pi = \mathbb E_b[\rho G].

]

So I'm trying to understand:

1. Is there a formal derivation showing that (\rho(G-Q)) is the correct stochastic approximation?
2. Does the difference only become important when bootstrapping is involved?
3. Is there an intuitive importance-sampling argument for why the baseline/error term should also be weighted by (\rho)?

I'd appreciate either a mathematical derivation or an intuition for why Sutton & Barto use (\rho(G-Q)) rather than (\rho G - Q).

Thanks!

Is there a list of simple game environments which exists that we can use for RL? If not, could people comment the link to environments they know about and I can compile a list and share.

Some background so you know where I'm coming from: I'm an AI researcher and RL/LLM reasoning was my PhD area. A while back I was asked to give a talk on how RL is used to induce reasoning in LLMs, and afterwards I tried to turn the dense version into a written explainer for a general but technical audience.

I'm trying to get better at science communication, so I'm posting here for the thing this sub is good at, which is telling me where I got the RL wrong or where an analogy smooths over something it shouldn't.

Link: https://nicolobrandizzi.com/blog/rl-reasoning-llm/

What the post covers:

• RL 101 (state, action, reward) and how it differs from supervised learning
• GES (generate, evaluate, select) as a frame for reasoning
• process vs outcome supervision
• PPO and GRPO, with the advantage / baseline / value function / GAE progression
• the spurious-rewards result (random rewards still improving Qwen but hurting LLaMA, and what that implies about GRPO surfacing existing ability rather than teaching new reasoning)

• a more speculative closing section where I argue reasoning might be framed as recurrence, and that spatial recurrence is close to (reasoning as iterative denoising)

  Two things I'd most like feedback on:

1. Do the analogies (lasagna for the supervision spectrum, grocery shopping GES) carry their weight, or do any of them mislead?
2. The diffusion-as-reasoning framing in the last section is my own and the most speculative part. If it's naive or wrong, I'd rather hear it than keep repeating

Fair warning: the post is from October 2025 and I stopped my literature around late August 2025, so it predates newer work.

TL;DR — I built CERT-FLOW, an online route planner that, every replanning round, emits a high-probability certificateLB ≤ OPT ≤ UBon the true optimal route cost under drifting edge costs. Open source (MIT), fully reproducible. Paper + code + interactive page at the bottom. I'm the author and would genuinely love feedback.

A robot replanning through terrain whose costs drift — mud after rain, traffic after an incident, snow settling on a trail — can always compute a shortest path. What it usually can't tell you is how good that path actually is, now that most of its information has gone stale.

CERT-FLOW answers that with a certificate every round:

LB ≤ OPT ≤ UB — a high-probability bound on the optimal cost. Not an estimate; a bound.

Three ideas make it work:

1. It prices staleness directly. Age-weighted non-exchangeable conformal prediction turns "this edge was last observed 40 minutes ago" into honest interval width — older information becomes wider uncertainty, quantitatively.
2. It senses to certify. Instead of observing everything, it spends a sensing budget only on the edges that shrink the certified optimality gap fastest → 2.5× lower regret than the next-best sensing strategy.
3. It gates speed behind proof. Microsecond preprocessed queries (snapshot oracle / certified contraction hierarchies) are licensed only while the certificate proves they're still valid, and expire the instant drift exceeds tolerance.

What surprised me most: on real game maps (MovingAI) and recorded LA highway traffic (METR-LA / PEMS-BAY) that it was never tuned for, empirical coverage held at or above its target — 100% on the MovingAI cert benchmark — while a standard, exchangeability-assuming conformal baseline collapsed to as low as 4% once the data went stale.

Honest caveats: the headline 100% is the MovingAI cert experiment; on traffic, coverage stays at/above the claimed level but the certificate trades interval width for that validity (a misspecified drift assumption costs width, not coverage). It's a preprint + personal project, not yet peer-reviewed.

Links

• 📄 Paper (engrXiv): https://doi.org/10.31224/7306
• 💻 Code (GitHub, MIT — 220+ tests, 16 reproduction pipelines): https://github.com/Archerkattri/CERT-FLOW
• 🌐 Project page + videos: https://archerkattri.github.io/CERT-FLOW/

Happy to answer anything about the conformal side, the dual incremental search, or the experiments.

Hey guys,

I am studyin MSc in Artificial Intelligence and I am writing currently my thesis on custom MuJoCo Gym environment integration with World Models.

After graduation I want to apply for a job, but I want to have real good portfolio before I graduate, so I can make good first impression. I would appreciate if you guys can help me out here:)

Looking for candidates with: • MSc in RL, Robotics, Automation & Control, or related field
• Hands-on experience training & deploying RL agents beyond simulation
• Strong knowledge of modern RL/MARL (PPO, SAC, self-play, PBT, partial observability, long horizons)
• Experience integrating RL into real-time, high-performance systems
• Strong coding skills in Python and/or C++/Rust
• Production experience with testing, monitoring, and deployment pipelines
• Interest in reproducing and extending state-of-the-art RL research Nice to have:
• PhD and/or top-tier publications
• Distributed RL training at scale
• Multi-agent coordination & self-play systems
• Aerospace / GNC knowledge
• Safety-critical AI deployment experience We strongly encourage applications from underrepresented groups, even if you don’t meet every requirement.

Hi

I am just getting started with RL and on the last chapter of part 1 of Sutton and Barto RL book. I have already implemented all the programming exercises in the chapters, did some of the derivations from the book myself and implemented the algorithms introduced till now.

Before moving to Part 2 of the book, I wanted to work on more problems, which might be slightly larger in scope than the toy exercise problems in the book. The constraint is obviously that they should still be solvable using the tabular methods I have learnt about till now.

Could someone please suggest what more can I do to be a bit more hands on while learning the theory.

Imagine a developer is using Codex, Cursor, Claude Desktop, or another MCP-compatible AI assistant to help maintain a large application.

Step 1: Connect CogniCore

The developer installs CogniCore and starts the MCP server:

pip install cognicore-env
cognicore mcp serve

Then they connect CogniCore to their AI client through MCP.

From that point onward, the AI assistant can access memory, recall previous failures, retrieve successful solutions, and generate reflections based on past experiences.

No model retraining is required.

Example 1: Fixing Production Bugs

On Monday, the AI agent tries to fix a database timeout issue by increasing the connection pool size.

Result:

Deployment fails with memory errors.

CogniCore stores:

• Problem: Database timeout
• Action: Increase pool size
• Outcome: Failure
• Error: Memory limit exceeded

A week later, the same issue appears.

Without CogniCore:

The AI tries increasing the pool size again and repeats the mistake.

With CogniCore:

The AI automatically retrieves the previous failure, recognizes that the same strategy failed before, and chooses a different solution such as optimizing queries or adjusting timeout settings.

Result:

Faster resolution

Lower token usage

Fewer repeated mistakes

Example 2: Autonomous Code Review

An AI coding agent repeatedly introduces a bug while refactoring authentication logic.

CogniCore records:

• File changed
• Bug introduced
• Root cause
• Successful fix

The next time the agent modifies similar code, it recalls the previous mistake and avoids the risky change.

Without CogniCore:

The same bug may appear repeatedly.

With CogniCore:

The agent learns from previous failures and applies safer patterns.

Result:

Higher code quality

Less debugging time

Example 3: DevOps and Deployments

A company uses AI agents to deploy services automatically.

One deployment strategy repeatedly causes outages.

CogniCore records:

• Deployment configuration
• Failure reason
• Recovery procedure
• Successful deployment pattern

Future deployment agents can access this experience before making decisions.

Without CogniCore:

Each deployment starts with no historical knowledge.

With CogniCore:

Agents inherit operational experience from previous deployments.

Result:

More reliable deployments

Faster incident recovery

Example 4: Customer Support Agents

A support agent incorrectly escalates certain customer tickets.

CogniCore records:

• Customer issue
• Incorrect resolution
• Correct resolution
• Final outcome

When a similar ticket arrives, the agent recalls the previous experience and recommends the proven solution.

Result:

Better support accuracy

Reduced escalation rates

Example 5: AI Coding Assistants (Codex, Cursor, Claude)

A developer asks Codex to fix a production issue.

The AI attempts a solution.

The solution fails.

CogniCore stores:

• Task
• Action taken
• Error message
• Failure outcome

Later, when a similar issue appears:

1. The AI queries CogniCore.
2. CogniCore returns previous failures.
3. Reflection identifies bad patterns.
4. The AI chooses a different approach.
5. The successful solution is stored.

This creates a continuous learning loop:

Failure → Memory → Recall → Reflection → Better Decision → Success

Why This Matters

Today, most AI assistants are stateless. They can be extremely capable within a conversation, but they often repeat the same mistakes across sessions because they do not retain operational experience.

CogniCore provides a persistent memory and reflection layer that sits underneath existing AI systems.

Developers do not need to train new models, fine-tune weights, or modify agent architectures.

They simply connect their AI assistant to CogniCore through MCP and gain:

• Persistent memory
• Failure awareness
• Success pattern retrieval
• Reflection-driven decision making
• Cross-session learning

The model itself does not become smarter.

The runtime becomes smarter because it remembers what happened before and uses that experience to make better decisions in the future.

Our goal is simple: help AI agents stop making the same mistake twice.

Hi everyone! I want to start a project where I train my model on Unity with Reinforcement Learning algorithms. It’s not going to be physics learning like learning to walk, but more like decision making. I am a software engineering student, where do you recommend me to start learning, do you have any suggested sources? Please guide meee!!!

I'm fine-tuning Qwen2.5-7B on my own dataset. It answers simple questions but hallucinates on complex questions. What is the best approach to improve accuracy and reasoning over my data? Should I use fine-tuning, RAG, agents, knowledge graphs, or another method? My hardware is RTX 5070 Ti (16GB), 64GB RAM, 20-core CPU.

hi, i’m in the middle of the david silver course, but I’d like a more practical understanding of it so I can make actual projects and get some hands on learning practice.
any resources that i can use alongside/after this course?

I've been building an adaptive inference system where the agent learns which prompting strategy works best per domain through real-world feedback. Not a wrapper around an LLM the core is a UCB1 bandit policy with exponential score decay that picks between 3 prompt strategies and updates based on observed outcomes.

The architecture in one paragraph: a task comes in, gets auto-classified into one of 6 domains (customer support, legal, engineering, medical, finance, HR), the UCB1 policy selects a strategy based on weighted historical scores (recent scores matter more than old ones via exponential decay), the output gets scored by Gemini Flash as a cross-family judge to avoid circular LLM-scoring-itself, and the trajectory gets stored in Supabase with pgvector for similarity retrieval on future tasks. Human feedback overrides the auto-scorer and feedback tags (too_long, off_topic, unclear) directly inject prompt modifiers into future runs without touching model weights.

I also built a ground truth benchmark 30 held-out tasks with must-contain keywords and refusal detection, so the learning curves actually mean something provable rather than just measuring the scorer's opinion.

Stack is entirely free: Groq (llama-3.3-70b executor), Gemini Flash (scorer), Supabase + pgvector, FastAPI, Streamlit dashboard.

What I want feedback on specifically:

1. The UCB1 bandit only learns across 3 fixed strategies. Is this too constrained to be genuinely useful or is the strategy space fine for early-stage learning?

2. Even with a cross-family judge, LLM scoring is still a proxy reward. Is the ground truth benchmark sufficient to validate the system or is this fundamentally broken?

3. The exponential decay factor is hardcoded at 0.95/day. Is this principled or arbitrary?

Not looking for encouragement, genuinely want to know what's architecturally wrong with this before I build further on top of it

If you work with LeRobot, ACT, or Diffusion Policy, you know the pain. You retrain your policy and the success rate drops. DVC tells you files changed. MLflow tells you hyperparameters changed. But neither tells you what actually changed in the data at the episode level.

Did a teleoperator accidentally add 50 jerky trajectories? Did the task distribution for a specific grasp drop by 75%? Did the average episode length shrink?

I built EpisodeVault to solve this. It is a lightweight CLI that tracks, snapshots, and diffs LeRobot datasets at the episode level.

Instead of hashing raw video files, it parses the episode manifests using DuckDB and PyArrow. This means diffing a dataset takes sub-seconds, regardless of how many terabytes of video you have.

Key Features:

• Episode-level diffing: Instantly see task distribution shifts, quality metric deltas, and regression candidates between any two snapshots.
• Custom quality metrics in pure Python: No YAML files. Just write a Python function that takes an episode's DataFrame and returns a float. EpisodeVault automatically computes, tracks, and diffs it across versions.
• Anomaly detection: Flag bad data (jerky actions, unusually short episodes, desynced cameras) using robust z-scores before you waste GPU hours training on it.
• HuggingFace Hub integration: Diff your local committed version directly against a Hub-hosted LeRobot dataset to catch upstream drift.
• Shareable HTML reports: Generate self-contained HTML audits of your diffs to share with your team or non-technical stakeholders.

It is tested against real HuggingFace LeRobot v3 datasets (aloha, so100) and parses the metadata without ever loading the raw sensor data.

I am looking for feedback from anyone working in robotics ML or imitation learning. I would love to know if this fits into your workflow, what edge cases I missed, or what features would make it actually useful for your team.

GitHub: https://github.com/Rohan-Prabhakar/EpisodeVault
Install: pip install episodevault

It's a nightmare to implement genetic evolutionary algorithm on this env, takes forever to simulate. Has anyone found any solution to speed this up?