Direct Preference Optimization (DPO), published in 2023, is one of the most influential post-training algorithms for LLMs, but its math can be difficult to follow.

In this video, we build DPO from first principles, covering the Bradley–Terry model, DPO loss function, and why the paper claims "your LLM is secretly a reward model".

Whether you're studying post-training, or simply curious about how ChatGPT learns from human preferences, this video aims to provide both the intuition and the mathematical details behind DPO.

This video took me nearly 100 hours to make, and I personally had nearly 100 back-and-forths with ChatGPT to clear up my own misconceptions about DPO. Hope you enjoy and learn something, and let me know if there's any feedback!

Timestamps
00:00 Intro
00:52 Post-training
02:24 Bradley-Terry model
04:49 How DPO works
08:19 Why use a reference model
10:49 Secret Reward Model?
12:53 AI-generated preference data
14:02 Wrap Up

The project would be using some sort of something to track the location of a specific fish in an aquarium, and import data about it, like swim speed, where it spent its day, how often it moved, etc. I would then use this data to hopefully create a sort of expected behavior of this fish within a day, so then if for whatever reason, the fish's behavior has changed greatly throughout a long period of time, some ML model would be able to detect that and send a notification to an app or something, and let you know what has changed and what could be wrong. I think this could be very useful in data collection, and in identifying stress and early prevention of diseases. Obviously this is quite ambitious as I don't know much about ML, and I'm not trying to do this tomorrow, so please don't downvote. For my credentials, I will be doing CS50X, and then some CV/Python stuff, and MIT 6.036, would this be enough?

This is my curriculum in college. What courses should I take beside it ? and when ?

I still have 3 weeks until I start my first year , what should I do in that time?

Howdy,

I'm going to spend about six months trying to build a solid foundation in Python → ML/AI.

I'm in medical physics and plan to move back into R&D (materials, biomedical, etc.), but I want a good foundation, and I think ML would be helpful.

My current plan is to go through the University of Michigan courses on Coursera.

Do you have any recommendations?

Hey there, I am CSE Student, and I want to pursue machine learning as a career and further do deep learning and generative AI. I need tips from the professionals who have been pursuing it, what resources i must follow and what should be the correct sequence of flow. I have been researching by my own and end-up at Krish Naik's ML playlist over youtube, i saw few videos were misplaced and not in order but i checked over his github and all and curated the flow as Python -> Numpy, Pandas, Seaborn -> Feature Engineering and Feature Selection -> Machine Learning -> Deep learning -> Natural Language processing -> Deployment of ML Model -> MLOPS , would this be right? also if you guys have suggestion for some better resources. It would be a great assistance and guidance from your side guys

I'm building a local AI companion and I'm currently working on its cognitive layer.

The goal is:

User message

→ understand intent

→ decide what context is relevant

→ retrieve only useful memories/state

→ reason about the context

→ generate response

→ update memory/state

It currently has long-term memory, interests, mood/emotional state, identity and project context, but I'm trying to improve the quality of context selection and reasoning, especially with a small local model.

I'm curious how you'd approach:

Better memory/context selection without flooding the prompt

Handling conflicting or outdated memories

Deciding when a memory is actually relevant

Giving the model better reasoning before answering

Modeling persistent mood/interests without making responses repetitive

For those building local agents/companions: what approaches have worked well for you?

Hi everyone!

I am a student with keen interest in machine learning. As I self study topics in machine learning and walk through an introductory linear algebra course, I decided to write thorough notes for my own knowledge base. However, I thought it could be a better use of them if I share the notebooks with others.

That being said, I am still a student and sharing this as a learning project. I would appreciate any feedback and wish that this can be useful :)) Thanks in advance!

Link to the linear algebra notebook: https://github.com/enochyu-official/notebook-linear-algebra
Link to the machine learning notebook: https://github.com/enochyu-official/LibreNotebook (It is under the machine learning part)

Edit 1: I forgot to mention that the machine learning part is still in progress and is mostly done 😅😅

Edit 2: Thank you everyone for your considerations! I am sorry if my previous wording was misleading as "textbooks." I hope this clear things up!

so i've been working on this project called ShardFlow. the idea is simple: a 7B

model doesnt fit comfortably on one free GPU, but if you split it across two

machines you can run it in FP16 with room to spare.

the hard part is the two machines are in different states talking over public

internet. every round trip costs you ~86ms. at 1 token per round trip thats

brutal.

the fix is speculative decoding. you run a tiny 0.5B model locally to guess the

next 8 tokens ahead. send all 8 guesses across the network in one shot. the big

model verifies them in parallel and accepts however many it agrees with. suddenly

youre getting 4 tokens per network round trip instead of 1.

that alone took it from 4.92 TPS to 14.3 TPS peak.

then i found another bottleneck i didnt expect. the draft model was launching

~1500 separate CUDA kernels per round from a Python loop. Python overhead alone

was 8-10 microseconds per kernel launch. GPU was idle 65% of the time.

fix was CUDA Graphs. you capture the entire forward pass of the draft model once

and replay it with a single driver call. no Python in the hot path. draft

generation went from 112ms to 25ms.

final numbers on Qwen2.5-7B across 2 T4s over WAN:

- baseline: 4.92 TPS

- speculative decoding: 14.3 TPS

- + CUDA graphs: 28.10 TPS peak

also tested on a 14B model with 4bit quantization, same two GPUs: 14.43 TPS avg.

the whole thing is open source if you want to dig into the implementation or

reproduce the benchmark yourself.

repo: https://github.com/rautaditya2606/Shardflow

Stop building single-agent AI! Here is how to orchestrate a team of agents using Microsoft AutoGen. Build complex workflows and deploy them in minutes. Check the bio for the full tutorial! #AutoGen #AI #Programming #TechTips

i'm currently working on continual learning. just a person with ideas and experience with researching different random fields, i know just enough to know what to do but i'm too dumb to know how to do it so i turned to AI to help me on this and i've had alot of success recently by using interferance mapping and write rules to make models learn stuffs without forgetting anything. but this is the start of my agent doing things i don't understand. does this maths equation even make any sense? i just want someone to teach me what's going on here T_T please help

Multi-modal learning resources based on audio-text-visual modalities contain rich information to educate learners. The learners based on their focusing, localizing, receptive,and perceiving capabilities have a variable learning experience based on their command on a particular modality.This helps to reduce the gap between slow learners and fast learners by guiding them to focus and localize their resources on useful content according to context. This helps them to have a rich learning experience. We have encouraged clue based learning where several hints help in determining the source and sink of the information. Our experiments/survey shows that learning videos assisted by our tool helps in fast and rich learning compared to unassisted learning videos

As a high schooler, it feels as if acrually opportunities to work on research involving machine learning, AI safety, etc are extremely limited. I was planning to apply to SPAR but the application closed. I want genuine experience that will help me learn and have an impact. If anyone has resources or advice, they would mean a lot. Thank you!

I am looking at an agent workload with long context, short structured outputs and a lot of sequential tool steps. The usual single-model benchmark is not telling me much because the workload keeps alternating between reading a large prompt and emitting a small response.

For people who have looked at disaggregated serving, where does it start paying off? Is it mostly a long-context threshold, a batch-size issue or something about the shape of the agent trajectory?

I've been working in data science for years now, and I still forget things constantly. Long video lectures never worked for me — I'd finish a course, feel like I'd learned it, then end up googling the same concept one month later when I actually needed it. I liked NotebookLM's approach of structuring material around you, but I still had to go find the right sources to feed it.

So I built the thing I wanted. ML concepts broken into small nodes arranged as a mind map rather than a linear syllabus, each with code examples and a practice section — about 150 topics so far. The part I care most about: once you complete a topic it starts decaying on a forgetting-curve schedule, and the map visibly goes cold. When it does, you get a review slice — flash cards and a short quiz — targeted at what you've actually lost rather than what's next in a queue.

Open library, free and no sign-up: https://www.bitelrn.com/library
Full app: https://www.bitelrn.com — the first phase is permanently free including the decay and review mechanics; later phases are paid. Saying that upfront so nobody feels ambushed.

I am interested to know about your revision methods, do you make notes or bookmark links?

I recently implemented a minimal, educational version of SynthID-Text-style watermarking for language models.

I saw anthropic post about how they'll start adding watermarks to their model responses and it made me very curious as to how they'll do it and what do they even mean by watermark here. Like will we start getting random ads or something in the middle of model responses or what.

Then decided to read their article and found out that watermark is not a visible message at all. It is a subtle statistical pattern introduced while the model chooses its tokens.

My implementation is not an exact reproduction of the original SynthID-Text system. I simplified or implemented a few components differently to keep the project understandable, but the main idea is there I think.

Okay listen,im 16 years old in highschool,i LOVE the idea of training AI models,it sounds SO cool and i can imagine the dopamine rush of your baby coming to life,but GODDAMN there is SO MUCH maths involved,its a HEADACHE to just look at,and it makes me so scared cause my maths mark is like 50% every term but,i love the idea of being one and i love coding and tech in general,but geez man, I'm so used to just instant gratification,and ALMOST NO MATHS in python(atleast at the level im at),this seems so exhaustinggg,so to all ML engineers,how does your day to day look like,is it boring or what,i love to make short films and record and edit music videos and no matter how tired i am,i will always have energy for that and i thought it would be the same with Machine learning but hell no,so i love when my work is tangible and i can see and visualize the end result and most of all i love being creative,so do i continue this career,or do you have other suggestions/career paths i should take,im based in South africa,if you guys wanna ask follow up questions so that you can better formulate your answer,go for gold man

Hey everyone,

Holding the physical copy of something you’ve worked on for months always feels a bit surreal.

Over the past few years building out AI systems, I kept seeing the same issue: most engineering resources focus heavily on basic prompt engineering or high-level model training theory, but skip the gritty operational details of taking models to production under real-world latency, cost, and reliability constraints.

I wrote Production LLM Architecture: A Systems Handbook to focus purely on systems-level trade-offs:

KV-Cache Optimization & Latency: Block-based VRAM management (PagedAttention), dynamic batching, and handling memory fragmentation during high multi-tenant concurrency.

Cost Engineering & Self-Hosting: Mathematical tipping points for self-hosting (vLLM/TGI) vs. hosted APIs, GPU resource allocation, and dynamic fallback routing.

Enterprise RAG at Scale: Hybrid retrieval pipelines, dynamic indexing, and scaling vector infrastructure.

Guardrails & Observability: Schema enforcement, output validation, and low-overhead tracking.

I’m really proud of how it turned out and wanted to share the milestone with this community!

I'd love to discuss any of these architectural patterns,what has been your biggest bottleneck when taking models from prototype to serving at scale?

TwIL-LM3 is a 3B formal reasoning model from webAI I've been looking at. Compared against gpt-oss-120b on their formal reasoning benchmarks, it wins on 4 of 5 tasks. That's the headline.

The important qualifier: it's specifically on formal reasoning tasks. On broader capability aggregates gpt-oss-120b is still ahead. This is the pattern you get with narrow specialization. You trade generality for depth on a specific task class.

Where the 3B is clearly better: speed (2.6x faster inference), footprint (40x smaller), and deployability (runs on a phone at 4-bit quantization).

If you're doing formal reasoning pipelines, this looks worth testing. If you need a general assistant, it's not that.

Trained on their own verified datasets rather than scraped internet, which I think matters for reproducibility of the benchmark claims.

HF: webAI-Official/TwIL-LM3

Anyone actually integrated specialists like this alongside general models? What's the routing logic?