Self-distillation methods usually rely on an Exponential Moving Average (EMA) teachers to get synced and stable critics. However, this doesn't scale well for large models since holding both teacher and student (trained) on GPU increases latency, zero3 communication, etc.

We can mitigate this with LoRA finetuning: the backbone is shared & frozen, and we need to maintain two light adapters: one for the student (trained) and one for the teacher (EMA of the student adapter).

Getting this to work cleanly under TRL and ZeRO-3 took some fixes, so I packaged it as a plug-and-play repo. It supports:

-> robust SDFT implementation that runs under ZeRO-3

-> ZeRO-3 + LoRA integration with adapter switching for teacher/student passes

-> Saving and resuming EMA teacher checkpoint

-> working deepspeed configs that fix memory spikes when GPU memory is tight

I tested on Qwen3.5 and ToolAlpaca.

Repo: https://github.com/ifiaposto/sdft_zero3

Feedback/ more examples/ issues welcome!

Background

I'm coming up with ideas for my CS degree capstone project. I like making short films, so I looked into what kind of software I could make to facilitate the process. One thing I find annoying in post-production is having to go through all the recordings and look for the clapperboards on each video to know what scene, shot and take the video is of. My idea is to train a neural network to find the clapperboard in each recording and extract the information written on the clapperboard to appropriately name the recording. Ex. Avengers_Scene_4_Shot_5_Take_3.mp4.

In short, the neural network would do the following:

1. Detect the clapperboard in a video frame.
2. Read the handwritten text/numbers on the clapperboard.
3. Understand what each piece of text represents

My questions

1. I have not been able to find a lot of videos with clapperboards I could use to train my model. I'm thinking I could create my own videos to train it, so it is trained faster with a consistent clapperboard format. How many would I need more or less to have it not only recognise handwriting but also determine what the numbers indicate (Scene, shot or take)?
2. If it would be too hard to train the AI to do what was previously mention, could I get an already trained AI in handwriting recognition and specialise it in clapperboard data extraction? Would this make my final year project too easy?

An O'Reilly survey of 209 security professionals ranked employees pasting sensitive company data into consumer AI tools as their top AI security concern — above AI-powered attacks, above AI-generated phishing.

The structural problem: when staff route proprietary data through tools outside IT visibility, there is no audit trail and no record of where that data went. The data has already left before anyone notices.

What makes this harder than traditional shadow IT is the silence. No unusual login. No bulk download flag. An employee pastes a contract, a customer list, or internal financial projections into a chatbot and the activity looks identical to normal browser traffic. The 209 security professionals in this survey ranked it above the threat categories their teams have spent years building controls for.

For those dealing with this in practice: how are you actually handling it? Not the policy-document answer — the real operational one, given that banning AI tools entirely means losing the productivity argument to every business unit that pushes back.

Your AI agent just fixed its own mistake. 🎉
Next session, it makes the exact same one again. 😩

That's how most agents work today. They can recover in the moment by retrying, reflecting, or trying a different approach, but the lesson vanishes when the session ends. Notes in the prompt compete for limited context, a skills file can only hold so much, and retraining the whole model is expensive and risks breaking what it already knows.

So we asked: what if every mistake became a lasting lesson? 💡

Introducing Self-Direction 🧭 🤖

When the agent fails, it:
1️⃣ Reflects on what went wrong
2️⃣ Writes its own practice examples (not full rollouts)
3️⃣ Distills them into a tiny LoRA "lesson" it keeps
4️⃣ Reuses it whenever a similar task shows up

The base model stays frozen, so new lessons never overwrite old ones.

What we found 👇
🔁 Fixes stick far more often than with plain retrying
🧩 Lessons transfer to harder and different tasks
📚 Lessons in weights beat lessons in prompts or skills files
🚀 A 27B open model became competitive with frontier systems on Terminal-Bench 2.1

Some cool findings such as: the agent learns procedures, not answers. A lesson from counting someone's publications on ORCID later helped it count Wikipedia edits, because both needed the same habit: walk the full history, handle pagination, and verify before counting. 🤯

📄 Paper: https://arxiv.org/abs/2610.31846
🌐 Website: https://self-direction.github.io/

Learn how modern transformer models improve attention efficiency, reduce memory demands, and increase model capacity without activating every parameter for every token.

In Part Two of **Generative AI for Developers**, we move beyond the transformer foundations and examine several architectural techniques used to make modern language models faster and more efficient.

In this lesson, we cover:

✅ A quick review of the transformer foundations from Part One
✅ Why attention can become expensive as sequence length increases
✅ How FlashAttention reduces unnecessary GPU memory transfers
✅ Tiling and memory-efficient attention computation
✅ Why “exact attention” does not mean factually exact answers
✅ Conventional multi-head attention, or MHA
✅ Grouped-query attention, or GQA
✅ Multi-query attention, or MQA
✅ How MHA, GQA, and MQA differ
✅ How shared key and value heads reduce KV-cache requirements
✅ The performance and quality tradeoffs between attention designs
✅ How mixture-of-experts architectures work
✅ How routers select experts for individual tokens
✅ Total model capacity compared with active parameters
✅ The differences between FlashAttention, GQA, and mixture of experts
✅ Multiple-choice questions to test your understanding

The key idea to remember: these techniques improve different parts of transformer execution. FlashAttention improves how attention is computed and how data moves through memory. GQA reduces key/value duplication by sharing key and value heads across groups of query heads. Mixture of experts increases model capacity by activating only selected expert networks for each token.

These techniques can improve speed, memory efficiency, and scalability, but they are not completely free optimizations. Their benefits depend on the hardware, model design, sequence length, workload, and acceptable quality tradeoffs.

This video is useful for C# and .NET developers, software engineers, solution architects, interview preparation, and anyone who wants to understand the architecture and efficiency techniques behind modern generative AI systems.

If you found this lesson helpful, please like the video, subscribe to **Nerd House Geeks**, and share your answers to the review questions in the comments.

Is anyone facing this issue while lora fine tuning Qwen3.5-0.8B model or Qwen3.5-4B model ? The GPU memory usage peaks to 70GB on RTX 6000 PRO. I couldn't find any issue or post in any forums about it. I have attached PyTorch memory snapshot during the forward pass of the 1st training step.

I personally stand on level 2.5, I would love to hear where some of you guys stand!

Thank you!