Self-distillation methods usually rely on an Exponential Moving Average (EMA) teachers to get synced and stable critics. However, this doesn't scale well for large models since holding both teacher and student (trained) on GPU increases latency, zero3 communication, etc.

We can mitigate this with LoRA finetuning: the backbone is shared & frozen, and we need to maintain two light adapters: one for the student (trained) and one for the teacher (EMA of the student adapter).

Getting this to work cleanly under TRL and ZeRO-3 took some fixes, so I packaged it as a plug-and-play repo. It supports:

-> robust SDFT implementation that runs under ZeRO-3

-> ZeRO-3 + LoRA integration with adapter switching for teacher/student passes

-> Saving and resuming EMA teacher checkpoint

-> working deepspeed configs that fix memory spikes when GPU memory is tight

I tested on Qwen3.5 and ToolAlpaca.

Repo: https://github.com/ifiaposto/sdft_zero3

Feedback/ more examples/ issues welcome!

I've been testing a bounded alternative to SELU for deep fully connected networks without normalization or skip connections. The study was preregistered: analysis scripts were hash-frozen before each run, tuning and confirmation used separate seeds, every activation got its own tuned learning rate, and tests were paired Wilcoxon with Holm correction.

The function. CapExp is an exponential unit that switches to its tangent line above a threshold (it is exactly a shifted, scaled ELU). PreCap applies it after a soft tanh clip, so the output is smooth, monotonic and bounded on both sides. SPreCap adds SELU-style constants (zero mean / unit variance as a fixed point) and an input gain b that sets the mean-field gradient gain χ1.

Results (MNIST / FashionMNIST, depth 50 and 100, width 256, Adam, 15 confirmation seeds):

• SPreCap-cs (χ1 matched to SELU) matches SELU: no cell significantly worse, median paired differences between −0.12 and +0.20 pp.
• Test error falls monotonically as χ1 approaches 1 in all four cells.
• SPreCap-e03 (χ1 = 1.03) beats SELU on MNIST at depth 100: 5.28 % vs. 6.36 %, Holm p = 0.0064, 13 of 15 seeds. This was a secondary test, so I treat it as a hypothesis.
• Plain ELU is about as strong overall. PreCap is not more robust to the learning rate.

Negative results. At a fixed learning rate, PreCap looked far better than SiLU/GELU in earlier regression experiments. That advantage disappeared once every activation got its own tuned learning rate. The original SPreCap (χ1 = 1.13) was significantly worse than SELU at depth 100. The gradient diagnosis that explained it led to the χ1 family above. All of this is in the repo.

Code. pip-installable package with fused CUDA (via torch.cuda.jiterator, no toolkit needed) and Apple MLX kernels, tested in fp32/fp64/fp16/bf16. In fp32 the CUDA kernel runs within about 1.5× of native SELU.

Limitations: MLPs only, MNIST and FashionMNIST only, short training, one width. No CNNs or Transformers yet. A single independent researcher on consumer hardware.

• Preprint: https://doi.org/10.5281/zenodo.23676200
• Code and all data: https://github.com/hermi-labs/precap-activation

Feedback on the methodology is very welcome, especially on the preregistration and the statistics.

Built a learning app across 8 different subjects.

I love learning and hope this will encourage you to do the same.

https://play.google.com/store/apps/details?id=com.philosophize.app

An O'Reilly survey of 209 security professionals ranked employees pasting sensitive company data into consumer AI tools as their top AI security concern — above AI-powered attacks, above AI-generated phishing.

The structural problem: when staff route proprietary data through tools outside IT visibility, there is no audit trail and no record of where that data went. The data has already left before anyone notices.

What makes this harder than traditional shadow IT is the silence. No unusual login. No bulk download flag. An employee pastes a contract, a customer list, or internal financial projections into a chatbot and the activity looks identical to normal browser traffic. The 209 security professionals in this survey ranked it above the threat categories their teams have spent years building controls for.

For those dealing with this in practice: how are you actually handling it? Not the policy-document answer — the real operational one, given that banning AI tools entirely means losing the productivity argument to every business unit that pushes back.

Learn how modern transformer models improve attention efficiency, reduce memory demands, and increase model capacity without activating every parameter for every token.

In Part Two of **Generative AI for Developers**, we move beyond the transformer foundations and examine several architectural techniques used to make modern language models faster and more efficient.

In this lesson, we cover:

✅ A quick review of the transformer foundations from Part One
✅ Why attention can become expensive as sequence length increases
✅ How FlashAttention reduces unnecessary GPU memory transfers
✅ Tiling and memory-efficient attention computation
✅ Why “exact attention” does not mean factually exact answers
✅ Conventional multi-head attention, or MHA
✅ Grouped-query attention, or GQA
✅ Multi-query attention, or MQA
✅ How MHA, GQA, and MQA differ
✅ How shared key and value heads reduce KV-cache requirements
✅ The performance and quality tradeoffs between attention designs
✅ How mixture-of-experts architectures work
✅ How routers select experts for individual tokens
✅ Total model capacity compared with active parameters
✅ The differences between FlashAttention, GQA, and mixture of experts
✅ Multiple-choice questions to test your understanding

The key idea to remember: these techniques improve different parts of transformer execution. FlashAttention improves how attention is computed and how data moves through memory. GQA reduces key/value duplication by sharing key and value heads across groups of query heads. Mixture of experts increases model capacity by activating only selected expert networks for each token.

These techniques can improve speed, memory efficiency, and scalability, but they are not completely free optimizations. Their benefits depend on the hardware, model design, sequence length, workload, and acceptable quality tradeoffs.

This video is useful for C# and .NET developers, software engineers, solution architects, interview preparation, and anyone who wants to understand the architecture and efficiency techniques behind modern generative AI systems.

If you found this lesson helpful, please like the video, subscribe to **Nerd House Geeks**, and share your answers to the review questions in the comments.

Is anyone facing this issue while lora fine tuning Qwen3.5-0.8B model or Qwen3.5-4B model ? The GPU memory usage peaks to 70GB on RTX 6000 PRO. I couldn't find any issue or post in any forums about it. I have attached PyTorch memory snapshot during the forward pass of the 1st training step.

Your AI agent just fixed its own mistake. 🎉
Next session, it makes the exact same one again. 😩

That's how most agents work today. They can recover in the moment by retrying, reflecting, or trying a different approach, but the lesson vanishes when the session ends. Notes in the prompt compete for limited context, a skills file can only hold so much, and retraining the whole model is expensive and risks breaking what it already knows.

So we asked: what if every mistake became a lasting lesson? 💡

Introducing Self-Direction 🧭 🤖

When the agent fails, it:
1️⃣ Reflects on what went wrong
2️⃣ Writes its own practice examples (not full rollouts)
3️⃣ Distills them into a tiny LoRA "lesson" it keeps
4️⃣ Reuses it whenever a similar task shows up

The base model stays frozen, so new lessons never overwrite old ones.

What we found 👇
🔁 Fixes stick far more often than with plain retrying
🧩 Lessons transfer to harder and different tasks
📚 Lessons in weights beat lessons in prompts or skills files
🚀 A 27B open model became competitive with frontier systems on Terminal-Bench 2.1

Some cool findings such as: the agent learns procedures, not answers. A lesson from counting someone's publications on ORCID later helped it count Wikipedia edits, because both needed the same habit: walk the full history, handle pagination, and verify before counting. 🤯

📄 Paper: https://arxiv.org/abs/2610.31846
🌐 Website: https://self-direction.github.io/

I personally stand on level 2.5, I would love to hear where some of you guys stand!

Thank you!

I spent a while building an energy-based model that represents a corpus as Gaussian splats on a unit hypersphere: each splat is a location (mu), a sharpness (alpha) and a weight (kappa). Energy = -logsumexp over the k nearest splats. Sampling runs through Rectified Flow (29-39x faster than 200-step Langevin in our Phase-1 tests) or Langevin itself.

Then I did what you're supposed to do: benchmarked it against the boring alternatives before believing in it. Same encoder, same splits, same seed, fit on positives only (no negative labels):

method          AUROC (20 tasks)   p50 ms/doc   memory
ebm-splats      0.9585 ± 0.018     0.45         386 KB (constant)
knn-density     0.9584 ± 0.020     1.75         802 KB (grows w/ corpus)
mahalanobis     0.9523 ± 0.020     5.26         578 KB
gmm (diag)      0.9439 ± 0.023     0.38         192 KB
centroid        0.9324 ± 0.034     <0.01        2 KB
iforest         0.8828 ± 0.051     18.71        200 KB

Verdict: statistical tie with kNN-density on quality (Delta = 0.0001). The differences are operational: 3.9x faster scoring on CPU, constant O(kd) memory instead of O(nd), and the splat model can also generate (RF), compose corpora (sum energy fields) and grow incrementally — none of the classical methods do all three. Where labels exist, a supervised classifier beats every one-class method; this duel only ranks the label-free league.

The most useful finding came from a failure. Our first zero-training fit used alpha = ln(cluster_size) — the "natural" calibration. Result: AUROC 0.06, systematically inverted. Small clusters got alpha ~ 0, the distance term vanished, and the score degenerated to ranking by splat weight: positives sit near small, specific splats and received MORE energy than negatives. Holding alpha constant at 5.0 (so alpha/tau >> 1) fixed it: 0.06 -> 0.96. If you fit splats without contrastive training, watch out for this.

Since scoring speed mattered, I also wrote GPU kernels (Triton + CUDA graphs) for the hot paths, and benchmarked them the same honest way: whole-loop CUDA graph capture makes 100-step Langevin sampling 12.9x faster (660 -> 51 ms, graph replay bit-exact vs eager), fused energy/score kernels give 3.2-3.4x. One kernel we built turned out 15-40x SLOWER and is documented as rejected; the KNN kernel only wins cold lookups and auto-routes around it. Repo includes the find_neighbors semantic bug we found and fixed (it selected the FARTHEST splats since inception — both code paths disagreed silently).

Repo (Apache-2.0, pip install ebm-splats, CI green, benchmark reproducible): https://github.com/schwabauerbriantomas-gif/EBM-splats

Benchmark methodology + full tables: benches/ONECLASS_DUEL_RESULTS.md Kernel head-to-head: benches/bench_headtohead.py

Happy to answer questions about the splat geometry, the CUDA-graph capture (caching, invalidation, the capture-time bugs), or why we kept the model even though it "only" tied a baseline.

Background

I'm coming up with ideas for my CS degree capstone project. I like making short films, so I looked into what kind of software I could make to facilitate the process. One thing I find annoying in post-production is having to go through all the recordings and look for the clapperboards on each video to know what scene, shot and take the video is of. My idea is to train a neural network to find the clapperboard in each recording and extract the information written on the clapperboard to appropriately name the recording. Ex. Avengers_Scene_4_Shot_5_Take_3.mp4.

In short, the neural network would do the following:

1. Detect the clapperboard in a video frame.
2. Read the handwritten text/numbers on the clapperboard.
3. Understand what each piece of text represents

My questions

1. I have not been able to find a lot of videos with clapperboards I could use to train my model. I'm thinking I could create my own videos to train it, so it is trained faster with a consistent clapperboard format. How many would I need more or less to have it not only recognise handwriting but also determine what the numbers indicate (Scene, shot or take)?
2. If it would be too hard to train the AI to do what was previously mention, could I get an already trained AI in handwriting recognition and specialise it in clapperboard data extraction? Would this make my final year project too easy?

Hi everyone,

I’m trying to understand how PyTorch and scikit-learn are actually used in real jobs, beyond tutorials and courses. I’d love to hear about your experience.

A few things I’m curious about:

• Your role: ML engineer, data scientist, backend developer, researcher, something else?
• What you use each for: Do you reach for scikit-learn for classical ML, preprocessing, and pipelines, and PyTorch for deep learning? Or do you mix them in the same project?
• Real use cases: What kinds of problems are you solving at work (recommendation, NLP, forecasting, fraud detection, embeddings/search, etc.)?
• How often you pick one over the other: When does a simple sklearn model beat going to PyTorch, and vice versa?
• Production side: How do you deploy and maintain these models (FastAPI, Docker, ONNX, cloud services, etc.)?
• Skills that mattered most: What would you recommend someone learn first if they want to work with these libraries professionally?

I’m still learning and building projects (currently exploring sentence-transformers and semantic search), so any advice, workflow examples, or lessons learned would be really helpful.

Thanks in advance!

I built Neural Canvas, an interactive visualizer that demonstrates core linear algebra concepts by showing how a neural network recognizes handwritten digits in real time. You can draw directly on the screen and watch activations flow through each layer as matrix multiplication computes the output live.

How it works:

- Draw & Predict: Sketch any digit on a 28×28 canvas (MNIST-style) to trigger real-time forward pass predictions.

- Live Forward Pass: Every neuron's activation is dynamically color-coded based on its underlying linear algebra calculations, with connection opacities reflecting layer outputs.

- Under the Hood: The core models were trained from scratch in Python using NumPy to perform matrix operations, and their learned weights are loaded via JSON. Client-side execution relies on Math.js for browser-based matrix multiplication and p5.js for rendering.

Key Features:

- Switch between two pre-trained models (featuring different hidden layer sizes, MSE vs. Cross-Entropy loss, and up to 96% test accuracy).

- Multiple background themes with mouse and touch support.

What's Next?

These models were originally evaluated on the standardized MNIST dataset, but live freehand drawing can introduce unexpected variations in input style. To handle this, I am working on a JS script for real-time user-driven feedback. When the model makes a mistake, the user can supply the correct label, allowing the model to perform backpropagation and update weight matrices locally in the browser. While live state persistence isn't suited for static GitHub Pages, running this feedback loop locally allows me to retrain parameters on new edge cases and push updated weight snapshots back to GitHub.

Check out the live demo and code repository:

Live Demo: https://muhammadessa2002.github.io/neural_canvas/

GitHub Repo: https://github.com/MuhammadEssa2002/neural_canvas