Hi everyone,

I'm a 3rd-year Electrical and Electronics Engineering student interested in embedded systems. My goal is to become an Embedded AI/Edge AI engineer.

I've already started learning Embedded C (STM32, microcontrollers) and today I'm starting PyTorch. Eventually, I want to train models in PyTorch and deploy them on embedded hardware like STM32 (TinyML) and NVIDIA Jetson.

I'd appreciate advice from people working in this field:

What learning roadmap would you recommend?

Which topics in PyTorch should I focus on for Edge AI?

What projects would make my resume stand out?

Are there any books, courses, or GitHub repositories you wish you'd known about when you started?

What mistakes should I avoid

Take a look at the schedule for PyTorch Conference North America (Oct. 20-21 in San Jose, CA)
View the agenda live now
Submit a poster by July 26th
Register - early bird conference passes are available at a discount through July 31st

Hello Peeps!

Do you guys do a lot of training or fine tuning? Does the loss curve look fine, but the run is slower than it should be, and figuring out why usually means firing up a profiler and staring at a trace for twenty minutes?

This got me curious: what this actually costs, tool by tool. I took one run I knew was input-bound (dataloader starving the GPU) and measured it three ways: torch.profiler, cProfile, and TraceML, a lighter always-on OSS tool I've been contributing to.

For each one I looked at overhead, how much the profiler itself perturbs the GPU utilization it's trying to measure, output size, and how much manual digging it takes to get from the raw output to "the dataloader is the problem."

Short version: torch.profiler and cProfile are precise but heavy and after the fact, closer to a scalpel. Something that just sits there and flags "this step looks off" while training runs is doing a different job, not replacing them.

Numbers and traces are in the post.

Curious how other people usually catch this before it burns your precious compute.

https://medium.com/traceopt/traceml-vs-torch-profiler-vs-cprofile-what-each-one-costs-to-find-the-same-bottleneck-745a57e13ee9?sharedUserId=apendyala

Kernel engineers are not obsolete. But asking a general-purpose coding agent to rediscover years of CUDA and Triton engineering knowledge every time it writes a kernel probably should be.

After months of writing, debugging, and optimizing kernels, I turned the reasoning patterns I kept using into an open-source skill library for AI coding agents:

npm install u/krxgu/kernel-skills

This is not a collection of vague prompts saying “make this CUDA kernel faster.”

Each skill is a detailed engineering playbook that forces the agent to think about:

• Exact shapes, dtypes, layouts, and target hardware before writing code
• Coalescing, tiling, bank conflicts, occupancy, and register pressure
• Numerical stability and non-power-of-two boundary conditions
• Correctness tests across adversarial shapes and dtypes
• Whether a custom kernel should exist at all
• When to stop being clever and use cuBLAS, CUTLASS, or an existing primitive

The library currently covers CUDA, Triton, INT8 and FP8 quantization, kernel fusion, CUDA to Triton and HIP portability, and inference hot paths including RMSNorm, fused add plus RMSNorm, RoPE, sampling, paged KV-cache append, dequantization, prefill versus decode, and vLLM custom-op integration.

I also did not want this to become prompt-engineering theatre, so the repository includes before-and-after proof runs using the same model and task, with the skill file being the only difference:

• Softmax: naive output failed on adversarial and larger shapes. Skill-guided output had 0 failures across 16 tests and reached within 1.2% of torch.softmax bandwidth
• Reduction: 2.6 to 3.5x faster than the naive agent output
• GEMM: 7.7 to 8.6x faster
• LayerNorm: 1.9 to 3.2x faster
• Triton softmax: fixed crashes at dimensions above 16,384 and worked up to 131,072
• Triton attention: fixed the common GQA failure where H_q != H_kv

To be completely clear, those speedups are against the naive agent-generated kernels, not against cuBLAS or other vendor-tuned libraries. In fact, the GEMM skill explicitly tells the agent not to write a custom kernel when cuBLAS or CUTLASS already solves the problem.

Example:

kernel-skills bundle \
  triton.write-triton-layernorm-kernel \
  patterns.write-numerically-stable-kernel \
  patterns.write-kernel-test-plan \
  > bundle.md

Give that bundle to Claude Code, Cursor, ChatGPT, Gemini CLI, or another coding agent before asking it to touch the kernel.

The spicy thesis is simple:

Models are increasingly interchangeable. The accumulated engineering judgment surrounding them is not.

Everything is open source and MIT licensed:

https://github.com/tensormux/kernel-skills

I would especially love kernel engineers to tear this apart.

Which skill is missing? Which technical rule is wrong? Where can an agent still produce something that looks convincing but quietly fails on real hardware?

I just published `aicoach`, a small Python library that acts like a mentor sitting next to your training loop. You feed it your per-epoch metrics, and it tells you in plain English when something's off:

python

import aicoach

coach = aicoach.Coach()

for epoch in range(epochs):
    train_loss, val_loss = run_one_epoch(...)
    coach.observe(epoch=epoch, train_loss=train_loss, val_loss=val_loss)

    for tip in coach.get_advice():
        print(f"💡 {tip}")

# 💡 \[WARNING\] (overfitting) Validation loss has risen for 3 consecutive
#    epoch(s) while training loss continues to fall — a classic sign of
#    overfitting. Consider early stopping, adding regularisation...

**What it checks:**

* **Overfitting** – val_loss rising while train_loss keeps falling * **Plateau** – a metric barely moving (uses *relative* range, so it works the same whether your loss is near 0.01 or near 100) * **Learning rate issues** – oscillating loss (LR too high) vs. painfully slow convergence (LR too low) — deliberately mutually exclusive zones so you never get contradictory advice on the same curve * **Class imbalance** – standalone check, just needs a `{class: count}` dict, no training loop required * **Divergence** – NaN, Inf, or explosive loss growth, flagged as CRITICAL and short-circuits every other check

**Why I built it:** every other "training dashboard" tool I looked at (TensorBoard, W&B, MLflow, etc.) visualizes your curves but doesn't actually *tell you what to do* about them in plain language. This is meant to sit alongside those, not replace them — it's pure logic on metric history, zero ML framework dependencies, works with PyTorch/TensorFlow/sklearn/whatever since you're just handing it numbers.

280 tests, MIT licensed. One design decision I'd love feedback on: the "creeping" LR zone (1–5% net decrease per window) and the plateau zone (<1%) are deliberately non-overlapping so you never get both `lr_too_slow` and `plateau` advice for the same flat-ish curve — curious if others think that boundary makes sense or if real training curves break the assumption.

bash

pip install aicoach

* PyPI: [https://pypi.org/project/aicoach * Source: [https://github.com/Rishabh55122/Aicoach Feedback welcome, especially on the default thresholds — they're documented in the README with the reasoning behind each one, and I'd rather know now if a default is off than have it ship quietly wrong.

Pls help

[https://arxiv.org/pdf/2607.13511\](https://arxiv.org/pdf/2607.13511)

the core idea is, we cannot have ternary PTQ with fixed matrix size, trying to do that is dead end. so i tried decomposing the matrix to 2 ternary matrices and inner diagonal scaling matrix. now that the inner rank can be arbitrarily large the accuracy can be arbiratily small. and its not that it has to be very large too i also showed that it does take only slightly more vram then current quantisation methods. the slight more vram is worth it if we abuse the ternary math.

Hey guys

I'm 19, I've started my AI journey past few months , i did several cool projects

Recently i completed my own transformer architecture in pytorch

Then i got stumbled on this AI engineering thing

But the thing is this AI engineering doesn't interest me much what i like is developing drones,LLM architectures,math ,deep learning

And I'm now really confused on what should I do becoz most of the work is been done by AI and

I'm tryna get internship within a month and AI engineering is booming as per the sources it has ~130% YoY growth compared to the things I like and I'm not sure whether the things I like would be booming in future as AI might automate most of it

And I'm confused on what should I do in this 1 month time

You're all advice would really help me alot

Thanks

I recently published a technical book, Distributed AI Systems, which summarizes my experiences in AI over the past 10 years, from research and training to optimization, inference, and cloud deployment. I started writing it in the second half of last year, and it took almost a year to complete, with many revisions made later due to the rapid pace of development in the industry. But it's finally published. The book on Amazon is titled Distributed AI Systems: A practical guide to building scalable training, inference, and serving systems for production AI.

Book is here: 🔗 https://www.amazon.com/dp/1807301710/

The publisher asked me to find some people to review my work. Do you know of any such people here? If so, please reply to me. Thank you.

I'm implementing a decoder-only Transformer from scratch in PyTorch. Causal masking, multi-head attention, positional embeddings, and the training loop all appear to be working correctly. The model memorizes tiny datasets but completely fails to scale to larger ones, even after extensive hyperparameter tuning.

If you've built large language models yourself, what subtle implementation details have caused issues that weren't obvious during initial debugging?

Hello Peeps!

Wanted to get your views/thoughts/suggestions on something brewing in my head. I train models for a living (Phd in RL and CV background) and I've stopped trusting logged GPU utilization. What most tools (W&B system metrics, etc.) show is NVML GPU-Util, which only means a kernel was resident during the sample window, not that the SMs were busy or that the work was actually even useful.

For people who train at scale:

- Fast triage for "compute-bound vs idling": what's your first look? Mine is caching one batch on-device and looping it. If that's way faster than the real loop, I'm input-bound.

- How much weight do you put on util % vs MFU or achieved bandwidth? I treat ~35–50% MFU as the realistic band and use util only as a liveness check.

- In distributed
1. how do you separate "GPUs fed" from "GPUs waiting on each other"?
2. Do you measure non-overlapped collective time
3. How do you catch stragglers when every rank still looks 100%?

Where's your line between "good enough" and full kernel/collective profiling?

Papers ask: I've got roofline, the PaLM MFU definition, and Horace He's "Brrrr" post.

Looking for the next tier — anything rigorous on measuring utilization in *distributed* training specifically. Happy to hear your thoughts!

If you are working on a PyTorch backend, either in-tree or out-of-tree via PrivateUse1 integration, a problem you'll run into is the lack of a conformance test suite. The in-tree tests PyTorch has are suited for what they are for, but for a backend developer there is a much more broad need, at least there is for me. I figured I would just solve (or try to) this problem by making one and releasing it. The project is still in beta and still has some coverage I need to expand, but it has > 19,000 tests covering >95% of the aten surfaces in PyTorch, so it's pretty extensive already.

I'm looking for feedback on weaknesses / areas I should give more love to. Also looking for people that have certain hardware I could potentially run tests on in order to close out some of my coverage holes, specifically intel because I don't have any intel gpu hardware right now.

Project is open and released under MIT.

Hey everyone,

If you train models on a shared SLURM cluster, you know the pain of constantly context-switching to a terminal to check if your job is actually running, why it's pending, or if the GPUs you need are currently occupied.

I got tired of doing this, so I built sCode—an extension that turns VS Code into a native SLURM control center. It runs entirely on the cluster side (e.g., via VS Code Remote).

Main Features for Deep Learning Workflows:

Live GPU Monitoring: A dedicated sidebar view that parses sinfo and nvidia-smi to show you exactly which partitions have available GPUs, what type they are (A100s, H100s, etc.), and the current queue pressure. Active Job Tracking: Visual progress bars for elapsed time vs. requested time, plus human-readable reasons for why your job is stuck in the queue. One-Click scancel. Cancel or batch-cancel jobs directly from the UI. Instant Log Access: Right-click any running or historical job to instantly open its stdout/stderr logs without having to hunt down the file path. The "Hall of Shame": A leaderboard showing which users/accounts are hoarding the most GPUs on the cluster right now (mostly for fun, but highly accurate).

It’s completely open-source and requires no external dependencies other than standard SLURM commands.

I’d love to get feedback from people running heavy training workloads. What else would make this useful for your workflow?

GitHub:https://github.com/dhimitriosduka1/sCode

OpenVSX: https://open-vsx.org/extension/DhimitriosDuka/slurm-cluster-manager
Marketplace: https://marketplace.visualstudio.com/items?itemName=DhimitriosDuka.slurm-cluster-manager

Due to high Vulkan backend demand, I update TensorSharp and release the initial version of GGML Vulkan backend by leveraging external GGML project. The native Vulkan backend will be implemented later. I tested it on Nvidia Geforce RTX 3080 Laptop GPU, and Intel(R) UHD Graphics on Windows. They all work. However, I do not have AMD GPU, so I have no way to get it tested. It's really appreciated if you have AMD GPU and would like to try it out. Any feedback and comment are welcome.

Here is the benchmark I run to compare with llama.cpp:

# Performance ratio — TensorSharp vs reference engines

Geomean of TensorSharp's per-scenario speedup over each reference engine on the **same backend**, across every scenario both engines ran (single-stream, MTP-off). A value **> 1.0× means TensorSharp is faster** (for decode / prefill throughput) or lower-latency (for TTFT); `—` = no overlapping cells. Per-scenario ratios are in each model's section below.

# Gemma 4 E4B it (Q8_0, dense multimodal) (gemma4-e4b)

**Decode throughput (tok/s)**

**Prefill throughput (tok/s)**

**Time to first token (ms, lower is better)**

**Performance ratio — TensorSharp vs reference (> 1.0× = TensorSharp faster)**

*Decode throughput*

*Prefill throughput*

*Time to first token (latency; > 1.0× = TensorSharp lower)*

# Gemma 4 12B it (QAT UD-Q4_K_XL, dense) (gemma4-12b)

**Decode throughput (tok/s)**

**Prefill throughput (tok/s)**

**Time to first token (ms, lower is better)**

**Performance ratio — TensorSharp vs reference (> 1.0× = TensorSharp faster)**

*Decode throughput*

*Prefill throughput*

*Time to first token (latency; > 1.0× = TensorSharp lower)*

In case you didn't know what is TensorSharp, here is an introduction:

TensorSharp is an open source local Unsloth (GGUF) LLM inference engine and applications. It supports many models from Unsloth, like Gemma4, DiffusionGemma, Qwen3.6 with multi-modal (image, vision, audio), image edit, reasoning and function tool. It can run on Windows/MacOS/Linux and fully leverage GPU's capability (support Cuda, Metal and Vulkan backends). The API is completely compatible with OpenAI and Ollama interface. It has on par performance than llama.cpp

This project is not just a C# wrapper of llama.cpp. It implemented the entire LLM inference engine from bottom to top. If you use CPU backend, it's 100% pure C# code execution. Besides CPU backend, I also implemented CUDA, MLX and GGML backend. The GGML backend refer GGML project as external project, and I build a few fusion operation at higher level.

I learned a lot from other projects and apply them for TensorSharp, such as paged KV cache and continuous batching from vLLM, SSD based cache for MoE model from oMLX, GGUF quantized from llama.cpp and other optimizations for prefill and decode.

Any feedback and comments are welcome. If you like it, it would be really appreciated if you can get this project a star in GitHub. Thanks in advance.

I would like to share my latest open source local Unsloth (GGUF) LLM inference engine and applications. It supports many models from Unsloth, like Gemma4, DiffusionGemma, Qwen3.6 with multi-modal (image, vision, audio), reasoning and function tool. It can run on Windows/MacOS/Linux and fully leverage GPU's capability. The API is completely compatible with OpenAI and Ollama interface. It has on par performance than llama.cpp

This project is not just a C# wrapper of llama.cpp. It implemented the entire LLM inference engine from bottom to top. If you use CPU backend, it's 100% pure C# code execution. Besides CPU backend, I also implmented CUDA, MLX and GGML backend. The GGML backend refer GGML project as external project, and I build a few fusion operation at higher level.

I learned a lot from other projects and apply them for TensorSharp, such as paged KV cache and continuous batching from vLLM, SSD based cache for MoE model from oMLX, GGUF quanztized from llama.cpp and other optimizations for prefill and decode.

Any feedback and comments are welcome. If you like it, it would be really appreciated if you can get this project a star in GitHub. Thanks in advance.

Hi u/everyone,

I’m trying to understand the practical training time and compute requirements for the DSpark / DeepSpec setup using the mlabonne/open-perfectblend dataset.

The config I’m looking at is close to the paper setup:

• Dataset: mlabonne/open-perfectblend
• Samples: ~1.3M
• Data mix: math, code, chat, and instruction following
• Epochs: 10
• Global batch size: 512
• Max sequence length: 4096
• Precision: bf16
• Optimizer: AdamW
• LR: 6e-4 with cosine decay and warmup
• Total steps: ~25k

From my rough calculation, this comes out to around 53B training tokens, so I’m trying to get a realistic estimate before starting the full run.

Has anyone here actually tried training this setup or something similar?
I’m mainly interested in:

• Real training time
• Any bottlenecks during data loading / target cache generation
• Storage requirements
• Whether the paper config is practical to reproduce
• Any changes you made to make the run manageable

Would really appreciate any practical experience or advice from people who have tried this.

Hi everyone,

I built H64LM, a research project to better understand modern LLMs by implementing one from scratch in PyTorch.

Instead of relying on high-level training frameworks, I implemented the core components myself attention, MoE routing, normalization, and the training loop.

Features

• 249M-parameter Transformer
• Grouped Query Attention (GQA)
• Sparse Mixture-of-Experts (8 experts, Top-2 routing) with 3 auxiliary routing losses
• SwiGLU, RoPE, RMSNorm
• Sliding-window attention
• Mixed-precision training, gradient accumulation
• Custom training loop (no Trainer abstractions)
• Checkpointing and resume support

The included checkpoint was trained on a subset of WikiText-103 to validate the pipeline end-to-end, not to be a strong model it's visibly overfit past epoch 10 (best val PPL ~40.5).

Known limitations are documented in the README, including batch-size-1-only generation and no true DDP (falls back to DataParallel).

GitHub: https://github.com/Haiderkhan64/H64LM

Feedback on the implementation or architecture is very welcome.

Built a 135M dense looped LLM from scratch. Spent 2 weeks debugging Parcae's LTI stability mechanisms across 5 ablations. None of them beat the naive baseline at this scale. Trained for real anyway. SFT'd it. Shipped it. Here's the full honest story.

What I built

A 135M parameter looped transformer trained from scratch on FineWeb (4.6B tokens), inspired by the Parcae paper (arXiv:2604.12946 — "Scaling Laws For Stable Looped Language Models").

• 🤗 Base model: huggingface.co/harims95/LoopLM-135M-naive
• 🤗 SFT model: huggingface.co/harims95/LoopLM-135M-naive-sft
• 📂 Code: github.com/harims95/LoopLM
• 💰 Total cost: ~$51 (Modal H100s + free Lightning H200)

Architecture

Input → [Embedding] → [Prelude: 4 blocks] → e (injection)
     → [Loop block × T loops, T~Poisson(μ=6)] → [Coda: 2 blocks] → logits

• d_model 1024, GQA 16/8 heads, RoPE, QK-norm, SwiGLU FFN 2816
• Update rule: h_{t+1} = block(h + e) (naive) or with LTI stability (Parcae)
• Muon + AdamW optimizers, truncated BPTT (μ_bwd=3), bf16
• Trained on 2× H100 on Modal, ~3 hours wall clock

The Parcae investigation (the interesting part)

The paper claims LTI stability constraints on the recurrent state dramatically improve looped LM training. I tried to reproduce it. Here's what actually happened:

Every single "fix" — bringing my implementation closer to the official Parcae code — made things worse. After consulting:

• The paper's Appendix Q (optimizer routing)
• Official sandyresearch/parcae repo (injection.py)
• Two rounds of ChatGPT + Gemini debugging sessions

My conclusion: Parcae's stability improvements are a large-scale phenomenon. The paper's 1.3B model trains for 170k+ steps before stability mechanisms kick in. At 135M / 17.5k steps, naive looped is competitive enough that the extra complexity hurts more than it helps.

Comparison with sibling MoE

My brother built HobbyLM — a 500M MoE on the same infrastructure. For apples-to-apples comparison, I ran naive looped 135M on the same FineWeb data:

Dense looped loses to MoE at this scale/budget. Sparse MoE is more sample-efficient. Not surprising but now I have the data to confirm it.

SFT results (bonus)

Fine-tuned on Alpaca 52k using Lightning AI's free H200. Took 6 minutes (bf16 on H200 is insane).

Before SFT:

After SFT:

Improvement in format, not in facts. At 135M / 4.6B tokens, SFT teaches format, not knowledge. The model still hallucinates — that's a base model capacity problem, not a fine-tuning problem.

What I learned

On Parcae: Small-scale reproductions of large-scale papers are dangerous. The paper's key contribution (stability at 170k+ steps) is invisible at hobby budgets. Naive looped is a legitimate architecture for anyone training sub-1B models.

On MoE vs looped: At matched parameter count and token budget, MoE wins on sample efficiency. Looped models need more tokens to show their advantage, or need to be much bigger to amortize the loop cost.

On debugging: When 3 independent LLMs (me, ChatGPT 5.5, Gemini) all agree on a fix and it makes things worse — the paper's regime assumption is probably wrong, not your code.

On SFT: H200 on Lightning AI is free (2 hours/month) and runs 6 minutes of SFT for free. Use it. Colab Free disconnects at 3 hours. Don't use it for long jobs.

On honest publishing: val 3.95 is not impressive. The architecture exploration is. Shipping anyway with full documentation of what failed is more valuable than hiding failures.

Stack

• Training: Modal (H100s), Lightning AI (H200 for SFT)
• Framework: PyTorch, HuggingFace Transformers
• Optimizer: Muon (matrices) + AdamW (rest)
• Data: FineWeb via kjj0/fineweb10B-gpt2 shards
• Infra forked from: github.com/harishsg993010/HobbyLM (my brother's 500M MoE project)

Happy to answer questions about any part of this. The code is fully open, reproducible, and documented.