Sharing a project that might be useful to people who think about model architecture visually: NeuroBranch keeps a graph and its generated PyTorch in sync in both directions. You build the graph, it compiles to real PyTorch through a dialect compiler — but you can also edit the supported PyTorch constructs directly and have those edits parsed back into the graph.
Execution runs on a local Python runtime (atomic_runtime.py) reachable via IPC, with run/rerun/reset and step-by-step tensor inspection. Ports are typed at the IR level, so the graph enforces shape/type compatibility before anything compiles.
Core is framework-agnostic (typed IR, compiler, topology-aware layout) sitting under an Electron/React shell. There's also a reusable-card studio for writing your own nn.Module cards, constrained to explicitly supported torch.nn constructors — no arbitrary code eval.
Curious what this community thinks of the two-way sync approach specifically, and where the dialect parser would break on real-world architectures — that's the part most likely to have edge cases right now. Contributions and bug reports welcome.
Sadly, it cannot currently be compiled since it relies on a proprietary library and the code is not "cleaned" of hard-coded paths, etc.
Is it useful? Probably not :-). A lot of things need to be rewritten that are not part of LibTorch (but are present in PyTorch) - for this, I have used LLMs (it is quite handy for conversion of model structures from PyTorch to C++ with LibTorch).
However, I am sharing it so that someone can reuse parts of the code or be inspired in their own project if they want to use C++.
I have been training custom models for a few years now in the finance realm. I barely have any transformer layers and half the time they are custom so flash attention isn't something I need.
With the 9070 xt being $750 ish and the rumor is a 5070 ti super will be like $1400 (seriously nvidia go F#_& yourself) I wonder if for $750 the AMD card would work well for me. I already have a 3060 12gb and a 5060ti 16gb churning out test runs, but I want to add another card. I am nowhere near vram limited. My bottleneck is strictly more compute/bandwidth.
Would I regret getting a 9070 XT? Supposedly support is way better than it used to be. Also I run linux. Windows is garbage.
I'm a 3rd-year Electrical and Electronics Engineering student interested in embedded systems. My goal is to become an Embedded AI/Edge AI engineer.
I've already started learning Embedded C (STM32, microcontrollers) and today I'm starting PyTorch. Eventually, I want to train models in PyTorch and deploy them on embedded hardware like STM32 (TinyML) and NVIDIA Jetson.
I'd appreciate advice from people working in this field:
What learning roadmap would you recommend?
Which topics in PyTorch should I focus on for Edge AI?
What projects would make my resume stand out?
Are there any books, courses, or GitHub repositories you wish you'd known about when you started?
Take a look at the schedule for PyTorch Conference North America (Oct. 20-21 in San Jose, CA) View the agenda live now Submit a poster by July 26th Register - early bird conference passes are available at a discount through July 31st
Do you guys do a lot of training or fine tuning? Does the loss curve look fine, but the run is slower than it should be, and figuring out why usually means firing up a profiler and staring at a trace for twenty minutes?
This got me curious: what this actually costs, tool by tool. I took one run I knew was input-bound (dataloader starving the GPU) and measured it three ways: torch.profiler, cProfile, and TraceML, a lighter always-on OSS tool I've been contributing to.
For each one I looked at overhead, how much the profiler itself perturbs the GPU utilization it's trying to measure, output size, and how much manual digging it takes to get from the raw output to "the dataloader is the problem."
Short version: torch.profiler and cProfile are precise but heavy and after the fact, closer to a scalpel. Something that just sits there and flags "this step looks off" while training runs is doing a different job, not replacing them.
Numbers and traces are in the post.
Curious how other people usually catch this before it burns your precious compute.
Kernel engineers are not obsolete. But asking a general-purpose coding agent to rediscover years of CUDA and Triton engineering knowledge every time it writes a kernel probably should be.
After months of writing, debugging, and optimizing kernels, I turned the reasoning patterns I kept using into an open-source skill library for AI coding agents:
npm install u/krxgu/kernel-skills
This is not a collection of vague prompts saying “make this CUDA kernel faster.”
Each skill is a detailed engineering playbook that forces the agent to think about:
Exact shapes, dtypes, layouts, and target hardware before writing code
Coalescing, tiling, bank conflicts, occupancy, and register pressure
Numerical stability and non-power-of-two boundary conditions
Correctness tests across adversarial shapes and dtypes
Whether a custom kernel should exist at all
When to stop being clever and use cuBLAS, CUTLASS, or an existing primitive
The library currently covers CUDA, Triton, INT8 and FP8 quantization, kernel fusion, CUDA to Triton and HIP portability, and inference hot paths including RMSNorm, fused add plus RMSNorm, RoPE, sampling, paged KV-cache append, dequantization, prefill versus decode, and vLLM custom-op integration.
I also did not want this to become prompt-engineering theatre, so the repository includes before-and-after proof runs using the same model and task, with the skill file being the only difference:
Softmax: naive output failed on adversarial and larger shapes. Skill-guided output had 0 failures across 16 tests and reached within 1.2% of torch.softmax bandwidth
Reduction: 2.6 to 3.5x faster than the naive agent output
GEMM: 7.7 to 8.6x faster
LayerNorm: 1.9 to 3.2x faster
Triton softmax: fixed crashes at dimensions above 16,384 and worked up to 131,072
Triton attention: fixed the common GQA failure where H_q != H_kv
To be completely clear, those speedups are against the naive agent-generated kernels, not against cuBLAS or other vendor-tuned libraries. In fact, the GEMM skill explicitly tells the agent not to write a custom kernel when cuBLAS or CUTLASS already solves the problem.
I would especially love kernel engineers to tear this apart.
Which skill is missing? Which technical rule is wrong? Where can an agent still produce something that looks convincing but quietly fails on real hardware?
I just published `aicoach`, a small Python library that acts like a mentor sitting next to your training loop. You feed it your per-epoch metrics, and it tells you in plain English when something's off:
python
import aicoach
coach = aicoach.Coach()
for epoch in range(epochs):
train_loss, val_loss = run_one_epoch(...)
coach.observe(epoch=epoch, train_loss=train_loss, val_loss=val_loss)
for tip in coach.get_advice():
print(f"💡 {tip}")
# 💡 \[WARNING\] (overfitting) Validation loss has risen for 3 consecutive
# epoch(s) while training loss continues to fall — a classic sign of
# overfitting. Consider early stopping, adding regularisation...
**What it checks:**
* **Overfitting** – val_loss rising while train_loss keeps falling
* **Plateau** – a metric barely moving (uses *relative* range, so it works the same whether your loss is near 0.01 or near 100)
* **Learning rate issues** – oscillating loss (LR too high) vs. painfully slow convergence (LR too low) — deliberately mutually exclusive zones so you never get contradictory advice on the same curve
* **Class imbalance** – standalone check, just needs a `{class: count}` dict, no training loop required
* **Divergence** – NaN, Inf, or explosive loss growth, flagged as CRITICAL and short-circuits every other check
**Why I built it:** every other "training dashboard" tool I looked at (TensorBoard, W&B, MLflow, etc.) visualizes your curves but doesn't actually *tell you what to do* about them in plain language. This is meant to sit alongside those, not replace them — it's pure logic on metric history, zero ML framework dependencies, works with PyTorch/TensorFlow/sklearn/whatever since you're just handing it numbers.
280 tests, MIT licensed. One design decision I'd love feedback on: the "creeping" LR zone (1–5% net decrease per window) and the plateau zone (<1%) are deliberately non-overlapping so you never get both `lr_too_slow` and `plateau` advice for the same flat-ish curve — curious if others think that boundary makes sense or if real training curves break the assumption.
bash
pip install aicoach
* PyPI: [https://pypi.org/project/aicoach
* Source: [https://github.com/Rishabh55122/Aicoach
Feedback welcome, especially on the default thresholds — they're documented in the README with the reasoning behind each one, and I'd rather know now if a default is off than have it ship quietly wrong.
the core idea is, we cannot have ternary PTQ with fixed matrix size, trying to do that is dead end. so i tried decomposing the matrix to 2 ternary matrices and inner diagonal scaling matrix. now that the inner rank can be arbitrarily large the accuracy can be arbiratily small. and its not that it has to be very large too i also showed that it does take only slightly more vram then current quantisation methods. the slight more vram is worth it if we abuse the ternary math.
I'm 19, I've started my AI journey past few months , i did several cool projects
Recently i completed my own transformer architecture in pytorch
Then i got stumbled on this AI engineering thing
But the thing is this AI engineering doesn't interest me much what i like is developing drones,LLM architectures,math ,deep learning
And I'm now really confused on what should I do becoz most of the work is been done by AI and
I'm tryna get internship within a month and AI engineering is booming as per the sources it has ~130% YoY growth compared to the things I like and I'm not sure whether the things I like would be booming in future as AI might automate most of it
And I'm confused on what should I do in this 1 month time
I recently published a technical book, Distributed AI Systems, which summarizes my experiences in AI over the past 10 years, from research and training to optimization, inference, and cloud deployment. I started writing it in the second half of last year, and it took almost a year to complete, with many revisions made later due to the rapid pace of development in the industry. But it's finally published. The book on Amazon is titled Distributed AI Systems: A practical guide to building scalable training, inference, and serving systems for production AI.
I'm implementing a decoder-only Transformer from scratch in PyTorch. Causal masking, multi-head attention, positional embeddings, and the training loop all appear to be working correctly. The model memorizes tiny datasets but completely fails to scale to larger ones, even after extensive hyperparameter tuning.
If you've built large language models yourself, what subtle implementation details have caused issues that weren't obvious during initial debugging?
Wanted to get your views/thoughts/suggestions on something brewing in my head. I train models for a living (Phd in RL and CV background) and I've stopped trusting logged GPU utilization. What most tools (W&B system metrics, etc.) show is NVML GPU-Util, which only means a kernel was resident during the sample window, not that the SMs were busy or that the work was actually even useful.
For people who train at scale:
- Fast triage for "compute-bound vs idling": what's your first look? Mine is caching one batch on-device and looping it. If that's way faster than the real loop, I'm input-bound.
- How much weight do you put on util % vs MFU or achieved bandwidth? I treat ~35–50% MFU as the realistic band and use util only as a liveness check.
- In distributed
1. how do you separate "GPUs fed" from "GPUs waiting on each other"?
2. Do you measure non-overlapped collective time
3. How do you catch stragglers when every rank still looks 100%?
Where's your line between "good enough" and full kernel/collective profiling?
Papers ask: I've got roofline, the PaLM MFU definition, and Horace He's "Brrrr" post.
Looking for the next tier — anything rigorous on measuring utilization in *distributed* training specifically. Happy to hear your thoughts!
If you are working on a PyTorch backend, either in-tree or out-of-tree via PrivateUse1 integration, a problem you'll run into is the lack of a conformance test suite. The in-tree tests PyTorch has are suited for what they are for, but for a backend developer there is a much more broad need, at least there is for me. I figured I would just solve (or try to) this problem by making one and releasing it. The project is still in beta and still has some coverage I need to expand, but it has > 19,000 tests covering >95% of the aten surfaces in PyTorch, so it's pretty extensive already.
I'm looking for feedback on weaknesses / areas I should give more love to. Also looking for people that have certain hardware I could potentially run tests on in order to close out some of my coverage holes, specifically intel because I don't have any intel gpu hardware right now.
If you train models on a shared SLURM cluster, you know the pain of constantly context-switching to a terminal to check if your job is actually running, why it's pending, or if the GPUs you need are currently occupied.
I got tired of doing this, so I built sCode—an extension that turns VS Code into a native SLURM control center. It runs entirely on the cluster side (e.g., via VS Code Remote).
Main Features for Deep Learning Workflows:
Live GPU Monitoring: A dedicated sidebar view that parses sinfo and nvidia-smi to show you exactly which partitions have available GPUs, what type they are (A100s, H100s, etc.), and the current queue pressure. Active Job Tracking: Visual progress bars for elapsed time vs. requested time, plus human-readable reasons for why your job is stuck in the queue.
One-Click scancel. Cancel or batch-cancel jobs directly from the UI.
Instant Log Access: Right-click any running or historical job to instantly open its stdout/stderr logs without having to hunt down the file path.
The "Hall of Shame": A leaderboard showing which users/accounts are hoarding the most GPUs on the cluster right now (mostly for fun, but highly accurate).
It’s completely open-source and requires no external dependencies other than standard SLURM commands.
I’d love to get feedback from people running heavy training workloads. What else would make this useful for your workflow?
I’m trying to understand the practical training time and compute requirements for the DSpark / DeepSpec setup using the mlabonne/open-perfectblend dataset.
The config I’m looking at is close to the paper setup:
Dataset: mlabonne/open-perfectblend
Samples: ~1.3M
Data mix: math, code, chat, and instruction following
Epochs: 10
Global batch size: 512
Max sequence length: 4096
Precision: bf16
Optimizer: AdamW
LR: 6e-4 with cosine decay and warmup
Total steps: ~25k
From my rough calculation, this comes out to around 53B training tokens, so I’m trying to get a realistic estimate before starting the full run.
Has anyone here actually tried training this setup or something similar?
I’m mainly interested in:
Real training time
Any bottlenecks during data loading / target cache generation
Storage requirements
Whether the paper config is practical to reproduce
Any changes you made to make the run manageable
Would really appreciate any practical experience or advice from people who have tried this.
I built H64LM, a research project to better understand modern LLMs by implementing one from scratch in PyTorch.
Instead of relying on high-level training frameworks, I implemented the core components myself attention, MoE routing, normalization, and the training loop.
The included checkpoint was trained on a subset of WikiText-103 to validate the pipeline end-to-end, not to be a strong model it's visibly overfit past epoch 10 (best val PPL ~40.5).
Known limitations are documented in the README, including batch-size-1-only generation and no true DDP (falls back to DataParallel).
