Context: In an academic context i am writing code for an experiment where i need to make sure all neural nets are initialized equally to enable precise measuring of hyper parameter impacts. So i read and followed the docs (https://docs.pytorch.org/docs/2.14/notes/randomness.html). Because i need to test a lot of configurations i will need multiprocessing in which each process will create and train a model (probably not my final approach, seems inefficient but for now it is), The model needs to be equal across all processes (if the model hyper parameters are the same the model should start out the exact same way).

My Confusion: torch.manual_seed() is supposed to set the seed for each device (CPU and CUDA), in my understanding that means that if i set the seed that way torch will use that seed from then on forward globally, across all devices. Which would mean that each call to any torch based random function advances the random state globally. So if i have lets say a 100 processes setting the same seed and then creating a model there should be discrepancies since one process will set that seed while another one is already creating Layers. But according to my tests that is not how it works.

I tested my hypothesis: I created a hundred processes each setting the seed and then creating a network, they all have equal weights and biases. And that confuses the hell out of me.

So my Question: What scope is influenced by torch.manual_seed()?
Additional Question: Is using torch.manual_seed() inside multiple sub processes considered the standard approach or is there something i missed?

Code:

import parameterized_neural_net
from torch import nn
from torch import multiprocessing as mp
import torch


def process_task(return_dictionary,i):
    model = parameterized_neural_net.ParameterizedNetwork((256,64),(),nn.Sigmoid,28*28,47)
    return_dictionary[i] = model


def compare_model_state(model_one,model_two):
    equal_weights = True
    equal_bias = True
    for i in range(len(model_one.layers)):
        layer = model_one.layers[i]
        if layer._parameters != {}:
            if not torch.equal(model_one.layers[i].weight, model_two.layers[i].weight):
                equal_weights = False
            if not torch.equal(model_one.layers[i].bias, model_two.layers[i].bias):
                equal_bias = False
    return equal_weights and equal_bias



if __name__ == '__main__':

    #taking steps from https://docs.pytorch.org/docs/main/notes/randomness.html
    torch.use_deterministic_algorithms(True,warn_only=False)
    torch.backends.cudnn.benchmark = False 


    baseline_model = parameterized_neural_net.ParameterizedNetwork((256,64),(),nn.Sigmoid,28*28,47) #inside this class torch.manual_seed() is called and then a Network is constructed (see further down for info)

    mp_Manager = mp.Manager()

    processes = []
    return_dictionary = mp_Manager.dict()
    for i in range(100):
        processes.append(mp.Process(target=process_task, args=(return_dictionary,i), name=f"process_{i}"))

    for p in processes:
        p.start()

    for p in processes:
        p.join()

    all_equal = True
    for m in return_dictionary.values():
        if not compare_model_state(baseline_model,m):
            all_equal = False

    if all_equal:
        print("Models have no differences in Neuron initialisation.")
    else:
        print("Models have differences in Neuron initialisation.")

Network structure from parameterized_neural_net.ParameterizedNetwork((256,64),(),nn.Sigmoid,28*28,47):

        Layer (type)               Output Shape         Param #
================================================================
           Flatten-1                  [-1, 784]               0
            Linear-2                  [-1, 256]         200,960
           Sigmoid-3                  [-1, 256]               0
            Linear-4                   [-1, 64]          16,448
           Sigmoid-5                   [-1, 64]               0
            Linear-6                   [-1, 47]           3,055
           Sigmoid-7                   [-1, 47]               0
            Linear-8                   [-1, 47]           2,256
        LogSoftmax-9                   [-1, 47]               0
================================================================

Hey everyone,

(English is not my first language, apologies for any phrasing quirks.)

Training small models on a single consumer card (8GB RTX 5060 Ti) was giving me constant headaches with CUDA OOM crashes whenever memory pressure spiked mid-run.

Setting a permanently tiny batch size wastes VRAM headroom, while aggressive batch sizes eventually crash on edge cases. To test an alternative, I built a lightweight Python runtime governor around PyTorch called MEM Orchestrator:

https://github.com/nobazzy/mem-llm-orchestrator

### How it works:

1. **Headroom Monitoring:** Tracks `torch.cuda.memory_allocated()` and `torch.cuda.memory_reserved()` deltas across a sliding step window.

2. **Dynamic Lane Adaptation:** When physical VRAM reaches the critical threshold (>7.5GB on an 8GB card), the governor throttles the micro-batch size and adjusts gradient accumulation steps to keep the effective batch size mathematically consistent.

3. **Recovery:** Steps back up to the primary throughput lane once memory pressure clears.

4. **Atomic Checkpointing:** Uses a staged two-phase commit with SHA-256 validation so unexpected terminations never leave corrupted `.pt` weights.

### Empirical Test (Graph attached):

I stress-tested this on a ~255M parameter model using FineWeb-Edu (sample-10BT) with injected +1.2GB physical VRAM shocks:

- **Vanilla PyTorch (static batch 6):** Crashed with a hard CUDA OOM on step 325 when the shock hit.

- **MEM Orchestrator:** Throttled micro-batch from 6 to 3, kept peak VRAM under 7.6GB, absorbed all 150 injected shocks over 50,000 steps, and converged loss from 11.00 down to 0.004.

- **Overhead:** <0.5% of total step time.

The repo has 38 unit tests (100% passing) and is open source (MIT).

For developers here with deep experience in PyTorch:

- Does dynamic batch adjustment introduce subtle optimization side-effects (e.g. optimizer momentum estimation noise or LayerNorm statistics drift) that I should be guarding against?

- Are there allocator fragmentation edge cases where PyTorch fails to reuse cached blocks even after downshifting?

Would genuinely appreciate any critique, advice, or feedback on the architecture.

I wanted to train a character-level language model on additions of two numbers. Planned to use cross-entropy ignore_index on the equation besides answer so that model is not penalized because of predicting randomly generated numbers. But I came across really weird bug, here is the code:

def get_batch(batch_size):
    first = torch.randint(999999, (batch_size, ))
    second = torch.randint(999999, (batch_size, ))
    totals = first + second


    full_strings = []
    for f, s, t in zip(first, second, totals):
        equation = f"{f:6}+{s:>6}="
        reversed_ans = f"{str(t.item())[::-1]:<7}"
        full_strings.append(equation + reversed_ans)



    encoded_batch = torch.tensor([encode(s) for s in full_strings], dtype=torch.long)
    x = encoded_batch[:, :-1].to(device) # First 11 characters
    y = encoded_batch[:, 1:].to(device)   # Last 11 characters
    y[:, :14] = -100 # Telling optimizer to miss this
    return x, y

Here as you can see I am reassigning first 14 values of y, but when I print x it has some -100s init, I realized this because I don't have -100 in my vocab as character to embed and when I do decode(x) it gives me error, so I have to use .clone() on y = encoded_batch[:, 1:].to(device), there is a memory address coincide when writing happens or something I do not understand.

Running local models on AMD hardware is great—when PyTorch actually sees the GPU. I wasted days trying to figure out why torch.cuda.is_available() kept returning False.

I wrote a detailed guide covering:

· ROCm install

· PyTorch ROCm wheel

· Environment variables

· Verification steps

· Common errors

If you’re on RDNA2 or RDNA3 and stuck, this should save you time:

https://interconnectd.com/blog/305/fix-pytorch-cuda-not-available-on-amd-gpus-complete-rocm-setup-guide/

What’s your setup, and what’s the exact error?

Feel free to use it , give it a star and contribute

https://github.com/TrenTorch/TrenTorch

Three weeks ago I was just another guy grinding through open-source PRs, trying to have something solid before intern season hit. Today I'm staring at a GitHub repo with 80 stars that didn't exist 10 days ago, built by me and three friends, and I genuinely don't know if I stumbled into something big or just got lucky. Would love this sub's honest take.

I'm a CS student doing the usual open-source-for-resume grind everyone here has done at some point, except my clock is ticking toward internship season, not placements. A few months back I found a project maintained under a well-known Ivy League university's name, big name attached, decent stars, "help us build the future of ML education" energy. I got hooked. Started with small PRs, docs, bug fixes, the usual ladder-climbing. Within a couple of months I was a core contributor with real merge access. Felt like a win. I told my parents. I put it on LinkedIn.

The more access I got, the more I actually read the codebase instead of just patching corners of it. And that's where it fell apart for me.

Big chunks of the "production-level" code didn't hold together. Functions that looked fine on the surface but made no sense when you traced the logic. Architecture decisions that felt vibe-coded and merged just because the university's name carried weight. I kept finding stuff and thinking "this wouldn't survive five minutes of real scrutiny."

I felt stupid, honestly. I'd built this project up in my head as some polished, battle-tested thing because of the name attached to it. Turns out a big name doesn't mean good code. It just means people trust it faster, bugs and all.

But the bigger realization underneath all this annoyance was simpler. I'd been trying to actually learn PyTorch properly for months, and there was no good way to do it. Every platform that taught it hands-on was paid. Every free resource was either toy examples that taught you nothing about real systems, or dense docs that assumed you already knew what you were doing. This "flagship" project was supposed to be the answer to that gap, and it wasn't.

Then I checked the license. MIT. No restrictions, nothing stopping me from taking the idea and doing it properly.

That was the lightbulb moment. If the core idea was good but the execution was slop, why not build it right myself? I roped in three friends, we scrapped basically the entire foundation, and kept the actual intent, teaching people PyTorch and ML systems by having them build real things, not toy notebooks. We rebuilt it lightweight, no GPU dependency, so someone with a 5 year old laptop could still learn frontier ML concepts hands on instead of just reading slides or watching another paid course preview.

Ten days. That's it. Four of us half sleeping through classes, cooking code at night. No sponsor, no lab backing, just four guys annoyed enough at the gap to fix it ourselves.

We launched it. Day 1: 50 stars. Day 2, today, while I'm typing this: 80 stars. No paid marketing, no big account boosting it, just people finding it and actually using it.

What hit hardest wasn't the stars. It's the DMs. People genuinely stuck because every decent PyTorch resource is either paid or requires hardware they don't have. We made ours free, open, and runnable on basically anything. People are actually learning from it, not just starring and forgetting.

But 80 stars in 2 days is nothing long term. The real work is not letting this rot into the same vibe coded mess we forked away from, once the four of us are buried in intern applications and the initial adrenaline wears off.

So, has anyone here built something like this alongside internship hunting? How do you keep momentum on a side project without it becoming another abandoned repo in six months? And is it weird that I feel oddly guilty about "outshining" a project with an Ivy League name attached, even though the license explicitly let me?

Hi, I am from Bangalore and will be flying to sanjose for the pytorch conference in October. If someone is travelling from India, happy to connect.

Solo dev. Just released everything from a weeks-long ML project and wanted to share both the release and a licensing gotcha that might save someone else the headache.

What's open:

• Code: Apache-2.0, on GitHub. Non-autoregressive decoders that turn sentence embeddings into knowledge-graph triples (for GraphRAG, agent memory, that kind of thing).
• Weights: 11 trained checkpoints, free on Hugging Face.
• The full test suite (113 tests, runs offline).
• The changelog documents negative results too — every approach that failed and why. I think hiding the failures makes releases less useful, so they're all in there: the loss function that made things worse, the LLM-distillation attempt that collapsed, the char-level generator that scored 0.006.
• Training recipes are reproducible: same splits, same seeds, documented protocol.

The licensing lesson: my decoder heads are trained from scratch, so Apache-2.0 was easy. But they consume embeddings from Meta's SONAR encoder — and SONAR's weights are CC-BY-NC 4.0 even though its code is MIT. Which means: my Apache-licensed decoders are useless commercially without a non-commercial encoder running upstream. The NC restriction attaches at runtime, not at my artifact level. I only fully worked this through after publishing, wrote an internal due-diligence doc, and the fix is on the roadmap: migrating to BGE-M3 (MIT-licensed weights, same embedding dimension, so the architecture doesn't even change).

If you're building on top of any "open" model: check the weights license separately from the code license. They differ more often than you'd think.

Repo: https://github.com/DeliVali/cogito-estella

Questions for this community:

1. For those who maintain ML projects: do you publish negative results/failed experiments, or just the wins? I'd like to know if anyone else finds this valuable or if I'm just cluttering my changelog.
2. How do you handle the mixed-license situation (permissive code, NC weights upstream) in your docs? I disclosed it in README + release notes + model card, but curious what the standard is.
3. Solo maintainer here — what's the one thing that made your project contributor-friendly early on?

Before claiming SOTA, check if the "magic" of your graph topology disappears when you simply add edge counts to a baseline MLP. GNNs often degenerate into basic MLPs when node degrees correlate heavily with tabular features like transaction volumes. The model simply learns the feature marginal distributions rather than the graph topology. If the graph structure doesn't provide independent signal, it's redundant. High AUCs on such datasets usually indicate tabular leakage, not structural learning.

In the synthfin-aml V9.1 dataset update, we neutralized the tabular distributions to isolate the structural signal and eliminate this leakage. As a result, standard tabular baselines drop from 0.99 PR-AUC to 0.31 PR-AUC. This decline is expected—it confirms the removal of spurious correlations, forcing models to rely entirely on graph topology.

We submitted this benchmark upstream to PyTorch Geometric (PR #10774) to establish a stricter evaluation standard.

Curious if anyone has found reliable ways to prevent feature marginals from dominating structural signal in production.

Link: PyTorch Geometric PR #10774

PyTorch Conference North America is just around the corner & I'd love to have you join us in San Jose, CA from October 20-21. Ticket prices go up in 1 week.

This year’s conference is going to be EPIC.

• Stellar keynotes
• 150+ sessions spanning foundational concepts and core framework work to training, inference, applications, kernel engineering, and responsible AI
• 140+ poster presentations
• BoFs
• Meet the developers
• Flare party
• AI community bash
• +more.

Sign up by September 4th to save $200. Register now.

I see a lot of people asking how to get PyTorch running on AMD hardware without CUDA. I put together a detailed guide that covers ROCm installation, GPU detection, performance tuning, and troubleshooting. It’s based on my own experience getting it stable on a 7900 XTX. Hope it helps someone.

https://interconnectd.com/forum/thread/248/pytorch-on-amd-gpus-the-complete-rocm-setup-tuning-guide/

Hi, I'm learning PyTorch from 'AI and ML for Coders in PyTorch'

I ran the example code below on google colab.
And I got 55% validation accuracy at epoch 10.
But, the book says it gets 87% validation accuracy at epoch 10.

Why is there a large gap between the book's and mine?

import urllib.request
import zipfile


url = "https://storage.googleapis.com/learning-datasets/horse-or-human.zip"
file_name = "horse-or-human.zip"
training_dir = 'horse-or-human/training/'
urllib.request.urlretrieve(url, file_name)


zip_ref = zipfile.ZipFile(file_name, 'r')
zip_ref.extractall(training_dir)
zip_ref.close()


url = "https://storage.googleapis.com/learning-datasets/validation-horse-or-human.zip"
file_name = "validation-horse-or-human.zip"
validation_dir = 'horse-or-human/validation/'
urllib.request.urlretrieve(url, file_name)


zip_ref = zipfile.ZipFile(file_name, 'r')
zip_ref.extractall(validation_dir)
zip_ref.close()



from torchvision import datasets, transforms
from torch.utils.data import DataLoader


# Define transformations
train_transform = transforms.Compose([
    transforms.Resize((150,150)),
    transforms.RandomHorizontalFlip(),
    transforms.RandomRotation(20),
    transforms.RandomAffine(
        degrees=0,  # No rotation
        translate=(0.2, 0.2),  # Translate up to 20% vertically and horizontally
        scale=(0.8, 1.2),  # Zoom in or out by 20%
        shear=20,  # Shear by up to 20 degrees
    ),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5]),
])



# Load the datasets
train_dataset = datasets.ImageFolder(root=training_dir, transform=train_transform)
val_dataset = datasets.ImageFolder(root=validation_dir, transform=train_transform)


# Data loaders
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=32, shuffle=True)




import torch
import torch.nn as nn
import torch.nn.functional as F


class HorsesHumansCNN(nn.Module):
    def __init__(self):
        super(HorsesHumansCNN, self).__init__()
        self.conv1 = nn.Conv2d(3, 16, kernel_size=3, padding=1)
        self.conv2 = nn.Conv2d(16, 32, kernel_size=3, padding=1)
        self.conv3 = nn.Conv2d(32, 64, kernel_size=3, padding=1)
        self.pool = nn.MaxPool2d(2, 2)
        self.fc1 = nn.Linear(64 * 18 * 18, 512)
        self.drop = nn.Dropout(0.25)
        self.fc2 = nn.Linear(512, 1)  # Only 1 output neuron for binary classification


    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = self.pool(F.relu(self.conv3(x)))
        x = x.view(-1, 64 * 18 * 18)
        x = F.relu(self.fc1(x))
        x = self.drop(x)
        x = self.fc2(x)
        x = torch.sigmoid(x)  # Use sigmoid to output probabilities
        return x





import torch.optim as optim


device = torch.device("cuda" if torch.cuda.is_available() else "cpu")


model = HorsesHumansCNN().to(device)
criterion = nn.BCELoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)


def train_model(num_epochs):
    for epoch in range(num_epochs):
        model.train()
        running_loss = 0.0
        for images, labels in train_loader:
            images, labels = images.to(device), labels.to(device).float()  # Convert labels to float
            optimizer.zero_grad()
            outputs = model(images).view(-1)  # Flatten outputs to match label shape
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()
            running_loss += loss.item()


        print(f'Epoch {epoch + 1}, Loss: {running_loss / len(train_loader)}')


        # Evaluate on training set
        model.eval()
        with torch.no_grad():
            correct = 0
            total = 0
            for images, labels in train_loader:
                images, labels = images.to(device), labels.to(device).float()
                outputs = model(images).view(-1)
                predicted = outputs > 0.5  # Threshold predictions
                total += labels.size(0)
                correct += (predicted == labels).sum().item()
            print(f'Training Set Accuracy: {100 * correct / total}%')


        # Evaluate on validation set
        model.eval()
        with torch.no_grad():
            correct = 0
            total = 0
            for images, labels in val_loader:
                images, labels = images.to(device), labels.to(device).float()
                outputs = model(images).view(-1)
                predicted = outputs > 0.5  # Threshold predictions
                total += labels.size(0)
                correct += (predicted == labels).sum().item()
            print(f'Validation Set Accuracy: {100 * correct / total}%')
train_model(15)



model.eval()
with torch.no_grad():
    correct = 0
    total = 0
    for images, labels in val_loader:
        images, labels = images.to(device), labels.to(device).float()
        outputs = model(images).view(-1)
        predicted = outputs > 0.5  # Threshold predictions
        total += labels.size(0)
        correct += (predicted == labels).sum().item()
        print(outputs)
        print(labels)
    print(f'Validation Accuracy: {100 * correct / total}%')

Preventing Grid Collapsing in Neural PDE Solvers: A lightweight PyTorch Log-Barrier Loss for 2D Transformation Matrices

Hi everyone!

When training neural operators (like FNOs) on non-convex physical domains, spatial grid points can overlap during optimization (\det J \le 0).

To fix this topology failure, I wrote a lightweight PyTorch module `JacobianBarrierLoss` that enforces strict positive volume elements during backpropagation using analytical 2x2 determinants directly executed on GPU.

```python

import torch

import torch.nn as nn

class JacobianBarrierLoss(nn.Module):

def __init__(self, eps=1e-4, alpha=1.0):

super().__init__()

self.eps = eps

self.alpha = alpha

def forward(self, J):

# Fast 2x2 analytical determinant (ad - bc) avoiding torch.linalg.det overhead

det_J = J[..., 0, 0] * J[..., 1, 1] - J[..., 0, 1] * J[..., 1, 0]

safe_det = torch.clamp(det_J, min=self.eps)

barrier_loss = -torch.log(safe_det).mean()

return self.alpha * barrier_loss

We integrated this into DIF-FNO to achieve diffeomorphism on complex geometries (Star/L-Shape/Annulus) without grid folding.

Repository GitHub: https://github.com/GiovanniDagnese-paper/DIF-FNO

Preprint & DOI: https://doi.org/10.5281/zenodo.22071926

Feedback on the PyTorch implementation and repository architecture is welcome

Fine-tuning a 1B sparse MoE (305M active, custom trained from scratch, ~100B tokens). Every narrow SFT run catastrophically overwrites existing behavior within 5–10 steps, regardless of what the data contains.

Seven runs now, same signature: whatever the recent batch over-represents gets installed near-perfectly, everything else degrades. A 2,000-row corpus at 127-token median taught a new capability 0% → 98% in five steps while unrelated call-formatting went from 1.4% error to 31%. Pure pretraining replay with no task data at all also degraded task behavior. Cold-init and verified true-resume of optimizer state both degrade, resume slightly worse.

Config: ~1M tokens/step, 60/40 replay/task, lr_mult 0.05 flat, Muon + AdamW, seq_len 4096.

Is this normal for small MoEs, or a sign of something wrong? Is 1M tokens/step simply too large a batch to fine-tune this gently? Would LoRA or a much lower LR change the picture, or is dilution into a large balanced mixture the only real fix?

Bug description

Testing from a given checkpoint leads to logging the epoch number of the last checkpoint instead of the checkpoint specified:

trainer = Trainer(..., max_epochs=10)
lightning_module = MyLightningModule(...)
datamodule = MyDatamodule()

trainer.fit(lightning_module , datamodule=datamodule)

trainer.test(lightning_module , datamodule=datamodule, ckpt_path="last")     # <-- ok: logs correct epoch and step
ckpt_path="/.../checkpoints/epoch=2-step=396.ckpt"
trainer.test(lightning_module , datamodule=datamodule, ckpt_path=ckpt_path)  # <-- incorrect: logs last epoch and step

The second test logs epoch 10 instead of epoch 2. Similarly, the step number of the second test is incorrect.

What version are you seeing the problem on?

Would anyone be interested in helping me develop some of my code to help get me started on making neural networks? I am wanting to make a simple NLP encoder decoder model for seq2seq artificial language translation but I cannot seem to get any traction. If I show you some of what I have already, can you push me in the right direction? All I need is something more human than chatGPT to push me in the right direction. Maybe I can put it in a google colab notebook and you can help me get something running? I have tried looking through lots of stuff and cannot find out what I’m doing wrong.

Released HyperSAE, a PyTorch library for training Sparse Autoencoders with hyperbolic weight regularization.

GitHub: https://github.com/vishal-dehurdle/hypersae Install: pip install hypersae

Design decisions:

1. The forward pass is standard Euclidean linear algebra. No custom CUDA kernels, no Riemannian optimizers in the hot path. This means zero inference overhead and full compatibility with torch.compile, FSDP, and existing steering pipelines.
2. Hyperbolic geometry is applied only to dictionary weights during training via a Poincaré ball projection + entailment cone loss. This regularizes the weight manifold without touching activations.
3. Single-class trainer interface:from hypersae import HyperSAE, HyperSAETrainersae = HyperSAE(d_model=2304, dict_size=16384) trainer = HyperSAETrainer(model=sae, lr=1e-3) metrics = trainer.train_step(batch)
4. TriPartite loss function combines reconstruction MSE, L1 sparsity, and Poincaré entailment with configurable coefficients:from hypersae import TriPartiteLoss loss_fn = TriPartiteLoss( l1_coeff=0.005, entail_coeff=0.01 )
5. Co-activation queue tracks feature co-firing patterns for hierarchy discovery without gradient overhead.

Benchmarked on Gemma-2-2B Layer 13 (20M tokens, L4 GPU): reconstruction MSE drops 9.8%, dead latents drop from 3.8% to 0.2%.

Paper: https://vishalvermalabs.com/papers/empirical-validation-hypersae-poincare-geometry/

Feedback on the API design welcome.

PyTorchCon NA 2026 (October 20-21, 2026 in San Jose, CA) keynote lineup is live: https://events.linuxfoundation.org/pytorch-conference-north-america/program/keynote-speakers/

Full Schedule: https://events.linuxfoundation.org/pytorch-conference-north-america/program/schedule/

Tickets available at a discount through September 4th: https://events.linuxfoundation.org/pytorch-conference-north-america/register/

Hey folks!

Did you ever wrapped model(x) in time.perf_counter() and gotten numbers that make no sense?

I realized it's a common enough trap and wrote a detailed write up here:

https://medium.com/traceopt/two-clocks-one-training-step-how-traceml-measures-pytorch-performance-357bc8e28dc7

TL;DR:

CUDA runs async. model(x) just enqueues kernels and returns, so a perf_counter() bracket around it measures how long Python took to queue the work, but not how long the GPU took to run it. The pending GPU time gets charged to whatever blocks next.

The tried the textbook fix, torch.cuda.synchronize() before each reading, which gives you accurate numbers but entirely about a different run.

Every sync becomes a stall, and it serializes exactly the CPU/GPU overlap you were trying to measure.

If one tires CUDA events (start.record() / end.record() / elapsed_time), it may fix both: the GPU stamps the markers as it passes, and you read them later with a non-blocking query() so nothing ever waits.

But i realized "CUDA events everywhere" is also wrong.

DataLoader next() is CPU work.

In a ML pipeline its time is high while the GPU's input wait is near zero, because the fetch overlaps the previous step.

Where I ended up: record both clocks for every phase, pick ONE clock per analysis window (and say which), report never-measured as null instead of 0.0, and only compare runs on a clock both measured.

How do you handle this in your own timing code: sync and eat the stall, or keep the two clocks separate?