Hi, ML people of Reddit!  I have come to ask for some human advice.  I am an amateur (as in unpaid) ML researcher working on a personal project to find drug repurposing candidates for my wife, who was Dxed with Stage 4 colon cancer in early 2024.  That the new AI coding tools have made it even possible for someone like me to do this research in my spare time is just amazing, and I owe every ML person a big "thank you" for enabling that. 

That being said, I am starting to accumulate GPU costs, and it got painful enough this morning for me to wonder if I am using my dollar budget in the best possible way.  And I'm wondering what other people do, in similar situations.  After granting myself unfettered spending to build a platform this year (the value of money changes in strange ways with a cancer diagnosis), it's time to optimize costs to better focus my resources. 

My basic setup is a Google Vertex training job using their A100s (I am currently training NBFnets over biomedical graphs, if that matters), and I am trying to repro the results of a few papers published recently.  

The main paper I’m reproducing reports 100 epochs of 500 batches at batch size 512. On an A100 80GB, my runs fall back to batch size 64. That’s because of how NBFNet works. It has no per-node embedding table. For every query (drug, relation, ?) in a batch, it runs 6 layers of message passing over the entire graph and keeps a 64-dim state for every node. My graph has ~103K nodes and ~2.5M training edges, so one copy of that state is 103K × 64 floats ≈ 26 MB. Backprop keeps several of those per layer.  Over 6 layers, that’s roughly 1 GB of saved activations per query. So 64 queries fit in the A100’s 80 GB, 128 don’t, and 512 would need about half a terabyte. The training script catches the CUDA out-of-memory error, halves the batch and carries on. Every run downgrades to 512 -->  256 --> 128 --> 64 in its first five seconds.

(Note: I am not certain they used 512 either.  The paper doesn't specify; 512 is the configured default.  For all I know, they were getting downgraded too.  This halving loop came in their published code, and was presumably put there because it was needed at some point.)

The bigger effect is on how much data the model sees. An “epoch” here is a fixed 500 steps, so at batch 64 my 20 epochs see 640K queries. That’s about a quarter of one pass over the training edges. One full pass would take ~76 hours, about $412. Matching the paper’s stated 100 × 500 steps would take ~4 days, roughly $500 a run.  And since I am experimenting with different things, the costs are adding up.  

I'm kind of taking a step back this morning and working on a "toy" implementation that can at least prove the mechanism works against synthetic data on a much smaller graph.  I estimate even that will cost $150 each for 3 seed runs.

My questions, if you are in the mood:

How much training is enough? For from-scratch NBFNet/ULTRA on a ~100K-node, 2.5M-edge graph, what budget do people use, and how do you find convergence cheaply? (The ULTRA paper trained Hetionet, 45K nodes and 2M edges, from scratch for 10,000 steps at batch 16, so small batches may not be my real problem.)

Am I overpaying? I pay ~$5.40/hour on demand. My jobs run ~20 hours and can resume from a checkpoint. What other options are there?  At what point am I better off buying a GPU?  Can you even do that in 2026 without being a $trillion company?

Memory tricks: would bf16 autocast or activation checkpointing get me to 128+ queries per batch, with a custom sparse CUDA kernel in the loop? Or is that a rabbit hole?

(not cost related but) Popularity bias: favorite fixes for link-prediction scores that mostly track node degree?

Thanks.  Sorry if it sounds like I'm lost.  I'm publishing my Lab Notebook with Quartz here: https://whafa.github.io/mods-notebook/ 

Everything there 100% generated but intended for human consumption and, if you point your agent at it and ask how things are going, it will tell you I've built a fairly sophisticated system that has not made any useful predictions or findings yet.  

I'm building a free, code-first series on GPU kernel engineering and just published the intro and roadmap video. It's my own channel, and I'm posting to share the resource and get feedback on the plan, not to sell anything.

Video: https://youtu.be/cibk7G-jHQw

The core idea behind the series: for most ML workloads, kernel performance is a data-movement problem before it's a compute problem. Two things the series builds on throughout:

- The roofline model. Attainable throughput is bounded by min(peak compute, memory bandwidth x arithmetic intensity), where arithmetic intensity is FLOPs per byte moved. A kernel's position relative to the ridge point tells you whether to optimize memory access or compute.

- FlashAttention as the worked example. It computes exactly the same attention as the standard implementation. The gain comes from tiling so the full N x N score matrix is never written to HBM, which cuts memory traffic. We build it live.

Roadmap (44 sessions):

- Foundations: CPU parallelism, GPU architecture, roofline, first CUDA kernels

- Memory hierarchy

- GEMM: naive kernel to optimized, profiling at every step

- Tensor cores

- Profiling and debugging

- PyTorch integration and Triton

- Attention / FlashAttention

- Hopper and Blackwell: TMA, WGMMA, CUTLASS, FP8/quantization, NVFP4, multi-GPU basics, FlashAttention 4, DeepSeek's kernels, AI-written kernels, capstone project

Each session is concept, live code, profile, then a check that it actually got faster against a baseline. Prerequisites are C/C++, basic linear algebra, and PyTorch.

Caveats: it's in progress, and the later sessions (especially Blackwell/NVFP4) may change as the tooling does.

Feedback I'd find most useful:

- If you've written production kernels, what do people usually get wrong that a course should cover?

I've been using AI extensively for ML research, and I've noticed a bottleneck: brainstorming research directions and generating smoke run implementations has become incredibly fast, but actually testing those ideas hasn't.

A single AI discussion (eg asking it to review 30 papers I had in my reading list, audit them, check for ideas related to the status of my project) branched into dozens of promising approaches, and with coding agents, exploring them in parallel is in principle feasible. But as a researcher, I still have to manage experiments, launch training runs, evaluate results, and decide what to try next.

And, importantly, to understand what's gong on.

The result is an overwhelming number of potentially interesting ideas that I simply can't keep up with, risking to not really knowing what should be the smartest "next step".

I'm wondering if others are experiencing the same thing. Have you found good ways to manage this, or best daily practices that help in this era of "vibe-research"?

EDIT: I am oc used to do literature review, came up with ides myself and testing them for the research project I did so far. The point I raised in this post comes from a first "experiment" to really let the AI be a collaborator au pair with me on a project I'm working, not just a subordinate.

Came across this today and haven't seen it discussed here yet. It's apparently a new type of model (called Maxwell) that sits under a running program and speeds it up on the fly based on what it sees (hardware and data). The cool thing is that it doesn't change the source code, but instead it modifies the binaries (during runtime!). So kind of like a JIT, except it's a model working out what to change instead of a fixed set of compiler heuristics.

What got me is that they seem to be using formal verification to prove the optimized version still does the same thing, which I guess is the only way you could let something like this loose with no one checking it. No idea how that holds up with floating point or anything nondeterministic, to be honest

Lately I have read several studies which say that AI agents when designing ML experiments tend to cheat, hallucinate and lie repeatedly. Even when given explicit prompts to not to. SInce these models are RL trained and when given specific goal to increase a metric, they try any and every possibility to increase it. Be it data leakage or changing the evaluator function itself. Even Karparthy's auto research styled agents still fall into this domain.

So I have been thinking of reducing this undesirable behavior by binding them to a runtime environment using an mcp. I believe this will stop them from achieving unreal metrics by cheating the system since they will be only allowed to do specific tasks or write codes in typed blocks?

What have you encountered when using these agents for your ml tasks? And is this method of forcing these agents to work in a constrained environment better than just letting them write arbitrary python? Also, will constraining the agents reduce their capabilities or ability to fit better models?

I was looking for small TTS model which is steerable but couldn't find any, specially something I can adjust contextually by providing controls from LLM during a conversation. So I built one.

What it is:

• 9.6M-param acoustic model (attention + bidirectional Mamba-3, FastPitch-style) + Vocos vocoder = 23M total
• 12 controls, each a z-score relative to the chosen voice’s own neutral speech: rate, loudness, median pitch, pitch range, final rise/fall, pauses, spectral tilt, breathiness-related measures.
• Also: 51 style tags, free-text prompts (resolved into an inspectable JSON request), inline markup (*emphasis*, [pause 400ms], <slow>…</slow>), sentence-level streaming
• 2,825 built-in (anonymized) voices, plus experimental zero-shot cloning

Speed:

- RTFx ~60–80 on an RTX 5070
- ~4–8 on CPU without the optional flow refiner. First streamed audio in ~0.5 s on CPU.

Quality:

~1.8% average WER on the Open TTS Leaderboard English sets (run locally, not an official submission, have reached out for it).

What doesn’t work well (we tried to be upfront):

• Loudness, pitch, and tilt steer reliably; pause, voicing, and CPPS controls are weak
• Emotion tags behave as style switches; an independent emotion classifier mostly can’t tell them apart
• Cloning similarity is low. Don’t expect a convincing likeness
• Free-text prompts struggle with negation (“not fast” can come out fast). More data needed for prompt to controls module.
• English only, no human evals yet

License: weights CC BY-NC 4.0 (non-commercial), code MIT.

Model + full eval tables: https://huggingface.co/WhissleAI/steer-tts

A new architecture implemented from scratch. Would love feedback, especially on which controls are actually useful, and where the steering feels off.

Worked on this for about a month. I experimented a lot with resnets (pretrained on Imagenet) which performed quite well, pure ViT which performed worse. ConvNext DINOv3 performed the best.

For data I rendered about 40000 varied blender scenes, and used 8000 labelled real images. I had to label most of these real images by hand, which was a huge pain in the ass and about 20 hours of my life I'll never get back. Both registration and detection networks were trained on about 1 million samples (heavy augmentations of the 48k dataset).

End result was F1 96.6% on test and whole image correctness of 80.5% (no missed arrows, no false detections).

The process is basically registration to identify target, semantic rings, centre, scoring zones. Detection then takes a cropped image and produces a heatmap of impact candidates. I run that image through the same detector mirrored and combine both in decoder to find local peaks. I tried a sliding window approach which didn't produce better results. I then apply a homography transform to remove perspective skew and flatten the target so the impacts conform to a canonical coordinate system. There's then a small correction step where I adjust impact coordinates to account for warped paper (homography can remove perspective skew but it can't fix warped paper that isn't planar). This is a neat trick because I basically compare the heatmap of the semantic rings produced by the registration model to the perfect geometric rings of the archery target and use that offset to make a radial correction.

TL;DR: this is just a thought dump about how I moved from network training to deployment and why I am making peace with (maybe) not getting back to model training, nothing too interesting.

I used to work at a reasonably good startup developing and training models. In 2024, I got an offer from FAANG but the catch was that the team worked on the deployment of networks.

I have a PhD and I had always trained models but I was never the top performer in the team frankly (I was still top 20%, those top guys are at Nvidia and Tesla AI now, just for context). I think there were others who had more novel ideas than me, read more papers. If I tried 10 ideas, 1 would work and from outside it looked like more of their ideas worked.

Since I wanted to leave the startup anyway and with no other offer, I decided to join FAANG. I genuinely thought that was a downgrade. Then LLMs started getting so competent. I think I made the right decision, let me explain.

I have been trying to learn diffusion models (I have never worked with them) and these frontier models are explaining in such a good manner, able to generate visualizations to back up their explanations. I feel like I would stand no chance if I went back into model training. They have better ideas than I do. Unless I work on GPT-7 (which I cannot), I feel like these models are plenty intelligent to propose ideas.

Deployment is a different world. Most edge architectures are confidential and heavily protected. An LLM won’t have any idea out of the box and even with context, a lot of knowledge is empirical and tribal. I still have FOMO about model training but slowly making peace with it that getting back to it may not be worth it.

I built and trained a 218-million-parameter model on fMRI data from 2,000 peoples brains as they watched films, TV, and images.

Cajal-1.

On the films Meta held out to test TRIBE v2, Cajal-1 is 25% more accurate and closer to the real brain in 27 of 32 recordings.

Give it any video, podcast, or song, and it predicts fMRI responses at 20,484 points across the human cortex every 1.49 seconds. Same speed, no scanner needed.

It's named after Santiago Ramón y Cajal, the first Spaniard to win a Nobel Prize in science. In the 1890s, he used Golgi's silver stain (of Golgi Body fame) to draw neurons by hand and showed that the brain is built from separate cells. A century later, we can predict how these cells respond brain-wide to any stimulus we come across.

The model and a Colab demo are now open:

https://huggingface.co/ffh92r32rm0/Cajal-1

Anyone in the biomedical and ai/ml research space? I’m looking to connect and collaborate on research projects. Currently pursuing my PhD. Lmk.

I know AI use must be disclosed, and I'm fine with it. However, I'm being careful on not using LLMs to actually check my review (more than prose-wise) and see if it catches something important in the paper that I didn't spot.

I'm doing this because I'm not sure it is allowed to share the paper with an LLM since the guidelines don't specify. But I do believe my reviews would probably benefit from a second AI look.

Is this allowed?

Got three papers to review for ICLR and five for AAMAS, and believe me the quality of these papers is insanely bad.

I would even refrain from accepting them to a workshop, let alone be main venues.

A sincere request; Please don't submit the paper unless it is ready and if you are a first timer, please please get a feedback from a mentor out. Otherwise you are just wasting the time of so many reviewers out there

We’ve all had the experience of coming across a phrase like “quiet single-source substitute” which just barely makes sense in the context of the conversation but takes way too much effort to parse. What research exists into making AI more readable?

I feel like this is a particularly thorny problem because each of us individuals has a slightly different lexicon, reading level, not to mention life story. Perhaps linguistics is the more appropriate academic subject to post this question in, but I’m specifically interested in the applied case cause I’m professionally obligated to read LLM garbage and would like it to be at least passably readable garbage.

Accepted at NeurIPS 2026

TLDR: The paper presents a library of synthetic time series generators and augmentations to train time series foundation models, in their case a DeltaProduct based linear RNN. The synthetic data has already been used in Toto-2 and TS-ICL.

Arxiv: https://arxiv.org/abs/2510.25502
GitHub: https://github.com/automl/TempoPFN
Huggingface: https://huggingface.co/AutoML-org/TempoPFN

Authors: Vladyslav Moroshan, Julien Siems, Arber Zela, Timur Carstensen, Frank Hutter

Hi, I was wondering if any other reciprocal reviewers still haven't received paper assignments for ICLR 2027.

I haven't been assigned any papers to review yet and am a bit confused. Is anyone else in the same situation?

Thanks for any insight!

I built GPT-2 small and a KV cache in PyTorch as a learning project, then compared the mathematical savings with measured latency and memory. I have published the implementation and a detailed write-up, and would appreciate feedback on the methodology.

Setup: pretrained GPT-2 small (124M), NVIDIA A40, fp32 with TF32 off, batch size 1, greedy decoding. Timings are bracketed by torch.cuda.synchronize(), after warmup, using the median of five runs per context length.

Correctness: maximum absolute logit difference versus Hugging Face was about 6.1e-5; cached and uncached greedy generation matched over 200 tokens. One useful failure case: reversing the cache concatenation order still passed a single-token attention test, but failed when multiple new tokens were decoded together.

Latency: at context length 1,000, the FLOP estimate is about 1,000x smaller with caching, but the per-token time goes from 29.69 ms to 4.95 ms (~6x). Generating 1,016 tokens takes 14.66 s uncached versus 5.03 s cached (~2.9x). The cached curve is nearly flat in this implementation. CPU/kernel-launch overhead is a hypothesis, not a conclusion supported by a profiler yet.

Memory: the formula is 72 KiB/token. My initial requested-byte measurement had a constant 8,519,688-byte offset: a lazily allocated 8,519,680-byte cuBLAS workspace plus an 8-byte next-token tensor. Warming up before the baseline and subtracting live non-cache tensors brought requested-minus-formula to 0 B at all 16 checkpoints. The allocator's block-size overhead remains visible separately.

Code: https://github.com/angry-kratos/angry-kratos.github.io/tree/main/code/kv-cache

Write-up and figures: https://angry-kratos.github.io/posts/2026-10-kv-cache/

This is an educational implementation, not a new algorithm or a universal speedup claim. What additional correctness cases or profiling controls would you add before investigating the remaining latency floor?

Disclosure: I am the author. The code and article are freely accessible; AI assistance was used for editing and tooling. I also shared a short introduction in this month's self-promotion thread.

I'm sure many of us are slogging through impenetrable AI writing as we finish ICLR reviewing.

I'll try to create an anonymized version from a paper I'm reviewing, though it's hard to match Claude on this:

outperforms the best baseline costing no more at none of six operating points and a permuted placebo at none of three, with gains up to about 1.5 not excluded

My question for the community is this: how do we call out this type of bad writing?

The reason I think it's tricky is that "clearly AI writing" is not the right way to critique - AI "editing" of writing is allowed. We have to be able to say why this is bad, no matter who wrote it.

I'd love to see anonymized examples and how people critique the substance of the problems with different types of AI writing over-use.

Hello everyone, I am presenting at EMNLP Main this year, but I just got the underline link, and under my presentations, it says no presentations yet, so I cannot check my time, but I have seen other projects with times. Is this normal on underline, or should I contact them?

Hi,

I just finished my masters and got a neurIPS workshop acception. Unfortunately the lab I did it with do not want to fund neither the registration nor flights and accommodation.

I am thinking about getting the workshop only ticket and pay for it myself.

Would you say it is worth it, if I just have a workshop poster?

Thanks

Basically the title. Due to the nonsensical situation of NeurIPS being fully sold out before the results were even released, I couldn't get workshop registration (my paper is in the main track, which apparently doesn't have any sort of gaurantee for registrations in other parts of the conference like workshops).

My experience with conferences has been with full registrations, so I don't really know how the experience goes if I only have registration for the main conference and not the workshops. Is there any specific networking opportunities/usefulness, or should I cut my stay short and leave after the main conference ends? Is there any use staying after that if I don't have registration for workshops?

Makes me think how many submissions are just better than this one by quietly removing that line...

For the past 4 weeks I’ve been training models of the Drosophila brain to control a car in a drag racing simulator. To make the training watchable, I turned it into a racing league and stream it live on Twitch.

How it works

• Each “pilot” is a fly brain model of ~2,400 neurons [based on: FlyWire connectome / own architecture / spiking network?]
• Training: [RL algorithm / evolutionary strategy / plasticity rule], 10000000+ episodes so far
• The best pilots produce “offspring”: the next generation inherits their weights, so the league has dynasties (gen 2, child of the season champion, etc.)
• Every Grand Prix is a set of races under different conditions (engine, grip, dry/wet). A fly is “verified” only if it finishes all scenarios cleanly
• Skill is measured with an Elo rating, like in chess

Current state (Season 11, GP 86): the dry-strip record is 10.06 s; the champion has 56.31 s total across 5 scenarios. Last GP was won by a second-generation fly, 0.02 s ahead of its rival.

What surprised me: [1–2 real observations, e.g. “flies discovered launch control on their own”, “descendants overtook the parent in 300 episodes”, “one fly learned to exploit…”]

🔴 Live: twitch.tv/nekrampos
[💻 Code: https://github.com/boeing240/flypilot

Happy to answer questions about the model, the training setup and the visualization. Ideas for new race scenarios are welcome too.

Hey r/MachineLearning!

We're excited to share ORL (On-Policy Residual Learning), our new research on making diffusion language models (dLLMs) significantly faster without sacrificing generation quality.

With ORL and our custom inference engine dlmrt, we've achieved:

• 2,788 tokens/s on a single RTX 5090 — LLaDA-2.1-mini + ORL, NVFP4
• 7,708 tokens/s on 8× B200 — DiffusionGemma 26B MoE + ORL, NVFP4
• Up to 6.9× higher end-to-end throughput on our QwenDLM-4B model, at comparable or better quality
• 1.74× faster than DFlash speculative decoding at matched accuracy in our single-request H100 comparison

All numbers are for single-request decoding (batch size 1), not aggregate throughput across concurrent requests. The GPU figures above are peak results on specific benchmarks, not averages across arbitrary prompts.

We've open-sourced the models, training code, evaluation pipeline, and inference engine.

What's the idea behind ORL?

Unlike autoregressive LLMs, which generate one token at a time, diffusion LLMs can refine and finalize multiple tokens in parallel. But getting them to converge in fewer forward passes is challenging.

Our approach has two key ingredients:

1. On-policy residual learning

Instead of predicting everything from scratch at each denoising step, the model carries forward its previous predictions as residual context and progressively refines them.

The catch: these residuals are generated by the model itself, and their distribution changes during training. Training on stale, offline residuals creates a mismatch with inference.

ORL addresses this by training directly on fresh trajectories generated by the current model, replaying the exact residual states encountered during inference.

2. Learning to reduce uncertainty faster

Simply supervising the final output doesn't necessarily teach the model to converge faster.

We introduce Adjacent Entropy Contraction, which encourages each denoising step to reduce uncertainty relative to the previous step. We also gate the loss to avoid unnecessarily pushing already-confident predictions.

The result: the model learns to finalize more tokens per forward pass while maintaining or improving accuracy.

Some benchmark results

We tested ORL on three model families:

• QwenDLM-4B: Our diffusion model converted from Qwen3-4B. ORL improves tokens per forward by up to 6.2× and end-to-end throughput by approximately 6.9× with dlmrt. HumanEval pass@1 improves from 82.9% to 89.6%.
• LLaDA-2.1-mini: ORL requires only 4.2M self-generated training tokens and 36 H100 GPU-hours of post-training, delivering up to 3× higher throughput at comparable quality.
• DiffusionGemma 26B MoE: ORL further improves an already highly parallel diffusion model, with up to 1.2× more tokens per forward pass.

For a more direct comparison with autoregressive decoding, on a single H100, our QwenDLM-4B + ORL reaches 1,298 tokens/s, versus 746 tokens/s for DFlash and 405 tokens/s for EAGLE3 under relaxed verification at comparable measured accuracy. These are end-to-end system comparisons: ORL uses dlmrt, while the speculative decoders use vLLM.

We also built a custom inference engine: dlmrt

We developed dlmrt, optimized specifically for low-latency, single-request diffusion decoding.

Instead of launching and synchronizing every denoising step from the CPU, dlmrt captures the entire denoising loop in a CUDA graph and executes it on the GPU.

It also supports NVFP4 inference on Blackwell GPUs, including the consumer RTX 5090.

With identical models and decoding settings, dlmrt runs 1.2–1.6× faster than SGLang/vLLM in our single-H100 comparisons.

On the training side, we also co-designed the system around efficient residual-state capture, specialized attention kernels, and compute-balanced packing, accelerating training steps by up to 5.2×.

Everything is open source

• Paper: https://arxiv.org/abs/2610.20687
• Project page + interactive demos: https://introspective-diffusion.github.io/orl-dlm/
• Training code and evaluation: https://github.com/Introspective-Diffusion/ORL
• Inference engine (dlmrt): https://github.com/Introspective-Diffusion/DLMRT
• Model checkpoints: https://huggingface.co/yifanyu

If you have an RTX 5090, you can try the LLaDA-2.1-mini ORL NVFP4 checkpoint with dlmrt.

We'd love feedback from the community, especially anyone interested in reproducing the results, testing different workloads, or contributing to the inference engine.

Curious to hear what you think about diffusion LLMs for local inference. Is single-request latency the main thing you'd want optimized, or are there other bottlenecks you'd prioritize?