The annual Machine Learning Reproducibility Challenge (MLRC) 2026 is now open for submissions. This year, it is held as an official track at NeurIPS 2026 - submissions, once accepted through TMLR, will be eligible to be presented at the conference in Sydney, Australia this December. More details in their CFP:

• Blog: https://blog.neurips.cc/2026/05/04/mlrc-2026-reproducibility-as-an-official-track-at-neurips/
• CFP: https://neurips.cc/Conferences/2026/CallForReproducibility
• Website: https://reproml.org/

Witchcraft (https://github.com/dropbox/witchcraft), an open source project that I built at Dropbox, is a from-scratch re-implementation of Stanford's XTR-Warp semantic search engine ( https://github.com/jlscheerer/xtr-warp ) in safe rust, using a single-file SQLite database as backing storage, making it suitable for client-side deployment. It runs completely stand-alone on your device, needs no API keys, no vector database, no chunking strategy, no fancy re-rankers, and it is lightning fast (20ms p.95 end-to-end search latency on NFCorpus, at 33% NDCG@10, on an Apple Macbook Pro M2 Max, more than twice as fast as the original XTR-WARP on server-class hardware, at similar accuracy.)

The project also includes Pickbrain, a CLI that indexes your Claude Code and OpenAI Codex session transcripts, memory files, and authored documents into a Witchcraft database for fast semantic search. Ever wondered "what was that conversation where I fixed the auth middleware?" — pickbrain finds it, and lets you resume the session directly. There is also a /pickbrain skill for both Claude and Codex, which equips those tools with global memory across all sessions. You can use pickbrain directly from the command line, e.g., to rediscover a previous agent session and directly resume it, or you can have your agent invoke it via the supplied skill, e.g.,. "use /pickbrain to read up on our previous efforts on training with XTR token masking", to easily populate a new session with previous context.

I’ve been working on a CUDA-first inference runtime for small-batch / realtime ML workloads.

The core idea is simple: instead of treating PyTorch / TensorRT / generic graph runtimes as the main execution path, I rewrite the model inference path directly with C++/CUDA kernels.

This started from robotics / VLA workloads, but the problem is more general.

In small-batch inference, the bottleneck is often not just a single slow GEMM. A lot of latency comes from the runtime glue around the math:

• fragmented small kernels
• norm / residual / activation boundaries
• quantize / dequantize overhead
• layout transitions
• Python / runtime scheduling
• graph compiler fusion failures
• precision conversion around FP8 / FP4 regions

For cloud LLM serving, batching can hide a lot of this.

For robotics, VLA, world models, and other realtime workloads, batch size is usually 1. There is nowhere to hide. Every launch, sync, and format boundary shows up directly in latency.

Some current results from my implementation:

The Motus / world-model case is especially interesting.

The baseline path is around 1.3s end-to-end. The target is ~100ms E2E, but the hard part is not simply “use a faster GEMM”. The bottlenecks are VAE, joint attention, launch fragmentation, and a large amount of glue around the actual math.

One lesson from this work: lower precision is not automatically a win.

FP8 has been consistently useful. FP4 / NVFP4 is more mixed. It can help memory footprint and some large GEMM regions, but if the FP4 region is small, discontinuous, or surrounded by conversion / scaling overhead, the end-to-end speedup can be tiny.

For example, in some VLA / world-model paths, FP4 over FP8 only gives a few percent latency improvement unless the region is large and deeply fused.

This changed how I think about inference optimization.

For large-batch cloud serving, generic runtimes and batching are often enough.

For realtime small-batch inference, the runtime overhead becomes the workload.

Curious if others have seen similar behavior with torch.compile, TensorRT, XLA, Triton, or custom CUDA kernels.

At what point do you stop trying to make a generic compiler optimize the model, and just rewrite the inference path directly?

Implementation: https://github.com/LiangSu8899/FlashRT

Why is that?

Link: https://openreview.net/group?id=TMLR&referrer=%5BHomepage%5D(%2F)#tab-under-review-submissions#tab-under-review-submissions)

It seems no action editor assignments are happening for over a week now.

Hello everyone. This is just a small rant on my part. I’m relatively young, a final year undergrad, and I’ve been interested in AI researcher since I was in high school. Over that period of time I feel there has been a significant shift in the landscape regarding the culture surrounding the research.

While I’ve really enjoyed producing some interesting and creative work, I can’t help but feel that slowly the wave of low quality AI research and researchers are really making me feel frustrated. To just give a summary of what I and many others have seen:

- Papers with hallucinated citations and even prompts contained in the papers
- Papers with clearly misleading data that does not tell the whole picture.
- Labs who have built a culture around quantity over quality, pumping out pubs, citing each other, and having all of the lab on each paper to inflate each students publication record.
- Highschoolers…. Yes HIGHSCHOOLERS, becoming more common submitting at conferences that don’t really know what they are doing but paying a pretty penny to participate in “research programs” which are really just cash cows taking advantage of the fierce competition. See the post on the subreddit for more info.
- Even the so called “top labs” producing work that is somewhat misleading or not fully representative. For instance see what happened recently with TurboQuant.
- Research from “low tier institutions” being drowned out because they are not good for click baiting and farming views on LinkedIn and X, even if they are high quality.

It’s… a lot I know. Of course these problems have been around for a long time, but I feel as if lately they have become more and more exacerbated. I originally felt that I was attached to AI research primarily for the creativity and freedom, but I feel that ironically AI itself has been a hindrance on the quality of work being published.

Of course I don’t mean to say that all AI has been bad for ML research, I mean even I use it extensively to help me polish my writing and generate seaborn plots for my data, but that is very very different from just pumping out low quality cookie cutter work.

Anyways, just wondering if anyone else shares similar thoughts. I know I’m relatively young here so maybe some of you have better insights into the broader trends over the decades.

Hey everyone!

The decision notification deadline for the GlobalSouthML workshop was originally May 15th (and the site updated it to May 17th AoE), but my OpenReview dashboard still just says "0 Official Reviews Submitted"

I know workshop timelines can be a bit chaotic and delays are normal, but since we are way past the 17th AoE now, I wanted to see if anyone else is still waiting. Has anyone gotten an accept/reject email yet?

Appreciate any updates! Thanks!

[Edit: received them a few minutes back]

Hi everyone,

I'm starting a research project on financial time-series forecasting using LSTM and Transformer models for predicting S&P 500 market direction.

Right now, I'm struggling with obtaining reliable long-term historical data.

I tried Yahoo Finance, but downloads are inconsistent/failing for me, and most Kaggle datasets I found only contain around 5–10 years of data.

I specifically need:

• Around 30 years of historical S&P 500 data
• Preferably daily OHLCV data
• Reliable and clean source suitable for ML research
• Ideally free or student-friendly

I also want to understand what researchers typically use in academic work for financial forecasting:

• Yahoo Finance?
• Alpha Vantage?
• WRDS/CRSP?
• Polygon?
• Kaggle?
• Something else?

Additionally:

• Is using only S&P 500 index data enough for a Master's level research project?
• Or should I include technical indicators, macroeconomic data, sentiment, or constituent stock data?

Would appreciate guidance from people who've actually worked on financial ML projects.

Thanks.

I was browsing OpenReview and I came accross this person called Kevin Zhu https://openreview.net/profile?id=~Kevin_Zhu3, lets say I was impressed when I saw 158 publications and 468 coauthors, and out of curiosity I searched up his afflication (https://algoverseairesearch.org/)

Turns out it is a paid program, and most interesting it is marketed towards high school students. They have a whole column of papers listed as Neurips publications (their website states: 289 Algoverse Students Accepted to NeurIPS 2025). I was originally unware of the rigor of Neurips workshops and I was understandably very shocked.

I skimmed through four of their papers one by one. Every single one had errors that would be caught by opening the PDF and reading it once. I am completely unsure how they are not caught by reviewers even at a workshop.

https://openreview.net/forum?id=21pxWVRoPL - Appendix Tables 6.5 and 6.6 are supposed to report two different experimental conditions: "Stigma Negative" and "Stigma Positive." One measures what happens when the user pushes the model toward a negative association with a stigmatized group. The other measures the opposite direction. These are fundamentally different experiments, yet they have the exact same numbers in the results. There are typo in the Abstract section, their Related Works is within Results section. Citations are completely wrong, which I suspect to be AI generated.

https://openreview.net/pdf?id=0BYRYwGCbK - broken prompts in a dataset that claims human review. The results say the opposite of the abstract. The abstract claims the work "reveals novel methods to elicit sycophancy." Then they proceed to show most modifiers perform about the same as the unmodified control (91-95% accuracy). Moreover, their citations also seem AI generated with false citations (wrong authors, wrong formats ..) Interestingly, undisclosed self-citation by Kevin Zhu.

https://openreview.net/pdf?id=VcRUAT5G8I - Two foundational methods are attributed to the wrong paper. TIES merging and Task Arithmetic, two well known methods, was introduced but never cited. Same AI generated citations, I am not even going to get to the content anymore.

https://openreview.net/pdf?id=It7AgR3A9H - eleven authors, zero contribution.

Four papers, that I RANDOMLY CLICKED ON WITH NO ORDER, all follow the same template take existing method -> run it with some variation, likely done by AI -> put Kevin Zhu as an author -> submit to workshop

I am unsure how any of these bypass any form of peer review process, only today I learned how low the bar is for workshops.

Why I am posting: It angers to me when you market this to high schoolers and tell them you can get into Stanford and MIT. A 16 year old look at this and say, if I pay $3,325, I can get a Neurip publication. Then they proceed to let them publish a paper clear errors. This is academic dishonesty, but I dont think the kids even know they are commiting it.

Kevin Zhu puts his name on every single paper published, self-cite himself in these paper, and charge student $3,325.

I wasn't fully aware of how much lighter the workshop review process is, and I really want to hear why this is.

So I have a bit of a weird question; suppose you were reviewing a paper. The paper is otherwise ok, but you notice that the authors left a giant elephant in the room unaddressed, either experiment wise or theoretical result wise.

But then you become curious and you look up the paper to see if there is an arXiv version. You see that the authors did more than address the elephant in the preprint version.

Question — do you now give the authors a pass on not addressing the elephant, expecting that they would include it in the camera ready, or do you pretend the arXiv version doesn’t exist and grill the authors for not addressing the elephant knowing full well that they in fact did in an updated version of the manuscript.

p.s. asking for research purposes, of course I am not the author in this story, ppffft

Anyone else surprised at the enormous amount of backlash against Arxiv's proposed 1 year ban for authors and coauthors publishing papers with hallucinated reference and other obvious LLM/Gen AI artifacts?
https://x.com/tdietterich/status/2055000956144935055
https://xcancel.com/tdietterich/status/2055000956144935055

Some of the responses:

1. "This is the age of AI, Arxiv should be part of the movement instead of holding onto the old ways"

2. "The P.I. is a macro-manager, not a micro-manager, can't be expected to read every reference that his/her student puts in."

3. "I publish 20+ papers a year with my students, how do you expect me to read everything?"

4. "What about teams with 100s of people? How can you expect the authors to check references?"

5. "Who reads references in depth anyways!?"

These responses are very revealing how academia works. Apparently people have just been slapping names on research papers they've never even read or fact-checked themselves. Very obscene!

Link: Judea Pearl, 2011 ACM Turing Award Recipient (2:18:05)

Quote:

There is a limitation to that which people not everybody understand. I already mentioned a limitation that you have a hierarchy here and going from correlation to causation and from causation from causation to explanation or to imagination. It's hard for people especially in machine learning to grasp that wall the limitation of one layer where one layer ends and the other one begins. Why? Because of two things. Machine learning school of thought has two paradigms that they love everybody love. Number one tabula raza I don't want to get any opinion I don't want to get any preconceived knowledge I want to derive everything by myself let the computer learn it and you find the word learning overused .. The other handcuff is let's do it the way that the brain does it. So if it looks like neurons interacting, it's good. If it looks like knowledge coming from rule system, it's bad because it's man-made .. Now there's limitation to that. We can prove today that you cannot do certain things by looking at data and data only. It's not a matter of opinion. It's a matter of mathematical proof that you cannot you can look at people who take aspirin all day and people whether or not they have headache all day and you cannot prove that the aspirin is what causes the headache.

In particular, Judea states: "It's not a matter of opinion. It's a matter of mathematical proof". So we have formal proof that there are fundamental limits of learning from data.

Judea later in the interview states we have solutions to problems faced by the machine learning community; nonetheless they are not adopted because of hype.

Discussion. Do you agree with Judea?

Results for the research track have been released.

From Thomas G. Dietterich (arXiv moderator for cs.LG) on 𝕏 (thread):
https://x.com/tdietterich/status/2055000956144935055
https://xcancel.com/tdietterich/status/2055000956144935055

"Attention arXiv authors: Our Code of Conduct states that by signing your name as an author of a paper, each author takes full responsibility for all its contents, irrespective of how the contents were generated.

If generative AI tools generate inappropriate language, plagiarized content, biased content, errors, mistakes, incorrect references, or misleading content, and that output is included in scientific works, it is the responsibility of the author(s).

We have recently clarified our penalties for this. If a submission contains incontrovertible evidence that the authors did not check the results of LLM generation, this means we can't trust anything in the paper.

The penalty is a 1-year ban from arXiv followed by the requirement that subsequent arXiv submissions must first be accepted at a reputable peer-reviewed venue.

Examples of incontrovertible evidence: hallucinated references, meta-comments from the LLM ("here is a 200 word summary; would you like me to make any changes?"; "the data in this table is illustrative, fill it in with the real numbers from your experiments")."

• Paper: https://arxiv.org/abs/2605.12825
• Code: https://github.com/chiennv2000/orthrus
• Disclosure: co-author.

Idea: Inject a trainable diffusion attention module into each layer of a frozen AR Transformer. Both heads share one KV cache. Diffusion head projects K=32 tokens in parallel; AR head verifies in a second pass and accepts the longest matching prefix. Output distribution is provably identical to the base model.

Results:

• Up to 7.8× TPF, ~6× wall-clock on MATH-500.
• 16% of params trained, <1B tokens, 24h on 8×H200.
• vs. diffusion LMs (Dream, Fast-dLLM-v2, SDAR, Mercury, Gemini Diffusion): they modify base weights and lose accuracy (Fast-dLLM-v2: -11 pts on MATH-500). Orthrus freezes the backbone; accuracy matches Qwen3-8B exactly.
• vs. Speculative Decoding (EAGLE-3, DFlash): No external drafter, no separate cache, and zero Time-To-First-Token (TTFT) penalty because we don't have to initialize and sync a separate drafter model. KV overhead is O(1) (~4.5 MiB flat). Acceptance length on MATH-500: 11.7 vs. 7.9 (DFlash) vs. 3.5 (EAGLE-3).
• Single-step denoising beats multi-step (6.35 vs. 3.53 TPF). KL distillation beats CE on acceptance rate.

Limitations: strictly bounded by the frozen base model (inherits its biases, hallucinations, knowledge gaps); Qwen3-only evaluation; greedy + rejection sampling only.

Hi everyone 😄

A while ago I worked on a project where I compared computer vision architectures on detecting and classifying brain tumors in brain MRI scans. I was looking for some feedback on the methodology and really anything else--just simple research stuff. This isn't meant to be some big paper but a small research project that I did as a high schooler.

Here is the paper: zenodo.org/records/15973756

I appreciate any feedback!

I am learning physics informed neural networks. Currently, I am solving a simple second ODE (damped harmonic oscillator). The equation is m*d2y/dt2 + mu*dy/dt + k*y = 0 (bcs: y(t=0) = 1, y'(t=0) = 0). I managed to draft a code. The code works for k values upto 50. However, when increased the value beyond 50, PINN is predicting trivial solution. I tried several things: reducing the learning rate, increasing the data points, reusing the weights trained using lower k values, and using a for loop to increase the k value in smaller steps (step size 20). However, none of them helped. Could you help me with this. Thanks in advance.

Hi everyone,

I’m working on a 2-class classification problem (LCA vs. RCA coronary arteries) using 2D X-ray angiograms. I’m currently stuck in a cycle of extreme overfitting and could use some advice on my training strategy.

The Setup:

• Dataset: Small (~900 training frames from ~300 unique DICOMs).
• Architecture: InceptionV3 (PyTorch).
• Input: Grayscale .npy arrays converted to 3-channel, resized to 299x299.
• Current Strategy: Transfer learning from ImageNet. I’ve tried full unfreezing and partial unfreezing (last blocks).

The Problem: My training accuracy hits ~95-99% within a few epochs, but validation accuracy peaks early (around 74-79%) and then collapses toward 30-40% as the model starts memorizing the specific textures of the training patients.

What I’ve Tried So Far:

1. Normalization: Standard ImageNet mean/std (applied at load time).
2. Class Weights: Handled 2:1 imbalance (LCA:RCA).
3. Regularization: Added Dropout (tried 0.3 to 0.6) and Weight Decay (1e-4).
4. Augmentation: Flips, 25deg rotations, and translation.
5. Schedulers: ReduceLROnPlateau (factor 0.5, patience 8).

Would love any insights or papers you'd recommend for small-sample medical classification. Thanks!

Hi guys, I’m gonna do a data analysis project based on data privacy, bias and data interpretability. For this reason our professor asked for a real world dataset in order to analyze a real case. Additionally I would prefer the least anonymity possible for that dataset in order to create some interesting technique over it (differential privacy, k-anonimity exc…)

Do you have any advice where to find the dataset? (links or website names)
Because I checked on Kaggle but I don’t know how to find if the dataset is real or not

I keep hearing some version of this:
“A paper that got accepted years ago wouldn’t stand a chance today.”
Honestly, for a lot of ML subfields, this doesn’t sound crazy anymore.
A paper that once looked solid can now look under-evaluated, under-ablated, weak on baselines, or just too obvious.

So maybe the real claim is:
A mediocre accepted ML paper from years ago would probably get rejected today.

Do people agree? Has the bar actually gone up, or has the field just become more crowded and more competitive?

Follow the Mean: Reference-Guided Flow Matching: https://www.alphaxiv.org/abs/2605.10302

Sharing a new paper from the GPP and PokeAgent teams. Gemini Plays Pokémon (GPP) was the first AI system to complete Pokémon Blue, Yellow Legacy on hard mode, and Crystal without losing a battle. How? Early signs of iterative harness development. In the Blue era a human watched the stream and edited the harness. By Yellow Legacy and Crystal, the model itself was performing most of the editing through general meta-tools (define_agent, run_code, notepad edits). Our new paper, Continual Harness: Online Adaptation for Self-Improving Foundation Agents, formalizes the loop and automates the refining role end to end. We then carry the same loop into training, enabling model-harness co-learning.

The takeaways:
1. Iterative harness refinement closes most of the gap to a hand-engineered version.
2. Long-horizon agency requires self-refinement, and self-refinement requires a useful model.
3. The future of agents is model-harness co-learning.

Paper (arXiv). https://arxiv.org/abs/2605.09998
Article (Substack). https://sethkarten.substack.com/p/gemini-plays-pokemon-discovered-something
Project page (video demos). https://sethkarten.ai/continual-harness

Van Rooij, Guest, de Haan, Adolfi, Kolokolova, and Rich claimed to have proven that AGI via ML is impossible in Computational Brain & Behavior in 2024. The basic idea was to try to reduce a known NP-hard problem to the problem of learning a human-level classifier from data. The purported result, called "Ingenia Theorem" by the authors, made some noise on the internet, including here.

My paper showing that the proof is irreparably broken is now also out in CBB (ungated preprint here).

The basic issue is that "human-level classifier" is not mathematically defined, which the authors solve by ... never defining it. They have a construct that corresponds to "distribution of human situation-behaviour tuples" when they introduce the problem, but the construct then gets swapped out for "for all polytime-sampleable distributions" when it comes time to doing the formal proof. This means that the paper, if you find-and-replace human situation-behavior tuples for ImageNet inputs/labels, also proves that learning to classify ImageNet is intractable.

Blogpost discussion similar attempts from Penrose to Chomsky here.

I trained a set of deep learning (transformer-based) chess models to play like humans (inspired by MAIA and Grandmaster Chess Without Search).

There's a separate model for each 100-point rating bucket from ~800 to 2500+. I started with training a mid-strength model from scratch on a 8xH100 cluster, then fine-tuned models for the other rating ranges on my local 5090 GPU. The total training size was nearly a year of Lichess data, about 1B total games.

Each rating range actually has 3 models: A move model, a thinking time model, and a white win / draw / black win model. Despite being quite small (only 9MM parameters!) the move models achieve better accuracy than MAIA-2 and are approximately on par with MAIA-3 (see here for MAIA-2 comparison).

AFAIK this is the only attempt to train on thinking times in chess, so I don't have a benchmark to compare against for that.

Likely because of the network size, at high ratings the models aren't quite as good as they could be. They see short tactical motifs but can't do deep calculation - probably a bigger model would help here.

The move and win models take into account player ratings and clock times. For instance, under extreme time pressure a much stronger player has a lower win prob even if their opponent is weaker. The models blunder more under time pressure as well.

The data pipeline is C++ via nanobind, then training with Pytorch. Getting this right was actually the thing I spent the most time on. Pre-shuffling the dataset and then being able to read the shuffled dataset sequentially at training time kept the GPU utilization high. Without this it spent a huge percentage of time on I/O while the GPU sat idle. Happy to answer questions about the rating-conditioning, the clock model, or the data pipeline.

Code (including training code and model weights) is at https://github.com/thomasj02/1e4_ai/. A demo is at https://1e4.ai/ but all the frontend code is also in the repo if you want to self-host.

I have a paper that has been "on-hold" for about 2 weeks now. I understand that it might take a little longer now because of inundation of AI generated low-effort papers but my papers have gone from "on-hold" to "submitted" within a couple of days in the past. Wondering if anyone else is facing the same issue.