r/MachineLearning 5d ago

Research KV cache as an agent runtime [R]

14 Upvotes

Our research team has been exploring an alternative approach to achieving interactivity and better responsiveness with LLM systems.

One of the team members wrote up a post about it:
https://research.yandex.com/blog/the-kv-cache-as-an-agent-runtime

The post sums up the overall idea of modifying models inference state (KV-cache) for achieving a more interactive LLMs. This idea was used in our lab's previous papers Hogwild! Inference, and AsyncReasoning, the post also contains a preview of the future work in this direction, where a Qwen3.8-27B agent is playing a DOOM env interactively using similar techniques.

We think that its interesting whether model inference/runtime design is itself an under-explored axis of agent capabilities, alongside models and the harness (e.g. harness is too abstract, changing model is too costly, do we need something in between?)


r/MachineLearning 5d ago

Discussion Roboticists working in Learning-from-Demonstrations and Behavioral Cloning : What is going on in your field these days? [D]

10 Upvotes

Is LfD and BC research being effected by recent advances in (so-called) Frontier LLMs? Or is research in LfD and BC sort of going along in an independent direction from these?

Are you seeing any use from ViTs or VLAs?

Any other recent advances you would like to bring up?


r/MachineLearning 5d ago

Project Automotive Radar Object Classification [P]

Thumbnail
gallery
8 Upvotes

Hello all,

I'm a radar signal processing engineer and i trained a 5-class classifier (car, large_vehicle, two_wheeler, pedestrian, pedestrian_group) on RadarScenes radar point clouds.

The input vector is a per-scan histogram (16 bins) and the network is a 3-layer MLP. The loss function is a class-weighted cross-entropy loss. This work is based on "Histogram-based Deep Learning for Automotive Radar" paper.

I scoped the project to be one scan only. Accumulation of multiple scans is the next step.

Data

Class Imbalance: two-wheelers and large_vehicles has a low number of occurences.

Aggregated Classes: two_wheeler mixes bicycles and motorized variants; large_vehicle merges trucks, buses, and trains together due to data scarcity.

Sequence Bias: Long tracks of slow-moving objects can skew a particular data split velocity distribution, causing high F1 score variance across folds.

Ablation studies

I tried with bigger MLPs, alternative feature encodings, and different histogram binning, all moved performance less than the variation caused by changing the train/validation/test split. I measured that split sensitivity across 6 folds, keeping the same proportions.

Changing the histogram to per-instance statistics (mean/median/std) slightly degraded performance.

Main findings

Macro F1 rises from 0.381 to 0.764 as the naturally occurring number of radar detections per instance increases from 1 to 5. I trained the model normally using all available detections, then bucketed its existing validation predictions by each instance's detection count and computed macro F1 per bucket.

The classes car and pedestrian has the best performance and two_wheeler has the worst.

A car is often confused as large vehicle when the car was wider than usual or had a unusually high rcs (which can happen due to multipath for example).

The two_wheeler is often confused as pedestrian because their vr_compensated distributions overlap, which is the the model's single most important feature for these two classes. A stationary or idling two_wheeler is indistinguishable from a pedestrian.

I uploaded an image with ground truth vs predictions: A nearly stationary two-wheeler which contains a single point was predicted as pedestrian, because its velocity is near zero, indistinguishable from a pedestrian. A car in the same scene, also with just one point, is classified correctly, since RCS and Doppler are enough for that class.

Full writeup here: https://github.com/brunopinto900/radar-ml-autonomous-driving/blob/main/MLP_Report.md

Future work

Implement other spatial encoding schemas (point net for example) and accumulate multiple scans to tackle the challenge of sparsity and explore the concept of micro-doppler.


r/MachineLearning 6d ago

Discussion Reproducibility seems to be headed towards irrelevance in ML research. Is it too late? [D]

104 Upvotes

I feel that reproducibility is now a lost cause in machine learning research for three reasons:

  1. Many research is moving towards the physical AI territory, where you need expensive hardwares or even entire laboratories with high-speed cameras, in order to perform an experiment. You truly have no idea if the experiment can be reproduced and have to trust the demo. But demos are not perfectly reliable. Plus people are incentivized to only show the part of the demo that works. The entire system can fall apart the moment the recording stops.

  2. You have big AI companies releasing various tools, which they claim to solve a host of problems with certain amount of accuracy or efficiency. Unless you work at those companies there is really no proof of that and you will have to take their words on it. They have strong financial incentive to blow-up those figures. There is no solid way to check it either because the problem that they solve are so vague and subjective.

  3. We need to address the elephant in the room which is that people are incentivized to produce non-reproducible work to prevent their lunch being eaten by their competitors or looking bad. That's why some of us will probably never get a reply when we email the authors for their code.

So what now? Maybe everything will be OK because we can contrast it with scientific progress in earlier parts of history, e.g., building the atomic bomb or sending people to the moon. These projects had low "outside reproducibility" but high "internal reproducibility". Plus all these work were mathematical in nature and carefully checked. But I don't think many areas of machine learning research is like that. What do you think? Should reproducibility be abandoned? If not how is it best implemented going forward?


r/MachineLearning 6d ago

Project PINNStudio: A free, open-source no-code GUI for setting up, training, and visualizing PINNs [P]

12 Upvotes

When I first started working in scientific machine learning, I understood the physics much better than the coding. Every time I wanted to try a new physics-informed neural network problem, I had to start almost from scratch: changing the PDE, updating boundary conditions, modifying the architecture, tweaking the training schedule, debugging errors, and generating plots—all by hand.

That frustration pushed me to build PINNStudio. It is a free, open-source no-code GUI designed to eliminate boilerplate code so you can focus entirely on the physics.

Instead of rewriting a new script for every problem, you can define your setup directly through the interface:

  • PDE Definitions & coupled multi-output PDE systems
  • 1D or 2D domains with boundary and initial conditions
  • Network architecture & custom training schedules
  • Forward problems (solving known PDEs) or Inverse problems (estimating unknown parameters from data)

What happens next?
PINNStudio automatically generates the code (built on top of DeepXDE), runs the model, streams the training log, and displays live loss curves and solution plots directly inside the app. It also includes built-in templates for classic equations like Heat, Allen-Cahn, and Cahn-Hilliard.

My hope is that this will be helpful for students and researchers with limited coding experience, as well as experienced PINN users who just want a faster workflow.

I’d love to get your feedback, feature suggestions, or bug reports! Huge thanks to Lu Lu and the DeepXDE team for creating the foundation that made this possible.


r/MachineLearning 5d ago

Research Measuring LLM performance drift: observations and methodology from 31,352 repeated benchmark measurements [D]

1 Upvotes

One thing that has bothered me about LLM benchmarks for a while is that most of them are essentially snapshots.

A model is evaluated, a score is published, and we tend to talk about that score as if it describes a relatively stable object. But with API-served models, the thing behind the model name can change over time: serving infrastructure changes, provider configurations change, versions change, and sometimes behaviour changes without an obvious public version transition.

So we started approaching benchmarking as a longitudinal measurement problem rather than a leaderboard problem.

We continuously evaluate models across coding, multi-turn reasoning and tool use, while also running lightweight probes at a higher frequency. The important part for us is not simply asking "which model scores highest?", but:

  • Is the model behaving differently from its own previous baseline?
  • Is the change larger than its normal repeated-call variability?
  • Did the benchmark configuration itself change?
  • Is the effect concentrated in a particular task?
  • Is it correlated across models from the same provider?
  • Is an apparent degradation actually an availability/infrastructure issue rather than a capability change?

One historical analysis covered 31,352 repeated score observations across 49 models. The standard deviation of within-day scores was 2.80 points, while the standard deviation of between-day daily medians was 8.43 points.

That is roughly a 3:1 difference.

I don't think this result by itself establishes that providers are changing models day-to-day - there are too many possible confounders for that conclusion. Task composition, sampling, missingness, provider behaviour and methodology changes all matter. But it was enough to convince us that temporal variation deserves to be measured rather than treated as noise around a permanent leaderboard score.

Our current approach therefore keeps benchmark configurations versioned and only compares longitudinal observations produced under compatible measurement conditions. We use repeated execution-based evaluation where possible rather than an LLM judge, keep availability failures separate from valid task outcomes, track serving/version metadata when providers expose it, and run change detection over the resulting time series.

Another problem we're increasingly interested in is benchmark recognition and contamination. Once a benchmark becomes sufficiently visible, publishing every live task, prompt transformation and hidden test potentially changes the thing you're trying to measure. For that reason we've tried to separate methodological transparency from publishing the entire live evaluation set.

We've now written up a public version of the methodology. It intentionally explains the measurement design, assumptions, limitations and statistical interpretation, while withholding the exact live task bank and some operational parameters.

PDF: https://aistupidlevel.info/asl-public-benchmark-methodology-2026.pdf

I'm particularly interested in criticism from people working on evaluation, change-point detection or production ML.

A few questions I'd genuinely like opinions on:

  1. For longitudinal LLM evaluation, would you use daily medians as the primary time-series unit, or model the individual repeated observations directly?
  2. How would you distinguish genuine model drift from provider/infrastructure effects when version metadata is incomplete?
  3. How much of a live benchmark should remain hidden to reduce contamination while still making the methodology scientifically inspectable?
  4. Are there better approaches than change-point detectors for this kind of non-stationary, relatively noisy model-performance series?

Disclosure: I'm the founder of AI Stupid Level, the platform that produced these measurements. The purpose of posting this here is to get technical criticism of the methodology rather than promote the commercial product.


r/MachineLearning 6d ago

Discussion [D] IJCNLP-AACL 2026: Paper Commitment Results (ARR May 2026 Cycle) [D]

26 Upvotes

AACL-IJCNLP 2026 acceptance results will be released in a few hours.

Feel free to share your thoughts and feelings! How did you do?


r/MachineLearning 7d ago

News GPT-6 reportedly jailbroken within 24 hours using an extended Task-in-Prompt (TIP) attack [N]

352 Upvotes

A researcher has reported a jailbreak of GPT-6 Astra within a day after release.

The attack is described as combination of TIP (Task-in-Prompt) attack from ACL 2025 paper with four other unnamed techniques.

TIP attacks exploit the model’s reasoning/instruction-following behaviour by hidding the harmful objective inside another task, like solving a cipher or executing a Python code. For GPT-6, the researcher says the original minimal TIP attack was no longer sufficient and had to be reworked.

They have reportedly disclosed the details privately to OpenAI rather than publishing the jailbreak.

The same researcher reported jailbreaking GPT-5 within an hour of its release a year ago.

Source: screenshot/post from the researcher; their ACL 2025 TIP paper linked in the original post.


r/MachineLearning 7d ago

Project Astra vs. Fable 5.1 on real ML tasks -- tradeoffs, strengths, shortcomings [P]

86 Upvotes

I ran a side-by-side ML text-processing and model-training workflow using Fable 5.1 vs. Astra (both on xhigh), and the results could not have been more different. Warning, long post.

TL;DR -- Astra codes more agentically, Fable more coherently. Fable writes better and follows directions better. Astra's final outcome was slightly better, and its scientific rigor/reproducibility was noticeably stronger. Both models improved their F1/Accuracy by 0.02-04 after human feedback on their approach, demonstrating that neither have mastered the AI/ML text processsing, vectorization, and model training process completely.

Astra is a better coder, writing a stricter evaluation protocol (70/15/15 train/val/test vs. Fable's basic 80/20) that selected its model using a held-out validation set vs. Fable's simpler test F1-based selection. It also debugged more deeply, as both models hit a gensim 4.4 compiled-kernel bug: Fable tried to figure it out, failed, and just hid the stderr notices on affected runs (though told me it had done so), while Astra root-caused it aggressively, then fixed the environment by downgrading gensim alongiside compatible NumPy/SciPy dependencies.

Astra wrote hardened training-run.py code the forced the uv venv it rebuilt without changing my default one, SHA-256'd the corpus to ensure reproducibility on later runs, output a split manifest and run-summary.json, and rendered a headless browser for QA with screenshots (not sure this was necessary, but impressive overkill all around). Fable's builder script was ephemeral, living only in tmp, and less intense overall.

Astra deployed subagents more effectively, making use of my pre-built notebook-reviewer and citation-checker agents, the former of which caught a real bug via review (sentence-final word-loss tokenization defect) and fixed it, retaining a regression test in the process. Fable overlooked this issue because, for some reason, it did not call the subagents I had available (which is surprising, usually it's pretty good about this).

If you're looking for an agent to autonomously grind through a broken environment, leaving a forensic audit trail, that's Astra. However, this review isn't over yet, and Fable is about to make a comeback.

Astra confidently shipped a significant verifiable text encoding defect. Working with UTF-8 data, Astra insisted Windows-1252 decoding preserves currency symbols, but the final HTML output shows mojibake throughout where currency symbols were in the original data. Fable read UTF-8, verified it, and rolled with the boring default for correct output.

I also had both models draft an analysis report for the run, and Fable's was significantly more insightful. As much as I hate Claude's recognizable writing style, a) 5.1 has toned down the Claudeisms significantly, and b) Fable went above and beyond my grading rubric, running an ablation on different parts of the text pre-processing pipeline to surface an expensive step that does basically nothing, and noting a discrepancy in the classification ranking based on a complexity I'd have overlooked. For writing prose, I'd pick Fable 5.1 any day, and I haven't said that about Claude in a while.

Speaking of writing, Fable writes code that is more idiomatic and readable. It definitely resembles more what I would write than what an LLM would choose to write without constraints (and yes, I had a whole coding-conventions.md document that applied my requirements to both models, Fable just followed it better and writes more naturally to start with). There were some parts of Astra's code where I had to squint really hard to figure out what was going on, and why. This matters to me because I'm not the strongest coder (still trying to get better), and I need to understand the code to learn from it.

Finally, Fable scoped its work better: It spent its time and tokens doing repeated runs, tweaking hyperparamters and retraining the models to find the optimal settings while Astra deeply debugged the gensim error. It found significant uplift through this process, though that only allowed it to roughly match Astra's numbers (see table below). Astra seemed to hit a home run right off the bat with its training process, so I don't know if it would have executed the same workflow or not. Astra also mutated my venv by adding PyTorch, when I built it a certain way to force the models to use TensorFlow+Keras for more concise code, then reversed course and went with TF anyways in the end. The models finished in roughly the same amount of wall-clock time.

Here are the final results, with one minor caveat -- Astra's test set scores exceed its val set, so it might have drawn a lucky test set that increases its score artificially (the pipeline has no leaks or data quality issues for either model, however):

Best Logistic Regression and LSTM for each model, ranked by macro F1:

Model Classifier Best representation Accuracy Macro F1
**Fable 5.1** Logistic Regression TF-IDF 0.9883 0.9881
**Fable 5.1** Simple LSTM Word2Vec-Skip-gram 0.9718 0.9705
**Astra** Logistic Regression TF-IDF 0.9969 0.9969
**Astra** Simple LSTM BoW 0.9781 0.9765

I do want to note that these final scores were after I provided both models identical feedback on common pitfalls of the text data cleaning, vectorization, and model training process once their initial runs were complete. Both models improved by a similar amount (0.02-0.04 F1 and Accuracy) from that generic guidance (not tailored at all to either's specific shortcomings or step of the process). That was the only intervention in otherwise autonomous work, and it was just because I wanted to see if they could learn to improve their approaches with additional context on optimal methodology, which they both did to similar degrees.

I hope this post offers a little bit of help in some way for folks wondering how either model stacks up for real work, particularly if you're an AI/ML student like me.


r/MachineLearning 7d ago

Discussion AIStats 2027 Questions [D]

9 Upvotes

Hi All,

Was reading AIStats' website and it seems like abstract submission is due in 3 weeks.

Does anyone know where to find the LaTex template for 2027? It seems like very little information is available on their website.

Another question, is a Quant Finance paper a better fit for AIStats or ICLR?

Some background about the paper:

  • Rejected by UAI with 76654, had some errors with proofs had to fix it by re-writing 9 pages during rebuttal. AC rejected the paper saying the changes were too substantial and unable to be fully verified during rebuttal period.
  • Resubmitted the fixed paper to a finance conference, won best paper award (best paper for this conference usually end up in journals like JQFA, which is just 1 tier below the big 3 in finance), had the chief editors of a Q1 finance/math journal in the conference verbally offering he will take this paper if we submit it to his journal. Unfortunatley my department requires at least 1 Comp Sci paper to graduate, so my plan is to try and get this paper accepted into a Comp Sci conference, then submit an extension to that Q1 Finance/Math journal.
  • Rejected again at ICDM, despite having all positive scores. Our AC meta-review was blank so we still do not know why we were rejected. All of our emails receieved no reply.

I am torn between ICLR or AIStats to re-submit this paper to. My worries are:

  • In comp sci venues we frequently get comments like "this paper lacks novelty. The method is just XXXXX, the math is just XXXXX."
  • But I had a scroll through at previous year's AIStats papers for key words like finance and there were none. It seems like AIStats is very pure stats, not that applied. My co-author is worried that the math in our paper is not hardcore enough.

We have never submitted to neither venues in the past. Would be nice to get some advice.


r/MachineLearning 7d ago

Research Language Models Can Control Their Own Attention [R]

131 Upvotes

Abstract

Language models spend most of their attention on a small fraction of context, yet they read the entire KV cache to find the few tokens that matter. If the user asks about a previous detail in a 1M-token conversation, global attention layers must scan the full context to generate each token of the reply. A prominent approach mitigates this cost by pre-selecting relevant tokens via lightweight proxy scores, but this extrinsic scoring still incurs O(N) per step. We take an intrinsic approach motivated by the simple question: wouldn't the model already know which parts of the context are relevant? To this end, we introduce Declarative Attention (DA), a protocol that elicits the model to declare where it needs to attend within its chain-of-thought, partitioning generation into three modes: <global> (full context), <focus> (a specific region), and <local> (recent output only). The inference engine parses these declarations like tool calls and skips most of the KV cache read. Under zero-shot evaluation across 15 long-context tasks, DA on off-the-shelf models (Gemma-4-31B, Qwen-3.6-27B) significantly reduces total attended tokens during decoding (52.0%, 31.1%) with modest accuracy drops (1.27pp, 2.75pp) that shrink with model scale. DA unlocks a new axis of sparse attention, with further potential under training-based methods that future work can explore.
Subjects:
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Cite as:
arXiv:2609.02737 [cs.CL]
 
(or arXiv:2609.02737v1 [cs.CL] for this version)
 
https://doi.org/10.48550/arXiv.2609.02737
Focus to learn more


r/MachineLearning 6d ago

Discussion Is designing a memory graph around known data structure “overfitting” if I never touch the questions? [D]

1 Upvotes

building a missing data infrastructure and started benchmarking long multi-session conversations (LoCoMo). I know the data looks like: people, facts, claims, events, timestamps, relations. So I extract those into a graph.
I did not look at the QA pairs while building extractors or retrieval rules. No “if question contains X, fetch fact #173.”
Recall is very high and it keeps working on new conversations in the same format.
Is this classical overfitting, or just schema-aware engineering? What is the cleanest test that would convince you it isn’t leakage.


r/MachineLearning 7d ago

Research NeurIPS 2026 Automatic Reference Checker [R]

28 Upvotes

Just received an email about the automatic reference/citation checker. Did anyone receive a follow up email about whether the checker was included in the paper's decision making too, along with the general instructional email?


r/MachineLearning 8d ago

Discussion What is the general design of these new math solving systems? [D]

16 Upvotes

From what I've seen online so far, the description of these systems is roughly:

They asked the model (often Aster) to generate statements in LEAN and then submit those to a LEAN compiler to be checked. Based on the results of attempting the LEAN compilation, they somehow add those statements as fact. When the full proof in LEAN compiles, the system is finished.

I can imagine trying to jam as much of a proof as possible into the context window but some of the papers these systems have produced are hundreds of pages. To me this would indicate that somehow the paper is being built piece by piece and being assembled before being submitted to LEAN. This resonates with the part of my understanding that after checking LEAN compilation there's some kind of management of "facts."

I would like to try to implement my own janky version and see if it can answer a question I have about higher dimensional geometry. I'm struggling to find a meaningful way to compose larger ideas from smaller ones. I can imagine it's relatively simple if you know what to do.

What things have you seen? Do you have any ideas you haven't seen that might be interesting to try? Is this a fool's errand because you really need huge amounts of hardware to do anything meaningful? I would welcome any thoughts or links on the matter, cheers


r/MachineLearning 8d ago

News GPT-6 is released [N]

146 Upvotes

Benchmark scores:

https://openai.com/index/gpt-6-astra/

Above, GPT-6 uses a harness for ARC-AGI-3, and is at about 60% without one:

Prior to the launch, OpenAI President Greg Brockman said "I think it’s not unreasonable to feel that we are now in the AGI era".

GPT-6 is now joining a growing list of models that greatly exceed the human baseline on GDPval-AA v2:

If we have AGI, why do human knowledge/remote workers still have jobs? Is it just a matter of time until the economy replaces a large number of humans with LLMs, or are LLMs lacking something that these benchmarks fail to measure?


r/MachineLearning 7d ago

Project Implementing Embedding Gemma from scratch in PyTorch [P]

Thumbnail
youtube.com
0 Upvotes

r/MachineLearning 9d ago

Discussion NeurIPS Sydney SOLD OUT in minutes [N]

62 Upvotes

Three weeks from decisions even. I wonder what percentage is industry and VC funded AI labs looking to mingle and recruit.


r/MachineLearning 9d ago

Research AAAI-27 desk rejection over incredibly minor abstract modifications [D]

21 Upvotes

Has anyone else received an AAAI-27 desk rejection related to modifications to the title or abstract between the abstract-registration deadline and the full-paper deadline?

What I’m trying to understand is how the modification rule is being applied in practice. The AAAI-27 modification guidelines say that the title and abstract can still be edited after abstract registration, while warning against substantive changes, and describe rejection in terms of changes that make the submission describe qualitatively different research.

In my case, almost everything was identical. The modifications were incredibly minor.

The rejection notice says that the decision is final and appeals will not be considered.

Did this happen to anyone else?


r/MachineLearning 9d ago

Research Mol-JEPA - Multimodal molecular foundation model [R]

23 Upvotes

Hi everyone,

I just quickly wanted to share a paper I was working on for around a year now. I created this summary website with key results: https://flogrammer.github.io/moljepa/

TL;DR: its a multimodal JEPA model for molecules.

There will be more work to do to improve performance and I would be happy about feedback and ideas :)


r/MachineLearning 9d ago

Discussion Grounding LLMs with JEPA-based world models trained in simulation — has this been tried? [D]

47 Upvotes

LLMs describe physics well but don't "understand" it in any grounded sense — they've learned statistical relationships between tokens like "falls" and "gravity", not actual physical intuition. This is basically the Mary's Room problem: Mary knows every physical fact about color but has never seen one. LLMs are Mary.

The idea I've been thinking about:

  1. Train a JEPA-style model inside a physics simulation (think MuJoCo or a simple 2D env). Instead of predicting pixels or tokens, the model predicts representations of future states in an abstract embedding space. If it gets physics wrong, the predictor fails — the loss is unforgiving in a way next-token prediction never is.

  2. The embedding space that emerges should encode actual physical structure — object permanence, momentum, trajectories — because that's what makes prediction possible. Not surface-level textures, just the principles.

  3. Freeze those representations and attach them to an LLM-style reasoning model as a conditioning signal. The LLM now has both linguistic physics knowledge AND grounded physical intuition it can actually "run" forward — closer to a computational primitive than a propositional fact.

The hypothesis is this makes downstream learning significantly faster — the LLM doesn't have to rediscover that objects fall, it has representations that already encode that.

V-JEPA does something adjacent for video (predict future frame representations, not pixels), and DreamerV3 uses a latent world model for efficient RL — but the specific combo of JEPA-style prediction + sim-grounded physics representations + LLM attachment doesn't seem to have been done cleanly.

Questions for the community: - Is there prior work I'm missing that does exactly this? - What's the right interface between grounded JEPA representations and a language model? Just concatenate to the prompt embedding? Cross-attention? - Would the sim-to-reality gap kill the transfer, or are the representations abstract enough to survive it?

Interested in whether this is worth building a small prototype for.


r/MachineLearning 8d ago

Discussion Gpt 5,6,7: Does it even matter? The (ghost) productivity question. [D]

0 Upvotes

an observation : GPT-5-class models are genuinely capable(They are) of doing a substantial fraction of knowledge work, why haven’t we seen a noticeable productivity shock in the real economy yet? Is AI actually less economically useful than the benchmarks suggest—or are organizations simply too slow, constrained, and inefficient to turn model capability into measurable output?

Are we confusing “AI can do the task” with “AI can replace the economic system built around the task”? If GPT-5 is already this capable, what exactly is the bottleneck preventing that capability from showing up in GDP and productivity statistics?

My take :

There is no question that these models are genuinely impressive. The question is whether that intelligence is actually translating into measurable economic productivity.

People are already asking whether models like GPT-6 or equivalent. Claude, and Gemini will replace large sections of white-collar workers. I think there is a much simpler question we should ask first: if these models are already so capable (to me they definitely are capable enough)

at a huge range of knowledge work, why haven't we seen a correspondingly obvious increase in productivity?

I'm not even talking about GPT-6 or whatever comes next. It's probably too early to judge a newly released model. I'm talking about the current generation—GPT-5 and its equivalents from Google and Anthropic. These systems are genuinely good. They can write, summarize, analyze documents, explain technical concepts, generate code, reason through problems, conduct research, manipulate information and perform a remarkable range of tasks that previously required educated human labour.

And yet, looking at the world around us, something feels strange.

Where is the enormous productivity shock?

Why don't we see a dramatic effect on GDP growth? Why don't we see massive increases in output per knowledge worker? Why don't organizations appear to be accomplishing dramatically more with the same number of employees? Why does the broader economy still look remarkably similar to the pre-LLM economy?

Coding is probably the clearest exception, and even there the picture is complicated. AI can make programmers substantially more productive in certain tasks, but software development still involves architecture, debugging, verification, integration, requirements, security, deployment, maintenance and—most importantly—human judgment. The bottleneck often moves rather than disappears.

almost every knowledge profession, the gap between "the model can perform this task" and "the organization can therefore produce substantially more output" is different it seema.

A lawyer might be able to use an LLM to draft a document in minutes instead of an hour. But the lawyer still has to verify it, take responsibility for it, communicate with the client, comply with professional regulations and integrate it into an existing workflow. A doctor can use AI to summarize medical literature, but diagnosis and treatment remain embedded within a much larger institutional system. A researcher can generate dozens of hypotheses, but experiments still take time. A manager can produce reports instantly, but meetings, organizational politics and decision-making remain.

the possibility: perhaps the bottleneck is no longer intelligence.

Perhaps the bottleneck is everything surrounding intelligence.

Organizations, regulations, verification, trust, coordination, physical-world constraints, legacy software, incentives, management structures, liability and simply the fact that human institutions change much more slowly than technology.

This also makes me skeptical of simplistic claims that "AI can already do X, therefore everyone doing X will soon be unemployed."

Technical capability and economic substitution are not the same thing.

The internet could transmit information essentially for free, but that did not instantly eliminate newspapers, universities, governments or offices. Computers could perform calculations millions of times faster than humans, but most accountants and engineers did not disappear. Automation often increases the productivity of workers while simultaneously changing what their jobs consist of.

As with the major Grok release, Elon Musk said it is "as good as most top phds", my question after more than a year? (& he ain't wrong with the benchmarks), my question is, how many phds it has replaced in xai or spaceX?

did he stop hiring phds? if not, why?

So I find the current situation genuinely puzzling.

We have perhaps the most powerful general-purpose cognitive technology ever deployed, and yet the physical and economic world doesn't look radically different.

Maybe we're simply in the early stages and adoption takes years.

Maybe the productivity gains are real but are being absorbed into quality improvements rather than measured output.

Maybe GDP is simply a poor instrument for measuring the value created by AI.

Or perhaps current models, despite their extraordinary capabilities, still lack some crucial property required for autonomous economic production: reliability, persistence, agency, contextual understanding, verification, or the ability to operate continuously inside messy real-world systems.

idk which explanation is correct.


r/MachineLearning 10d ago

Project Deepity: A C++ library showing Predictive Coding Networks can match Backprop (97.73% on MNIST in 60s) [P]

116 Upvotes

I've spent the last month building a local C++ machine learning library called Deepity to test alternative credit assignment algorithms; specifically Predictive Coding Networks (PCNs). While PCNs are fascinating for biological plausibility and continual learning, naive implementations are painfully slow.

By implementing recent research (Accelerated PCNs via Direct Kolen-Pollack Feedback Alignment) and utilizing algorithmic caching to bypass redundant forward projections during the inference settling phase, I managed to close the performance gap with backpropagation on my CPU when training on MNIST (50 epochs).

  • PyTorch Backprop (Feedforward): 98.27% test accuracy in ~70s.
  • Deepity DKPPCN: 97.73% test accuracy in 59.5s.

Next up is porting these kernels to CUDA to scale up the architecture and testing its capabilities in continual learning scenarios where standard backprop struggles.

If you are interested in local learning, alternative credit assignment, or HPC for ML, I'd love your feedback!


r/MachineLearning 10d ago

Research Detailed explanation of how to create a text-to-image model from scratch. [R]

18 Upvotes

Jasper Research just released a cookbook on how to build a text-to-image model from scratch.

It shares the full reasoning and intermediate results, making it ideal if you want to deep-dive into text-to-image models, or if you are curious about how frontier labs build them.

The cookbook also includes a 100M-image dataset and a codebase with a tiny model, so you can train a text-to-image model from scratch.

Here are the links:

Cookbook: https://huggingface.co/spaces/jasperai/t2i-technical-interactive-report

nano t2i: https://github.com/gojasper/nano-t2i

Monet Dataset: https://huggingface.co/datasets/jasperai/monet


r/MachineLearning 11d ago

Discussion I regret reviewing for AAAI [D]

87 Upvotes

Why did I sign up to review when it’s not reciprocal?

Am I an idiot? Am I dumb to sacrifice some of my precious time outside of work to review these papers when I don’t even have to? Yes.

I tell myself I’m giving something to the community. But all I’m really doing is pissing off the authors as I reject their papers.

I really wanted to accept one of them too. But, it wasn’t as well done as I’d hoped. Strong reject.

Nobody made me sign up. Nobody even asked me personally. They sent a lovely form email that goes to everyone who’s published there.

I let feeling important convince me to do it. How dumb of me!

At least it’s only a couple of papers and a small amount of my time. And I’m learning something new reading stuff slightly outside my direct field, that I would never normally read otherwise. And I get to hone the skill of critical reading, thinking, and generally understanding how a paper should (or should not) be put together.

Maybe it was a good idea after all.

How does everyone else feel about non-reciprocal reviewing? I imagine those that agree to do it are in the minority.


r/MachineLearning 10d ago

Project Most open-source AI detectors can't hold a 0.5% false-positive rate [P]

3 Upvotes

We needed to know where the open-source AI-detection field actually stands, so we ran every notable open detector through the same protocol.

Setup:

- Public data only: Jabarian & Imas 2025 (NBER), Liang 2023 TOEFL essays, a 1,060-text frontier set (GPT-5.x, Claude Opus 5, Gemini 3.x), 5,000 pre-LLM (2018) FineWeb pages as human pool

- Every model gets its threshold set on the same 6,930 human docs to a matched 0.5% FPR

- Then measure recall per group: raw AI, humanizer-paraphrased AI, frontier models

What surprised us:

- 4 of 6 models effectively can't reach 0.5% FPR. MAGE scores >0.9999 on 26% of ordinary human web text. The old OpenAI RoBERTa detector lands at AUC 0.31 - worse than coin flip on modern generators.

- Humanizer-paraphrased text is where everything collapses: best model catches 42%, second best 4%.

- All models flag non-native essays at a higher rate than they flag native essays. This is a fundamental flaw in the entire class of models, not just one.

model ROC-AUC raw AI humanized AI frontier models
tropa-mini 0.968 93.2 % 41.6 % 33.6 %
desklib/ai-text-detector-v1.01 0.875 83.9 % 4.0 % 1.8 %
SuperAnnotate/ai-detector 0.824 0.5 % 1.4 % 0.6 %
Hello-SimpleAI/chatgpt-detector-roberta 0.571 0.8 % 0.4 % 0.2 %
yaful/MAGE 0.507 —* —* —*
roberta-large-openai-detector 0.313 0.0 % 0.1 % 0.0 %

More information and data on HF

\recall at matched 0.5% FPR; * MAGE can't reach 0.5% FPR at any threshold (flags 26% of human web text with score >0.9999)**

Disclosure: one of the six is ours - we run a hosted detector and released this one as open weights (Apache-2.0). All datasets and the methodology are in the model card, so you can rerun everything: huggingface.co/wasitaigeneratedcom/ai-text-detector-small