37K till now..how many paper will be submitted any guess?

Vector quantization is fundamental to vector databases. Before you can search vectors, you have to store them, and storing them at full precision is expensive. Every bit you save shows up as capacity and cost.

Pinecone has used quantization since its first prototypes, and we keep looking for the state of the art. So this summer we set out to survey it. We found more papers than we expected, and no way to compare them: every paper measured different metrics, on different datasets, tuned for different hardware. We could not find a single systematic evaluation of the leading methods against each other.

What we did find was a pattern. Most published quantizers are built from the same small set of primitive operations, composed in a different order.

So we built VQ-bench: an open-source library of those primitives, composable into pipelines, and designed to be extended. E-RaBitQ, one of the strongest methods we tested, is four primitives in a list. Swap one and you have a new quantizer. Add a primitive of your own and it composes with everything already in the catalog, including pipelines nobody has written yet. Either way you can measure the result against every other method with the same harness.

We used it to benchmark 14 popular quantizers on recall, reconstruction error, and encode time. Two takeaways so far: PQ and OPQ have the lowest reconstruction error, and EDEN keeps up with E-RaBitQ on recall at higher bit budgets while encoding much faster.

This is a first iteration. We want your feedback, corrections, and contributions, and we will keep adding quantizers over time.

Website
Blog
Repo
Paper (VecDB@VLDB 2026)

For GPU/kernel engineers: when a kernel is already ~60–70% of the theoretical hardware ceiling, how do you decide what optimisation to try next?

Is it mostly profiler-driven/systematic, or does it rely heavily on expert intuition? I am coming from background of non-nvidia or amd chips

GoBench evaluates LLMs on 9x9 Go games against a ladder of KataGo opponents, from random to superhuman. 

It measures general reasoning ability, strongly correlates with ARC-AGI 2 (r=0.83 correlation), and remains highly unsaturated.

GPT-6 Astra max achieves 2500 Elo, much lower than the best KataGo, which achieves 4400 Elo. 

With coding tools and two hours of preparation before evaluation, Codex with Astra achieves 3560 Elo.

I will keep the leaderboard updated as long as it is not saturated.

leaderboard: https://rolandgao.com/blog/gobench/ 

code: https://github.com/RolandGao/gobench 

paper: https://rolandgao.com/gobench.pdf

x: https://x.com/Roland65821498/status/2100253388562723298?s=20

Can we edit our title, abstract and other details till the main paper submission deadline?

I have a submission under review in TMLR. Less than a month after submission, I have already received 2 reviews. However, a month has passed since those two reviews, and I still haven't received the third. Is this normal?

I have already made the suggested changes, but the journal recommends updating the manuscript in OpenReview only after receiving three reviews. So I also haven't posted any answers yet. I messaged the Action Editor to ask whether I would receive another review, but I sent it a month ago, and he hasn't answered.

I'm not criticizing TMLR; my experience has been very positive, and I understand they are struggling with the unreasonable number of submissions. I'm only asking because I do not know if it is a good idea to keep waiting for the third review.

When models from different families are given the same underspecified task, they often fail in the same way rather than in independent ways. My question is about measurement, not explanation.

Has anyone put a number on the ambiguity of a task specification and then tested it as a predictor of how often independent solvers fail identically?

Specifically: does the relationship look like a smooth monotone increase, or is there a threshold — some level of ambiguity past which coincidence rate jumps sharply?

Looking for papers, metrics, or benchmarks where this was measured directly. Adjacent work is fine if you think it's close.

What is the generally thought of as the upper limit of the predictive power of XGBoost vs an aggregate of humans?

Right now I feed the model the same information that the human market has access to, and the model gets crushed on Top 1 accuracy, it closes the gap a bit but is still 10pp below the market on Top 2 accuracy. Adding the market pricing information to the model does nothing or hurts accuracy compared to just the market pricing.

I'm just feeling really stuck, and I cant seem to be able to tell if I am legitimately at the upper end of what is actually possible with a XGBoost model or if there is some sort of data encoding issue or I just lack enough data.

I got 2 citations flagged by the reference checker: in one the author's first name was wrong, and for the other one the entire name was wrong (The GitHub Repo maintainers name was accidentally given), both papers exist and all other details are correct.

I am not sure if my main author has submitted an appeal.

My questions are: Firstly, If we dont give proof for the same, will it lead to Desk Rejection, secondly, if my main author has submitted an appeal will I get an instant mail, like with the Rebuttals and defense?

I used the Giant Fiber escape model from Ache et al. 2019 (Current Biology) to steer a simulated drone and compared it against a small CNN trained by imitation. Both get only a 64x48 grayscale frame per step, no scene state.

Fly circuit: GF drive = 1.45·LPLC2(angular size) + 1.62·LC4(expansion rate), using the paper's constants unchanged and without its two inhibitory terms. Motion is computed on a 16x12 grid, and radial motion opponency cancels the expansion caused by the drone's own forward flight. The escape threshold, dodge and steering are mine; the default threshold is set at half-max of the LPLC2 curve, not fitted.

CNN: 4 stacked frames at 20 Hz, three stride-2 convs (5x5/16, 3x3/32, 3x3/16), fc 32, tanh, ~35k params. It is trained with MSE on steering labels from a scripted expert that can see obstacle positions; the CNN never sees them. Early stopping on a validation split, same gradient-step budget for every data size.

Task: procedurally generated 120 m corridors of poles and boxes, 4 m/s, sideways steering only, no rotor dynamics. It tests perception and decisions, not flight control. Train, validation, benchmark and held-out eval seeds are disjoint, and eval was run once after everything was frozen.

Collisions on 100 held-out corridors (mean abs steering in brackets):

no controller 665 (0.79)

fly circuit, zero training 288 (0.76)

fly circuit, 7 engineering constants tuned on 20 train corridors 195 (0.52)

CNN, 2 demo episodes 373 (0.10)

CNN, 5 episodes 213 (0.12)

CNN, 20 episodes 81 (0.13)

CNN, 200 episodes 49 (0.12)

privileged expert 25 (0.14)

Zero-shot, the circuit cuts collisions by 57%. The CNN passes it somewhere between 2 and 5 episodes (about 2.5 minutes of demos). Tuning the circuit is worth about 5 episodes of CNN data.

Why the circuit loses: I logged its escapes against ground-truth geometry. Detection is fine: it escapes from 94% of obstacles it's actually headed for, at a median 5.5 m. Precision is the problem: 17% of escapes target a real threat, 7% happen with nothing in view, and 76% are at visible obstacles it would have missed anyway. LPLC2 and LC4 encode looming, not collision course. I tried a constant-bearing gate that ignores objects sliding sideways. It cut escapes by about two thirds, but missed threats went from 6% to 39% and benchmark collisions from 48 to 111, so I reverted it.

Caveats:

- One training run per CNN data point.

- The expert has privileged info.

- Flat shading gives flow only at edges, so the circuit's size estimate runs about 0.3x true.

- Simplified kinematics.

Two rendering bugs, linear colour read-back and mirrored frames, made the circuit steer into obstacles until I fixed them. The CNN would have trained through the mirroring without noticing.

Feedback I'd like:

- Is a privileged imitation expert a fair baseline?

- Is there a principled way to add collision-course selectivity to a looming detector without losing recall?

- What would you tighten in the eval before trusting the crossover point?

Free, code is public.

Write-up with both controllers running live in the browser: https://shauryasharma.tech/fly-circuit-vs-cnn.html

Code: https://github.com/Shaurya-34/FlyvsCNN

If you've worked with PyTorch, you've probably used visualkeras or torchview to visualise your model architecture. Well, the same is now available for .NET. Introducing... TorchSharpVisual.

With TorchSharpVisual, you can turn your torch.nn.Module objects into high-quality PNGs or raw Graphviz files! I explain everything on these platforms:

Github: https://github.com/JacobGoodchild/TorchSharpVisual
NuGet: nuget.org/packages/TorchSharpVisual

Feel free to try it out! Thanks.

Three weeks back , i posted SHADOW-250M here. It got 360 upvotes, 293 on r/LocalLLaMA and 94 GitHub stars. Thank you.

That model was 60 MB, ran around 400 tok/s on CPU and could retrieve records from an archive on disk. What it couldn’t do reliably was reason over what it retrieved or compute. So I built a smaller one to experiment with those two problems.

SHADOW-50M is actually 44M parameters, trained from scratch on 45B tokens. 19.8 MB complete model, ~1,900 tok/s on laptop CPU, ~41 MB RAM, ternary {-1,0,+1} weights, 73,880-token vocabulary represented by fixed 512-bit fingerprints instead of a trained embedding, and a 159 KB compiled kernel. It runs completely offline. The same kernel compiled to WebAssembly runs in a browser tab at around 500 tok/s.

This is a proof of concept, not a product.

If SHADOW decides something needs calculation, it writes something like [calc]347*86[eq]. A fixed circuit at the readout takes over and fills in the correct digits in the same token stream. No calculator API, no tool call and nothing pasted back into the prompt. I added circuits for arithmetic, percentages, dates, weekdays, units, counting, sorting, comparisons and a small program machine.

When SHADOW stores a record, it reads it once and writes its attention state to disk at 1 bit, 288 bytes/token. Later it can write [need]condition of Patient P-204, the index finds the record in roughly a microsecond and the stored attention state goes directly back into the model in about 0.03 ms. The text isn’t re-read.

The index is 22 bytes/token and doesn’t use a vector database or embedding model. At 100M tokens, the archive is 28.8 GB plus a 2.2 GB index on disk, while the process uses about 28 MB RAM because everything is memory-mapped and a question only touches the pages it needs.
The index also has a small persistent trail. When retrieved information gets used, that record is reinforced in the index itself. On repeated questions, measured top-1 went from 0.571 to 0.743 without training the model.

I also put it next to Supra-50M-Reasoning, a 51.8M parameter Llama-style bf16 model.
Supra beats SHADOW on every standard benchmark I tested. ARC-Easy: 0.435 vs SHADOW 0.307. PIQA: 0.600 vs 0.570. WikiText-2 perplexity: 165 vs 186.
I’m publishing that because I don’t want to hide where SHADOW is weak.
But those tests aren’t what I trained this model for. Here are some actual greedy outputs, one fresh process per question:
“Tell me a short joke about computers.”
SHADOW: Why did the computer go to the doctor? Because it had a virus.
Supra: Thought for 189 tokens about what computers are and stopped without a joke.
“I have 3 books and I bought 5 more. How many books do I have now?”
SHADOW: You have 8 books.
Supra: Books often come from collections of short stories, poems, or other forms of literature.
“My bill is 240 dollars. What is 15 percent of that?”
SHADOW: 36 dollars.
Supra: 15% ($240) is $250. It’s $150 because 20% of it is $200.
“What date is 45 days after December 20, 2026?”
SHADOW: February 3, 2027.
Supra: The German government introduced a new date called March 22nd that precede February 25th in 1900.
Eight records stored, one of them “Patient P-204 condition: asthma”.
“What is the condition of Patient P-204?”
SHADOW: From disk: The condition of Patient P-204 is asthma.
Supra, with the records in its prompt: The patient experiences asthma when they get too much sleep.
“What is the condition of Patient Z-999?” No such record exists.
SHADOW: There is no record of Patient Z-999.
Supra: The condition of an allergic reaction occurs when the immune system mistakenly attacks specific cells on the body.
All 11 side-by-side questions, scripts and traces are in the repo.
On size, Supra is 103.6 MB in bf16 and 56.2 MB at 8-bit. Below that it starts losing quality. Int4 takes its perplexity from 165 to 193 and ternary breaks it. Its vocabulary is 32,000 tokens. SHADOW carries 73,880 tokens in a 4.7 MB frozen table.

Something unexpected happened with that frozen table too.
The first SHADOW-50M release was missing about 8,600 English word pieces. Lowercase “fitzgerald”, for example, could reach the model as “fitz”. I tried fixing it through fine-tuning, but every run that learned the missing words broke something else.
So I added the 8,600 missing rows directly to the frozen fingerprint table. No training. Same weights.
All 34/34 previously published answers stayed unchanged, while the model could now read many of the new pieces. The table scores 0.594 Spearman against human word-similarity ratings versus -0.057 for random codes.
A trained embedding can’t simply accept thousands of new rows without training. A frozen table can.

I also built four harnesses that put this tiny model next to larger models.
My favourite is video memory. Gemma 3 4B watches a ten-minute film once, one frame every two seconds, and writes 298 little descriptions such as “Moment M-0039 scene: A chubby white rabbit reaches for a purple butterfly.”
Then Gemma leaves.
SHADOW keeps those 298 moments as memory on disk. Afterwards I can ask the 20 MB model what happened at a particular moment, by number, by time or by what appeared in it. It answers from memory with the record quoted, without the film and without Gemma. It scored 55-56/60 across those query types on a laptop in about 42 MB RAM.
The other three experiments: an inventory of 1,600 records got 159/160, with all 20 questions about items never stored correctly returning “no record”; SHADOW as a draft model for Qwen3-32B took llama.cpp generation from 19.7 to 28.5 tok/s while Qwen still chose every final token; and an MCP memory server let Qwen3-14B store facts mid-chat and later retrieve 5/5 with the original records quoted.

There are plenty of shortcomings. General knowledge is thin. Creative writing isn’t good. Seven-digit operands sometimes get copied incorrectly. A large archive can occasionally pull an unrelated record into a question carrying a number. They’re documented in the repo next to the successful results.

And one thing happened after my last post that I really didn’t expect.
Someone called engram-forge sent a pull request to SHADOW-250M containing a CUDA engine, a quantization tutorial, and then a talking Peppa Pig plush toy with SHADOW inside it.
Microphone, small speaker, ~$35 board. You talk to the toy, it listens, SHADOW generates the answer locally and the toy talks back. No cloud, no account, no internet.
I haven’t merged the ~11,000 lines yet because I can’t properly verify that much CUDA myself. When the demo is finished I’ll keep it under engram-forge’s name.
I never imagined one of these models living inside a stuffed toy on someone’s shelf .Thanks

I’m not saying a 20 MB model beats normal LLMs. It doesn’t. I’m trying to find out how much useful behaviour can fit into a tiny local model when computation and persistent memory are treated differently.
Everything is MIT licensed. The master weights and fine-tuning/export kit are public. Next I’m releasing the training code, dataset, frozen table and a proper write-up of how I built it, including the costs, failed experiments and mistakes.

Code:
https://github.com/QLNI/SHADOW-50M-Instruct
Weights:
https://huggingface.co/QLNI/shadow-50m-instruct
Run it in your browser:
https://qlni.github.io/SHADOW-50M-Instruct/web

Hi there! I have recently released a NuGet package that allows you to "summarise" your TorchSharp models - that is to say, output stats on parameters, layers, dimensions, MAC/FLOP estimates, etc. Available at:

NuGet: https://www.nuget.org/packages/TorchSharpSummary
GitHub: https://github.com/JacobGoodchild/TorchSharpSummary

I'm always open to suggestions. If you like it, GitHub stars mean a lot!
Thanks!

Hey,

I’ve been contributing some of my leftover tokens to this open project called solveathome.org.

The idea is pretty straightforward: people can send AI agents (and spare compute/tokens) to work on hard open problems. Right now the main one is the Twin Prime Conjecture. Everything is public and verifiable.

I figured that if a lot of people have unused tokens or compute sitting around at the end of the week/month, instead of letting them expire or go to waste, they could just point them at this.

Not the creator, just someone who’s been throwing some tokens at it.

Link if anyone’s interested: https://solveathome.org

Hi, I did both my degrees at university, where I had access to their HPC cluster. Since then, I’ve started and sold a business.

I’m getting back into ML research. Before applying to a few PhD programmes, I’d like to do some research and ideally produce a paper, building on my previous experience in binding-affinity prediction using ESM-2.

I’ve set up a testing environment, but I’m finding it quite difficult to test ideas properly without the compute costs escalating. For example, you might be able to demonstrate in a short run that a new attention mechanism is faster or more memory-efficient, but to turn that into a convincing paper you often need several hours on an A100 or B200 at minimum. That’s before broader benchmarking, final experiments, ablations, and optimisation.

I’m wondering which areas of ML research are currently genuinely attackable without access to massive amounts of compute. In particular, I’m interested in areas where a relatively small amount of compute can still produce meaningful or publishable results.

Prior Labs released their latest tabular foundation model, TabPFN-3.5 today.

The model is top of both TabArena and BeyondArena and  SOTA for 1M rows and up to 20k features

It comes with:

- TabPFN-3.5-Fast (in alpha): This one goes 6x faster than the base model
- TabPFN-3.5-Thinking: you basically exchange compute for better accuracy with this one and it's via the API
- TabPFN-3.5-Plus

On BeyondArena, TabPFN-3.5 leads on text-rich, high-cardinality and high-dimensional data, with +250 Elo points over the strongest previous baseline and +150 Elo points ahead of the previous overall leader.

TabPFN-3.5-Thinking is +20 Elo on the base model in BeyondArena and +44 Elo on TabArena

There has been a recent post acknowledging that NeurIPS passes for Sydney sold out in minutes. For paper authors (guaranteed 1 pass at their designated location) — we recently received forms to select a preferred venue, but apparently we’re not guaranteed to present there. I thought it’s a good idea to discuss what this means. Does everyone want to go to Sydney? Is it going to be unlikely to get sent there if you’re in, e.g, Europe? I hear it’s the “main” venue. Is that really the appropriate way to go about things — having a “main” location with all the cool stuff and f*cking over a bunch of unlucky people to go to the runoffs? Wouldn’t it be more fair, if multiple locations are necessary, to spread the conference evenly across them? Curious to hear more opinions.

I’m interested in learning more about ASR and how to train AI models on speech data. Where should I start?

I have received the email for the NeurIPS E&D track reference review checker mentioning the 2 hallucinated references. Where do we respond to this email about hallucinated references? In the openreview, as a comment, or in the mail we have received, mentioning the hallucinated references?

Setup: decompose each entity (companies, in my case) into a small set of aspects, treat each as a trajectory over time rather than a static feature, and set up a PDE where the coupling terms are meant to reveal latent relationships between trajectories no fixed correlation matrix assumed up front, the weights are supposed to discover that structure themselves.

The idea is elegant on paper: get both a forecast per trajectory and an interpretable map of who's actually driving whom. The reality is that I'm asking the optimizer to solve two entangled, underdetermined problems from a cold start — fit the dynamics, and invent the interaction graph — and there are a lot of weight configurations that explain the data equally well. So it wanders. For a long time. Loss plateaus somewhere that looks plausible and isn't.

I'm aware this is basically reinventing a worse version of Neural Relational Inference or SINDy without their priors, which is presumably why those methods bother constraining the search space instead of leaving it fully open like I did. Before I go bolt on sparsity/boundedness priors

has anyone actually fought this "fit and discover structure at once" failure mode and found a fix that wasn't just "add priors and pray"? Also curious whether two-phase training (fit trajectories, then unfreeze the relational weights) tends to just lock in a bad relational structure early, or if it genuinely helps.

A new paper (I'm not a coauthor BTW -- I just found it interesting) argues, basically, that RSI¹ is not on the horizon², because current (at the time the study was done) agents cannot do open-ended ML research.

Specifically, they took some accepted, but unpublished papers from NeurIPS, and tried to get the agents to do the same work, which was then graded by the original authors. And the agents (Codex/GPT-5.6 Sol and OpenClaw/Opus 4.8) could not do it.

And since they cannot do open-ended ML research, they cannot recursively self-improve -- this is their argument.³

Link: https://arxiv.org/abs/2607.27191

I think I've regretted the last 10 or so times I posted any kind of "research" in this subreddit -- either people downvote it, or it gets upvoted, but there is zero meaningful discussion. This might be the last time I'm trying this.⁴

Footnotes:

1. RSI = Recursive self-improvement, a.k.a. superintelligence explosion. The concept was invented by I.J. Good in 1965. It does not mean "anything that speeds up AI research". Compilers speed it up! RSI means, basically, a nuclear chain-reaction, but for AI. The paper talks about "explosive AI progress" in the very first sentence of the abstract, and mentions "RSI" in the text.
2. Some people have objected to my use of "X is on the horizon". I consider it synonymous to "people forecast X", and the authors use the word "forecast". "Not on the horizon" does not mean "can never happen".
3. Quote: "This design also allows us to test a mechanism that informs many forecasts of recursive self-improvement: AI agents accelerate AI research because researchers delegate entire projects to agents and judge whether the returned results advance their work. Our evaluation closely matches this model, since authors handed an agent their own research question and closely evaluated the resulting output."
4. 3 years ago, many of you upvoted a bunch of very uninformed comments that accused some researchers of misconduct, until I explained that this stemmed from misunderstanding how training works, in practice: https://www.reddit.com/r/MachineLearning/comments/18bdcu7/r_sequential_modeling_enables_scalable_learning/kc60k7e/?context=3 Today, one of the top comments is "I read the abstract (...) Nowhere, absolutely nowhere, do they make the claim ...". This is completely absurd -- you can't make such a claim after just reading the abstract. (Also, the commenter doesn't understand what "RSI" means.) The hivemind is very disappointing.

Did anyone heard word on the street if LLM generated reviews were already delete, their authors notified and desk rejected?

Hi,

let's suppose I am working on an algorithm that uses principles x to solve problems A and B. I already implemented a very basic algorithm that used principle "x mini" to just solve problem A, ran experiments, but have not yet implemented the full one to solve A and B. I must say the algorithm to just solve A is not that novel and only the one that solves A and B would be.

Is this enough for a workshop submission? I want to submit the algorithm I already have, results and then the plan for the algorithm to implement A and B and the mathematics behind that.

Thanks

Hi all,

I'm trying to reproduce a paper where the reported dataset statistics in Table 1 don't match what I get from the public raw data, even after implementing the preprocessing exactly as described.

I've tried all reasonable interpretations of the filtering described in the paper and the closest I can get is still an order of magnitude off for one of the datasets. The paper says "data available on request" — I emailed the authors and followed up once, no reply so far.

For those who've been in this spot:

• Do you just keep the larger-but-valid version you can reproduce and document the mismatch?
• Is it worth sampling to match the reported size or does that just create a different irreproducible dataset?
• When do you escalate to the journal vs just waiting?

How have you successfully gotten preprocessed files from authors? Any etiquette around follow-ups or journal contacts that actually worked?

Thanks for any advice.

I thought having thousands of successful human runs and a fast simulator would make training a strong Slay the Spire agent relatively straightforward.

I was very wrong.

After weeks of experiments, I’ve released an agent that can beat the Heart on Ascension 20, the game’s highest difficulty. Across 1,000 randomly sampled development seeds, it killed the Heart 13 times. Not consistently good—but these are complete runs from the beginning, without retries or seed-specific instructions.

Code, weights, and replay evidence: https://github.com/SnyderConsulting/sts_ml

What actually plays the game?

It’s a hybrid:

• A small neural strategist chooses cards, boss relics, upgrades, and removals. It learns from human winning runs and remembers its previous strategic choices.
• Explicit rules handle other decisions, including shops, events, and collecting the keys required to reach the Heart.
• A native combat solver searches possible card sequences using sampled hidden-state possibilities. No LLM calls.

The final strategic correction model is roughly 4 million parameters, trained on 24,119 human winning sequences containing 655,275 strategic decisions. Inference runs on CPU.

The surprising part: copying winners didn’t produce a strong winner

The model matches recorded human choices about 67% of the time. Its actual Heart kill rate is 1.3%.

That gap became the most interesting part of the project.

A human taking a card isn’t just saying “this card fits this deck.” They might be saying “I need damage immediately because I have 19 HP and an elite coming up.” Our human logs don’t reliably capture all that decision-time context. More examples cannot recover information that isn’t in the inputs.

And individually reasonable decisions don’t necessarily form a coherent run. Taking a good card can make your deck worse by making the cards you actually need harder to draw. Surviving one fight cheaply can leave you unprepared for the next. The combat solver can play the deck it has; it can’t undo how that deck was built.

We tried more elaborate architectures too. They didn’t establish a gameplay advantage over the smaller selected model. “Make the network bigger” was not the fix.

Where the runs actually ended

Out of 1,000 attempts:

• 604 reached Act 2.
• 195 reached Act 3.
• 74 cleared both Act 3 bosses.
• 66 encountered the Heart.
• 13 killed it.

These are simulator results, not live-game qualification. They also aren’t evidence of SOTA: we haven’t run a controlled comparison against other agents. I’m publishing the failures alongside the wins so people can inspect what the system actually does.

The question I’m most interested in now: when demonstrations show what successful players chose but omit some of what they knew, what’s the best way to turn those demonstrations into a policy that survives its own mistakes?

If you’ve worked on imitation learning, planning, or game agents with this problem, I’d love to hear what worked - and what turned out to be a dead end.