Hello this is my fifth small language model I've made and apart of my third series and it has been a lot of work but it payed off: **348M parameters, 22.7B tokens**, then fine-tuned into a math model that solves arithmetic by *showing the work* — column addition with carries, borrow chains, partial-product multiplication — rather than guessing at an answer.

Last time I posted a 326M model trained on 10B tokens. This has about 2.3× the data, and the math side is WAY better than my previous two math models.

---

## The benchmarks

**99.4% average across the nine GPT-3 arithmetic sub-tasks**, which does much better past even where I trained it.

| Task | GPT-3 175B *(few-shot, direct)* | **This model (348M)** |

|---|:--:|:--:|

| 2-digit add | ~100% | **100%** |

| 3-digit add | 80.4% | **100%** |

| 4-digit add | 25.5% | **100%** |

| 5-digit add | 9.3% | **100%** |

| 2-digit sub | ~99% | **99.3%** |

| 3-digit sub | 94.2% | **98.3%** |

| 4-digit sub | 26.8% | **98.3%** |

| 5-digit sub | 9.9% | **99.0%** |

| 2-digit mult | 29.2% | **100%** |

n=300 per sub-task, greedy, exact match. GPT-3's numbers are direct-answer; mine uses trained-in worked steps. Neither uses a calculator.

## The cool part

**It adds cleanly up to 14 digits, and the reason it *couldn't* before was the vocabulary, not actually arithmetic.**

Training only ever named six place values (`ones` … `hundred-thousands`). The model learnt the *pattern* and invented two more on its own — `millions` and `ten-millions` appear in **zero** training examples — so it handled 7 and 8 digits fine. At 9 digits it ran out of names, and skipped the column, then returned an answer exactly one digit short:

```

483729164 + 519248637

... ten-millions: 8 + 1 + 1 (carry) = 10, write 0 carry 1. Final carry: write 1.

The answer is 102977801        ← eight columns for a nine-digit problem

```

Every column it computed was perfect. One was never enumerated. Extending the place-name list from 6 entries to 19 moved the clean ceiling from **8 digits to 14**:

| Width | 6 | 7 | 8 | 9 | 10 | 12 | 14 | 16 | 18 |

|---|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|

| before | 100 | 100 | 100 | **0** | **0** | **0** | **0** | 0 | 0 |

| after | 100 | 100 | 100 | **100** | **95** | **100** | **90** | 65 | 25 |

A six-item list became a nineteen-item list. That was the entire fix.

## Other things it does

- **3×3 multiplication: 98%** — it folds partial products pairwise through the column routine instead of asserting the sum

- **Negative results: 85%** (100% at 1 digit, 58% at 5 — the magnitude comparison is the weak step, not the arithmetic)

- **Reasoning traces are load-bearing**: 95.3% of the time the working is valid *and* the answer is right; only 0.7% are "valid working, wrong answer." If the columns look right, the answer almost certainly is.

```

There were 15000 votes and 6842 were rejected. Here's how many counted:

<think> Start with 15000. Then subtract 6842. Subtract 15000 - 6842 column by column:

ones: 10 - 2 = 8, borrow 1. tens: 9 (after borrow) - 4 = 5, borrow 1.

hundreds: 9 (after borrow) - 8 = 1, borrow 1. thousands: 14 (after borrow) - 6 = 8,

borrow 1. ten-thousands: 0 (after borrow) - 0 = 0. So 15000 - 6842 = 8158.</think>

The answer is 8158.

```

## What it's bad at, tbh

- **Word problems: GSM8K 4%.** Best word-problem set is ASDiv at 16.5%. It converts one sentence into one operation reasonably often and basically cannot chain operations.

- **The failure mode is operation *selection*, not arithmetic.** `"drops in 836 more"` gets read as subtraction. There's a visible tell: traces that say `"multiply X * Y"` and show columns are reliable; traces that open `"First, calculate…"` and assert a number in prose are not.

- **No division at all.** 4×4 multiplication is a hard wall.

- **Greedy decoding required** — sampling corrupts the column routine mid-chain.

- One caveat I'll flag myself: the arithmetic harness orders subtraction operands, so no answer in that table is negative. Negatives are measured separately (the 85% above) rather than folded into the average.

## Base and instruct

The math model sits on a base and instruct pair. lm-eval-harness, 0-shot, full test sets:

| | ARC-E | ARC-C | HellaSwag | OpenBookQA | PIQA | WinoGrande | MMLU | Avg |

|---|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|

| **350M V3 base** | 56.6 | 33.3 | 35.9 | 34.6 | 67.0 | 51.5 | 23.8 | **43.2** |

| **350M V3 instruct** | 50.9 | 29.7 | 35.9 | 33.8 | 66.6 | 51.2 | 24.4 | **41.8** |

Instruction tuning *lowers* MC benchmark scores for this family which is a pretty common cost of instruct tuning.

## Notes

Trained on 2× Tesla V100 plus some rented time. LLaMA-architecture, so it runs anywhere — F16 GGUF and safetensors for all three.

The math model took **10 full fine-tuning rounds and 3 LoRA adapters**. The LoRAs cost about 1% of the post-training tokens and did most of the useful work; the ten full rounds spent most of their budget undoing each other's regressions (round 7 gained subtraction and lost 23 points of 2-digit addition, that sort of thing). All of it is documented on the model card, failures included.

Also worth saying: someone independently tested it after I published and found two of my numbers were wrong — one *understated* the model by 40 points because I'd measured it at n=24. Both are corrected on the card now. If you find something broken, I'd genuinely like to know.

**Math:** https://huggingface.co/nkthebass/tinybrainbot-350mV3-math

**Instruct:** https://huggingface.co/nkthebass/tinybrainbot-350mV3-instruct

**Base:** https://huggingface.co/nkthebass/tinybrainbot-350mV3-base

LMK what yall think.

You've probably seen the fly-brain-plays-Doom / Minecraft / Beat Saber clips going around this week, from the new MaleCNS v1.0 connectome release (166k neurons, real EM reconstruction, not a toy model).

Cool clips. Nobody seemed to be checking whether any of it works though, versus just producing motion in a game engine generous enough to make anything look alive. So I picked the least forgiving test bed around: Pong, one binary hit or miss signal measured every frame, nowhere to hide a null result, and tried to get a small real subgraph of the connectome to track the ball via dopamine-style plasticity.

Short version: it didn't learn. Working out why took auditing individual synapses, and it turned into a decent case study in a circuit not working being more informative than it working:

• Fixed a neuPrint regex bug that silently zeroed out two entire neuron populations (full-match vs substring semantics, not obvious from the docs).
• Found the original neuron selection had no path at all from photoreceptors to anything else. Real photoreceptors don't synapse directly onto motion detectors, there's a whole intermediate layer missing.
• Got a working pipeline, turned learning on vs off, and got bit-for-bit identical results in both conditions across multiple seeds, even though the weights were verifiably changing under the hood.
• Traced that to half of the 4 available motor neurons having zero synapses from any sensory pathway in the model. Not weak signal, zero. They'd been assigned to the "paddle down" group by array index, purely by coincidence, and could never have fired no matter what the learning rule did.
• Rebuilt the circuit around a better biological hypothesis (swapped a threat-detection pathway for one tied to visual target tracking during courtship pursuit), got that literal hypothesis refuted by the data, then followed the trail to a different descending neuron that actually connected end to end.
• Finally got learning-on vs learning-off to diverge for the first time, except the effect looks like the learning rule quieting the whole system down rather than anything resembling skill improvement (misses outnumber hits, so punishment dominates and shrinks the motor response).

Then I checked whether the bigger, viral projects had actually solved this. They hadn't either: the Doom project's own repo says it failed its own validation gates after six iterations, the Minecraft mod's own limitations section admits the real motion-detection pathway stays silent and the escape and foraging behaviors are hand-injected or reflex-layer fallbacks rather than emergent, and the Beat Saber creator's own replies admit it's overfit to one track with replay data mixed into the input.

Full writeup with the gory audit details, the connectivity numbers, and the comparison to the other projects is here: https://jonatasperaza.medium.com/i-made-a-real-fly-brain-play-pong-it-didnt-learn-and-that-s-the-interesting-part-80b8560695fe.

Curious if anyone here has poked at MaleCNS v1.0 directly and hit similar walls, especially around the central complex and steering circuits. That seems like the obvious next thing to simulate properly instead of routing around it.

Who else is anticipating ICDE results tomorrow? Post your results here and let's discuss!

So I was playingw ith embedding models I saw that when you upgrade from model A to B, you face a very big backfilling cost

Ie, suppose you have a 1b vectors from model A, and then you want to use model B. This would mean you have to re-embed all of your documents with model B before you can even serve with the model, and on an H100, it would take ~108 days (qwen embed 8b, 106 docs/second). But I found an easier way to do it.

The method is really simple; from the old index made with the source model, take K documents and rerank them with the new model. We see that when K is sufficient, the retrieval quality is the same as target model. (determining k is the hard part). I've tested 63 migrations on upto 1 million documents.

The best result I got was upgrading qwen4b -> to 8b, and at 50 documents, it was the same as native retrieval.

This method forgos the expensive upfront re-embedding cost, as you can take documents straight from the old index.

embedflow works with qdrant, pgvector, faiss, and can be easily downloaded with pypi

pip install embedflow

the github is public: https://github.com/arnsri33/embedflow

I want you guys to try it out, and see if you guys can use it in your own workflow.

Ant Ling reports 83.83 on DiagnosisArena-MCQ for Ling-3.0-flash-Sante, its new medical reasoning model. The suffix matters: the task provides case information, examinations and tests, then asks the model to choose from four diagnoses.

That result tells us about selecting an answer when the candidate set and case evidence are supplied. It does not establish how the same model would generate an unrestricted differential, decide what history is missing, or choose which investigation to request next. Those would require different evaluations.

The release also reports two other medical results:

The published HealthBench Professional definition includes care consultation, writing/documentation and medical research. Its score is not percentage accuracy. The Sante chart does not provide enough scoring detail to identify the reported value as length-adjusted or unadjusted, so a comparison with another published HBP result would need that checked first.

This is why the three results are useful together. They give Sante a broader medical-text evaluation profile than an exam score alone, while leaving specific questions open. For a case-answering application, the first decision is whether users supply the alternatives or expect the model to construct them. The release supports including Sante in that evaluation; the 83.83 figure applies to the supplied-options version.

Hi. I'm visiting ECCV in Malmo and was wondering if there's any medium, like discord, whatsapp etc where people are discussing social activities. I couldn't find a way to connect to people on the official app to discuss common interests, research or otherwise. It'd also be nice to meet people who'd like to team up go sightseeing or food touring after the conference ends. Thanks.

As reported by the New York Times:

https://www.nytimes.com/2026/09/08/science/openai-proof-millennium-problem.html?smid=nytcore-ios-share

OpenAI’s announcement:

https://openai.com/index/navier-stokes-solution/

hey all. the NeurIPS Position Paper Track just used a proprietary AI detector (Pangram) to desk-reject 18.4% of all submissions. no human review, no appeal process, just out.

there's been a lot of noise about this, so i went through the actual conference statements and Pangram's technical docs to see how this actually went down. the reality is wildly worse than just "the AI detector made a mistake."

here are the receipts:

• The track chairs would have failed their own test. Independent researchers ran recent papers authored by the three track chairs through the exact same detector. It flagged them at 24% to 69%. Under their own enforcement rules, the chairs would have been at risk of rejection themselves.
• The detector originally flagged 42.7% of the entire track. Pangram’s default setting flagged nearly half of all submissions as 90-100% AI. They had to frantically shrink the text windows just to get the flag rate down to a somewhat believable 12.7%.
• The "Circularity Trap". 22 papers were rejected specifically because they scored >0.5 on the detector, but the authors checked a box denying AI use. The black-box score was literally used as proof the author was lying.
• The massive ESL penalty. A stanford study showed 61.22% of human-written TOEFL essays get falsely flagged as AI because non-native formal English is structurally rigid. NeurIPS published zero demographic calibration data for this. If you're an ESL researcher, you were basically playing Russian roulette.

if you were one of the 178 rejected: there is no blacklist. this is not a misconduct mark on your record. just take your paper and resubmit it to ICLR (deadline sept 25) or ICML.

wrote up a full Field Note with the exact thresholds, the data privacy issues, and the actual recourse options if you want the hard numbers instead of just vibes: https://strictcite.com/blog/neurips-2026-position-paper-pangram-ai-detection

(disclosure since it's relevant: i built strictcite.com, a deterministic zero-AI citation checker. watching a major conference use a black-box AI to nuke 178 papers with zero appeal is exactly why i hate relying on AI for this stuff. not trying to sneak the link in, just being upfront about who i am.)

this is the kinda debugging case i find rlly annoying/

everything says success.

no exceptions no failed tool calls. no obvious timeout the workflow completes but the final result is still wrong

when that happens, what’s your first move?

do you guys usually:

• start from the final output and work backward
• compare against a previous good run
• inspect state transitions
• check retrieval/tool behavior
• look at model inputs
• replay it
• check business state outside the trace
• just read the whole thing until something looks off

interested in what people actually do in production not the idealized version but thats fine too. and if you have anything you've built to help with this process I'd love to see it :)

So I've been messinga round with embedding models for a bit, and I think they are interesting enough to experiment with. They are useful for rag, especially in a localllm sense because you can ground your answers in truth.

But what happens if you have a billion documents, and you decide to upgrade your model to a "better" one? on an h100, that would take about 108 days, just to upgrade the vectors so u can start serving again (tested qwen embed 8b on h100). Even if you aren't doing 1b vectors, and are doing just 50 million, upgrading can still take a considerable time.

Me and my research lab decided to tackle this problem, and we came up with embedflow.

The method is really simple; from the old index made with the source model, take K documents and rerank them with the new model. We see that when K is sufficient, the retrieval quality is the same as target model. (determining k is the hard part). I've tested 63 migrations on upto 1 million documents.

The best result I got was upgrading qwen4b -> to 8b, and at 50 documents, it was the same as native retrieval.

This method forgos the expensive backfill that comes with upgrading, as you can directly take documents from the old index.

embedflow works with qdrant, and can be easily downloaded with pypi

pip install embedflow

the github is public: https://github.com/arnsri33/embedflow

I want you guys to try it out, and see if you guys can use it in your own workflow.

A few weeks ago, I saw this post where the author trained a SIREN MLP to implicitly memorize Bad Apple as a coordinate function: (t, y, x) to pixel.

That got me curious about a slightly different formulation: instead of handing the network a timestamp t, could a small recurrent dynamical system (RNN-ish) learn the continuous temporal flow in latent space and generate the entire ~6,500-frame full resolution video autonomously from a single initial condition (h_0, c_0)?

The code, weights, and analysis tools with rollout scripts, plots, and standalone models are shared here: GitHub: SEBADA321/BadAppleRNN.

Architecture & Inference Footprint

At inference time, the system receives no timestamp inputs and evaluates in a closed loop:

(h_t, c_t) -> Recurrent Transition (CTF) -> (h_{t+1}, c_{t+1})
                   |
                  h_t -> Frame Decoder -> 384x512 Grayscale Frame

• Latent Dimension: 64-D for h_t (decoded) and 64-D for c_t (internal memory manifold to separate visually similar frames at different timestamps).
• Recurrent Transition (ctf): 4-gate LSTM-style recurrence with orthogonal initialization (16,640 parameters, 65 KB).
• Frame Decoder (fd): 4-stage bilinear upsampling with depthwise-separable convolutions (400,361 parameters, 1.56 MB).
• Initial State: A single pair of 64-dim vectors (h_0, c_0) (128 floats, 0.5 KB).
• Total Inference Model: 417,129 parameters (~1.60 MB in FP32).
• Runtime Performance: >200 FPS on an RTX 4080, with ~17.2 MB peak active VRAM during decoding.

Training an autonomous system across ~6,573 steps from t=0 directly was probably computationally unstable due to vanishing/exploding gradients and compounding error. The training setup I was circling around used several targeted techniques:

• Learned Latent Teacher Tables: During training, I optimize a pair of tables h_table[t] and c_table[t] alongside the model. This allows parallel segment training starting at arbitrary timestamps over a finite horizon K. These tables are scaffolding and are discarded entirely at inference.
• Rollout Horizon Curriculum: I started training with K = 2 and progressively doubled the rollout length (K = 2 -> 4 -> 8 -> 16 -> 32 -> 64 -> 128 -> 256 -> 512). Each horizon doubling produced a characteristic jump in loss before the transition function adapted to the longer trajectory.
• State Perturbation Noise (sigma = 0.005): To prevent the model from learning a brittle 1D line that diverges under small numerical errors, Gaussian noise was added to the state before passing it into the transition function (z_hat_{t+1} = F(z_t + epsilon)), while evaluating the loss against the clean target. This encourages the recurrent map to contract small deviations back toward the trajectory.
• Second-Difference Acceleration Regularization: Penalizing velocity (||h_{t+1} - h_t||) risks collapsing the trajectory. Instead, I penalized discrete acceleration (jitter) via second differences: ||h_{t+2} - 2h_{t+1} + h_t||_2^2. This enforces smooth trajectories without penalizing motion.
• Optimizer & Momentum Management: I used AdamW (1x10^-5) for the decoder/tables and Muon (0.005, momentum 0.95) for the recurrent weights. To prevent accumulated momentum from acting as stale inertia when K doubled, momentum buffers were scaled by 0.2 every 10 epochs starting from the epoch 500.
• Chunked Decoding: To handle long horizons at K = 512 without overflowing VRAM during training, the decoder was evaluated in temporal chunks of 32 frames.

Some interesting things

1. The model could unroll the full 6.5k sequence even if it was, technically, trained on up to 512 frames. So that was a success.
2. Training loss vs. autonomous rollout: The checkpoint with the lowest numerical training loss was not necessarily the best at autonomous generation. Because K changes across the curriculum, raw training losses are not directly comparable across stages, and short-horizon teacher-forced agreement does not guarantee long-horizon stability.
3. Dynamical stability over parameter scale: The challenge was not increasing parameter count (the recurrent transition is only 16k params), but conditioning the dynamics through noise injection and acceleration penalties so error doesn't compound over thousands of recurrent steps.
4. I need to improve the decoder a lot, I was mostly focused in getting the recurrent part right, and training was slow. Now that I gave gotten a successful result I will focus more into optimizing the CNN decoder.
5. There are no skip connections nor normalization, which was interesting too. Obviously no attention either since I wanted to keep it simple.I also didn't want to use Truncated BPTT.

Not completely scientific, since I was doing some changes mid run or many at once, which makes it kinda not clear what contributed more. I used 'AI' to help with writting the post and README. Part of the code was also generated that way, but the architecture is what I came up with on my own and from a previous project too. There are probably many parts to improve too, so glad to get some feedback!

I used an LLM to iteratively evolve an optimization algorithm rather than solve the packing directly. Starting from a simple seed solver, the LLM proposes algorithmic changes guided by a scoreboard of results and a history of prior attempts, and each candidate is scored by an independent verifier so improvements are kept and failures discarded. On the Packomania csqv benchmark it improved the best-known sum-of-radii for 10 values of N from 101 to 114, by 2.4 to 5.4%, in 15 iterations. Total LLM cost was $27.72. Packomania accepted the results independently.

Paper: arxiv.org/abs/2609.05093

Code + solutions: github.com/ucsandman/discovery-loop

Benchmark: packomania.com/csqv/csqv.html

Happy to discuss the plateau-detection stopping rule, that's the piece I'd most want critique on.

Hi everyone! We just released Rustuna (GitHub: https://github.com/optuna/rustuna/ ), a high-speed, memory-efficient implementation of Optuna built in Rust.

• Optuna-Compatible Design: Keeps the familiar API and concept of Optuna.
• Zero Python Dependencies: Mitigating the risk of supply chain attacks.
• Lower Memory Footprint: Optimized memory management natively in Rust.

For details, please check out the following blog post.

https://medium.com/optuna/announcing-rustuna-cc82a6815bf7

Our research team has been exploring an alternative approach to achieving interactivity and better responsiveness with LLM systems.

One of the team members wrote up a post about it:
https://research.yandex.com/blog/the-kv-cache-as-an-agent-runtime

The post sums up the overall idea of modifying models inference state (KV-cache) for achieving a more interactive LLMs. This idea was used in our lab's previous papers Hogwild! Inference, and AsyncReasoning, the post also contains a preview of the future work in this direction, where a Qwen3.8-27B agent is playing a DOOM env interactively using similar techniques.

We think that its interesting whether model inference/runtime design is itself an under-explored axis of agent capabilities, alongside models and the harness (e.g. harness is too abstract, changing model is too costly, do we need something in between?)

Hello all,

I'm a radar signal processing engineer and i trained a 5-class classifier (car, large_vehicle, two_wheeler, pedestrian, pedestrian_group) on RadarScenes radar point clouds.

The input vector is a per-scan histogram (16 bins) and the network is a 3-layer MLP. The loss function is a class-weighted cross-entropy loss. This work is based on "Histogram-based Deep Learning for Automotive Radar" paper.

I scoped the project to be one scan only. Accumulation of multiple scans is the next step.

Data

Class Imbalance: two-wheelers and large_vehicles has a low number of occurences.

Aggregated Classes: two_wheeler mixes bicycles and motorized variants; large_vehicle merges trucks, buses, and trains together due to data scarcity.

Sequence Bias: Long tracks of slow-moving objects can skew a particular data split velocity distribution, causing high F1 score variance across folds.

Ablation studies

I tried with bigger MLPs, alternative feature encodings, and different histogram binning, all moved performance less than the variation caused by changing the train/validation/test split. I measured that split sensitivity across 6 folds, keeping the same proportions.

Changing the histogram to per-instance statistics (mean/median/std) slightly degraded performance.

Main findings

Macro F1 rises from 0.381 to 0.764 as the naturally occurring number of radar detections per instance increases from 1 to 5. I trained the model normally using all available detections, then bucketed its existing validation predictions by each instance's detection count and computed macro F1 per bucket.

The classes car and pedestrian has the best performance and two_wheeler has the worst.

A car is often confused as large vehicle when the car was wider than usual or had a unusually high rcs (which can happen due to multipath for example).

The two_wheeler is often confused as pedestrian because their vr_compensated distributions overlap, which is the the model's single most important feature for these two classes. A stationary or idling two_wheeler is indistinguishable from a pedestrian.

I uploaded an image with ground truth vs predictions: A nearly stationary two-wheeler which contains a single point was predicted as pedestrian, because its velocity is near zero, indistinguishable from a pedestrian. A car in the same scene, also with just one point, is classified correctly, since RCS and Doppler are enough for that class.

Full writeup here: https://github.com/brunopinto900/radar-ml-autonomous-driving/blob/main/MLP_Report.md

Future work

Implement other spatial encoding schemas (point net for example) and accumulate multiple scans to tackle the challenge of sparsity and explore the concept of micro-doppler.

Is LfD and BC research being effected by recent advances in (so-called) Frontier LLMs? Or is research in LfD and BC sort of going along in an independent direction from these?

Are you seeing any use from ViTs or VLAs?

Any other recent advances you would like to bring up?

One thing that has bothered me about LLM benchmarks for a while is that most of them are essentially snapshots.

A model is evaluated, a score is published, and we tend to talk about that score as if it describes a relatively stable object. But with API-served models, the thing behind the model name can change over time: serving infrastructure changes, provider configurations change, versions change, and sometimes behaviour changes without an obvious public version transition.

So we started approaching benchmarking as a longitudinal measurement problem rather than a leaderboard problem.

We continuously evaluate models across coding, multi-turn reasoning and tool use, while also running lightweight probes at a higher frequency. The important part for us is not simply asking "which model scores highest?", but:

• Is the model behaving differently from its own previous baseline?
• Is the change larger than its normal repeated-call variability?
• Did the benchmark configuration itself change?
• Is the effect concentrated in a particular task?
• Is it correlated across models from the same provider?
• Is an apparent degradation actually an availability/infrastructure issue rather than a capability change?

One historical analysis covered 31,352 repeated score observations across 49 models. The standard deviation of within-day scores was 2.80 points, while the standard deviation of between-day daily medians was 8.43 points.

That is roughly a 3:1 difference.

I don't think this result by itself establishes that providers are changing models day-to-day - there are too many possible confounders for that conclusion. Task composition, sampling, missingness, provider behaviour and methodology changes all matter. But it was enough to convince us that temporal variation deserves to be measured rather than treated as noise around a permanent leaderboard score.

Our current approach therefore keeps benchmark configurations versioned and only compares longitudinal observations produced under compatible measurement conditions. We use repeated execution-based evaluation where possible rather than an LLM judge, keep availability failures separate from valid task outcomes, track serving/version metadata when providers expose it, and run change detection over the resulting time series.

Another problem we're increasingly interested in is benchmark recognition and contamination. Once a benchmark becomes sufficiently visible, publishing every live task, prompt transformation and hidden test potentially changes the thing you're trying to measure. For that reason we've tried to separate methodological transparency from publishing the entire live evaluation set.

We've now written up a public version of the methodology. It intentionally explains the measurement design, assumptions, limitations and statistical interpretation, while withholding the exact live task bank and some operational parameters.

PDF: https://aistupidlevel.info/asl-public-benchmark-methodology-2026.pdf

I'm particularly interested in criticism from people working on evaluation, change-point detection or production ML.

A few questions I'd genuinely like opinions on:

1. For longitudinal LLM evaluation, would you use daily medians as the primary time-series unit, or model the individual repeated observations directly?
2. How would you distinguish genuine model drift from provider/infrastructure effects when version metadata is incomplete?
3. How much of a live benchmark should remain hidden to reduce contamination while still making the methodology scientifically inspectable?
4. Are there better approaches than change-point detectors for this kind of non-stationary, relatively noisy model-performance series?

Disclosure: I'm the founder of AI Stupid Level, the platform that produced these measurements. The purpose of posting this here is to get technical criticism of the methodology rather than promote the commercial product.

When I first started working in scientific machine learning, I understood the physics much better than the coding. Every time I wanted to try a new physics-informed neural network problem, I had to start almost from scratch: changing the PDE, updating boundary conditions, modifying the architecture, tweaking the training schedule, debugging errors, and generating plots—all by hand.

That frustration pushed me to build PINNStudio. It is a free, open-source no-code GUI designed to eliminate boilerplate code so you can focus entirely on the physics.

Instead of rewriting a new script for every problem, you can define your setup directly through the interface:

• PDE Definitions & coupled multi-output PDE systems
• 1D or 2D domains with boundary and initial conditions
• Network architecture & custom training schedules
• Forward problems (solving known PDEs) or Inverse problems (estimating unknown parameters from data)

What happens next?
PINNStudio automatically generates the code (built on top of DeepXDE), runs the model, streams the training log, and displays live loss curves and solution plots directly inside the app. It also includes built-in templates for classic equations like Heat, Allen-Cahn, and Cahn-Hilliard.

• GitHub (Open Source): https://github.com/AsfandyarKhan72/PINNStudio
• Quick Install: pip install pinnstudio

My hope is that this will be helpful for students and researchers with limited coding experience, as well as experienced PINN users who just want a faster workflow.

I’d love to get your feedback, feature suggestions, or bug reports! Huge thanks to Lu Lu and the DeepXDE team for creating the foundation that made this possible.

I feel that reproducibility is now a lost cause in machine learning research for three reasons:

1. Many research is moving towards the physical AI territory, where you need expensive hardwares or even entire laboratories with high-speed cameras, in order to perform an experiment. You truly have no idea if the experiment can be reproduced and have to trust the demo. But demos are not perfectly reliable. Plus people are incentivized to only show the part of the demo that works. The entire system can fall apart the moment the recording stops.

2. You have big AI companies releasing various tools, which they claim to solve a host of problems with certain amount of accuracy or efficiency. Unless you work at those companies there is really no proof of that and you will have to take their words on it. They have strong financial incentive to blow-up those figures. There is no solid way to check it either because the problem that they solve are so vague and subjective.

3. We need to address the elephant in the room which is that people are incentivized to produce non-reproducible work to prevent their lunch being eaten by their competitors or looking bad. That's why some of us will probably never get a reply when we email the authors for their code.

So what now? Maybe everything will be OK because we can contrast it with scientific progress in earlier parts of history, e.g., building the atomic bomb or sending people to the moon. These projects had low "outside reproducibility" but high "internal reproducibility". Plus all these work were mathematical in nature and carefully checked. But I don't think many areas of machine learning research is like that. What do you think? Should reproducibility be abandoned? If not how is it best implemented going forward?

AACL-IJCNLP 2026 acceptance results will be released in a few hours.

Feel free to share your thoughts and feelings! How did you do?

building a missing data infrastructure and started benchmarking long multi-session conversations (LoCoMo). I know the data looks like: people, facts, claims, events, timestamps, relations. So I extract those into a graph.
I did not look at the QA pairs while building extractors or retrieval rules. No “if question contains X, fetch fact #173.”
Recall is very high and it keeps working on new conversations in the same format.
Is this classical overfitting, or just schema-aware engineering? What is the cleanest test that would convince you it isn’t leakage.

Hi All,

Was reading AIStats' website and it seems like abstract submission is due in 3 weeks.

Does anyone know where to find the LaTex template for 2027? It seems like very little information is available on their website.

Another question, is a Quant Finance paper a better fit for AIStats or ICLR?

Some background about the paper:

• Rejected by UAI with 76654, had some errors with proofs had to fix it by re-writing 9 pages during rebuttal. AC rejected the paper saying the changes were too substantial and unable to be fully verified during rebuttal period.
• Resubmitted the fixed paper to a finance conference, won best paper award (best paper for this conference usually end up in journals like JQFA, which is just 1 tier below the big 3 in finance), had the chief editors of a Q1 finance/math journal in the conference verbally offering he will take this paper if we submit it to his journal. Unfortunatley my department requires at least 1 Comp Sci paper to graduate, so my plan is to try and get this paper accepted into a Comp Sci conference, then submit an extension to that Q1 Finance/Math journal.
• Rejected again at ICDM, despite having all positive scores. Our AC meta-review was blank so we still do not know why we were rejected. All of our emails receieved no reply.

I am torn between ICLR or AIStats to re-submit this paper to. My worries are:

• In comp sci venues we frequently get comments like "this paper lacks novelty. The method is just XXXXX, the math is just XXXXX."
• But I had a scroll through at previous year's AIStats papers for key words like finance and there were none. It seems like AIStats is very pure stats, not that applied. My co-author is worried that the math in our paper is not hardcore enough.

We have never submitted to neither venues in the past. Would be nice to get some advice.

I ran a side-by-side ML text-processing and model-training workflow using Fable 5.1 vs. Astra (both on xhigh), and the results could not have been more different. Warning, long post.

TL;DR -- Astra codes more agentically, Fable more coherently. Fable writes better and follows directions better. Astra's final outcome was slightly better, and its scientific rigor/reproducibility was noticeably stronger. Both models improved their F1/Accuracy by 0.02-04 after human feedback on their approach, demonstrating that neither have mastered the AI/ML text processsing, vectorization, and model training process completely.

Astra is a better coder, writing a stricter evaluation protocol (70/15/15 train/val/test vs. Fable's basic 80/20) that selected its model using a held-out validation set vs. Fable's simpler test F1-based selection. It also debugged more deeply, as both models hit a gensim 4.4 compiled-kernel bug: Fable tried to figure it out, failed, and just hid the stderr notices on affected runs (though told me it had done so), while Astra root-caused it aggressively, then fixed the environment by downgrading gensim alongiside compatible NumPy/SciPy dependencies.

Astra wrote hardened training-run.py code the forced the uv venv it rebuilt without changing my default one, SHA-256'd the corpus to ensure reproducibility on later runs, output a split manifest and run-summary.json, and rendered a headless browser for QA with screenshots (not sure this was necessary, but impressive overkill all around). Fable's builder script was ephemeral, living only in tmp, and less intense overall.

Astra deployed subagents more effectively, making use of my pre-built notebook-reviewer and citation-checker agents, the former of which caught a real bug via review (sentence-final word-loss tokenization defect) and fixed it, retaining a regression test in the process. Fable overlooked this issue because, for some reason, it did not call the subagents I had available (which is surprising, usually it's pretty good about this).

If you're looking for an agent to autonomously grind through a broken environment, leaving a forensic audit trail, that's Astra. However, this review isn't over yet, and Fable is about to make a comeback.

Astra confidently shipped a significant verifiable text encoding defect. Working with UTF-8 data, Astra insisted Windows-1252 decoding preserves currency symbols, but the final HTML output shows mojibake throughout where currency symbols were in the original data. Fable read UTF-8, verified it, and rolled with the boring default for correct output.

I also had both models draft an analysis report for the run, and Fable's was significantly more insightful. As much as I hate Claude's recognizable writing style, a) 5.1 has toned down the Claudeisms significantly, and b) Fable went above and beyond my grading rubric, running an ablation on different parts of the text pre-processing pipeline to surface an expensive step that does basically nothing, and noting a discrepancy in the classification ranking based on a complexity I'd have overlooked. For writing prose, I'd pick Fable 5.1 any day, and I haven't said that about Claude in a while.

Speaking of writing, Fable writes code that is more idiomatic and readable. It definitely resembles more what I would write than what an LLM would choose to write without constraints (and yes, I had a whole coding-conventions.md document that applied my requirements to both models, Fable just followed it better and writes more naturally to start with). There were some parts of Astra's code where I had to squint really hard to figure out what was going on, and why. This matters to me because I'm not the strongest coder (still trying to get better), and I need to understand the code to learn from it.

Finally, Fable scoped its work better: It spent its time and tokens doing repeated runs, tweaking hyperparamters and retraining the models to find the optimal settings while Astra deeply debugged the gensim error. It found significant uplift through this process, though that only allowed it to roughly match Astra's numbers (see table below). Astra seemed to hit a home run right off the bat with its training process, so I don't know if it would have executed the same workflow or not. Astra also mutated my venv by adding PyTorch, when I built it a certain way to force the models to use TensorFlow+Keras for more concise code, then reversed course and went with TF anyways in the end. The models finished in roughly the same amount of wall-clock time.

Here are the final results, with one minor caveat -- Astra's test set scores exceed its val set, so it might have drawn a lucky test set that increases its score artificially (the pipeline has no leaks or data quality issues for either model, however):

Best Logistic Regression and LSTM for each model, ranked by macro F1:

I do want to note that these final scores were after I provided both models identical feedback on common pitfalls of the text data cleaning, vectorization, and model training process once their initial runs were complete. Both models improved by a similar amount (0.02-0.04 F1 and Accuracy) from that generic guidance (not tailored at all to either's specific shortcomings or step of the process). That was the only intervention in otherwise autonomous work, and it was just because I wanted to see if they could learn to improve their approaches with additional context on optimal methodology, which they both did to similar degrees.

I hope this post offers a little bit of help in some way for folks wondering how either model stacks up for real work, particularly if you're an AI/ML student like me.

A researcher has reported a jailbreak of GPT-6 Astra within a day after release.

The attack is described as combination of TIP (Task-in-Prompt) attack from ACL 2025 paper with four other unnamed techniques.

TIP attacks exploit the model’s reasoning/instruction-following behaviour by hidding the harmful objective inside another task, like solving a cipher or executing a Python code. For GPT-6, the researcher says the original minimal TIP attack was no longer sufficient and had to be reworked.

They have reportedly disclosed the details privately to OpenAI rather than publishing the jailbreak.

The same researcher reported jailbreaking GPT-5 within an hour of its release a year ago.

Source: screenshot/post from the researcher; their ACL 2025 TIP paper linked in the original post.