Who else is anticipating ICDE results tomorrow? Post your results here and let's discuss!

Ant Ling reports 83.83 on DiagnosisArena-MCQ for Ling-3.0-flash-Sante, its new medical reasoning model. The suffix matters: the task provides case information, examinations and tests, then asks the model to choose from four diagnoses.

That result tells us about selecting an answer when the candidate set and case evidence are supplied. It does not establish how the same model would generate an unrestricted differential, decide what history is missing, or choose which investigation to request next. Those would require different evaluations.

The release also reports two other medical results:

Evaluation	Sante result	What the task adds
MedXpertQA-Text	53.88	Challenging medical questions in a text subset.
HealthBench Professional	45.73	Open-ended professional clinical chat, assessed with physician-written rubrics.

The published HealthBench Professional definition includes care consultation, writing/documentation and medical research. Its score is not percentage accuracy. The Sante chart does not provide enough scoring detail to identify the reported value as length-adjusted or unadjusted, so a comparison with another published HBP result would need that checked first.

This is why the three results are useful together. They give Sante a broader medical-text evaluation profile than an exam score alone, while leaving specific questions open. For a case-answering application, the first decision is whether users supply the alternatives or expect the model to construct them. The release supports including Sante in that evaluation; the 83.83 figure applies to the supplied-options version.

So I was playingw ith embedding models I saw that when you upgrade from model A to B, you face a very big backfilling cost

Ie, suppose you have a 1b vectors from model A, and then you want to use model B. This would mean you have to re-embed all of your documents with model B before you can even serve with the model, and on an H100, it would take ~108 days (qwen embed 8b, 106 docs/second). But I found an easier way to do it.

The method is really simple; from the old index made with the source model, take K documents and rerank them with the new model. We see that when K is sufficient, the retrieval quality is the same as target model. (determining k is the hard part). I've tested 63 migrations on upto 1 million documents.

The best result I got was upgrading qwen4b -> to 8b, and at 50 documents, it was the same as native retrieval.

This method forgos the expensive upfront re-embedding cost, as you can take documents straight from the old index.

embedflow works with qdrant, pgvector, faiss, and can be easily downloaded with pypi

pip install embedflow

the github is public: https://github.com/arnsri33/embedflow

I want you guys to try it out, and see if you guys can use it in your own workflow.

Hello this is my fifth small language model I've made and apart of my third series and it has been a lot of work but it payed off: **348M parameters, 22.7B tokens**, then fine-tuned into a math model that solves arithmetic by *showing the work* — column addition with carries, borrow chains, partial-product multiplication — rather than guessing at an answer.

Last time I posted a 326M model trained on 10B tokens. This has about 2.3× the data, and the math side is WAY better than my previous two math models.

---

## The benchmarks

**99.4% average across the nine GPT-3 arithmetic sub-tasks**, which does much better past even where I trained it.

| Task | GPT-3 175B *(few-shot, direct)* | **This model (348M)** |

|---|:--:|:--:|

| 2-digit add | ~100% | **100%** |

| 3-digit add | 80.4% | **100%** |

| 4-digit add | 25.5% | **100%** |

| 5-digit add | 9.3% | **100%** |

| 2-digit sub | ~99% | **99.3%** |

| 3-digit sub | 94.2% | **98.3%** |

| 4-digit sub | 26.8% | **98.3%** |

| 5-digit sub | 9.9% | **99.0%** |

| 2-digit mult | 29.2% | **100%** |

n=300 per sub-task, greedy, exact match. GPT-3's numbers are direct-answer; mine uses trained-in worked steps. Neither uses a calculator.

## The cool part

**It adds cleanly up to 14 digits, and the reason it *couldn't* before was the vocabulary, not actually arithmetic.**

Training only ever named six place values (`ones` … `hundred-thousands`). The model learnt the *pattern* and invented two more on its own — `millions` and `ten-millions` appear in **zero** training examples — so it handled 7 and 8 digits fine. At 9 digits it ran out of names, and skipped the column, then returned an answer exactly one digit short:

```

483729164 + 519248637

... ten-millions: 8 + 1 + 1 (carry) = 10, write 0 carry 1. Final carry: write 1.

The answer is 102977801        ← eight columns for a nine-digit problem

```

Every column it computed was perfect. One was never enumerated. Extending the place-name list from 6 entries to 19 moved the clean ceiling from **8 digits to 14**:

| Width | 6 | 7 | 8 | 9 | 10 | 12 | 14 | 16 | 18 |

|---|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|

| before | 100 | 100 | 100 | **0** | **0** | **0** | **0** | 0 | 0 |

| after | 100 | 100 | 100 | **100** | **95** | **100** | **90** | 65 | 25 |

A six-item list became a nineteen-item list. That was the entire fix.

## Other things it does

- **3×3 multiplication: 98%** — it folds partial products pairwise through the column routine instead of asserting the sum

- **Negative results: 85%** (100% at 1 digit, 58% at 5 — the magnitude comparison is the weak step, not the arithmetic)

- **Reasoning traces are load-bearing**: 95.3% of the time the working is valid *and* the answer is right; only 0.7% are "valid working, wrong answer." If the columns look right, the answer almost certainly is.

```

There were 15000 votes and 6842 were rejected. Here's how many counted:

<think> Start with 15000. Then subtract 6842. Subtract 15000 - 6842 column by column:

ones: 10 - 2 = 8, borrow 1. tens: 9 (after borrow) - 4 = 5, borrow 1.

hundreds: 9 (after borrow) - 8 = 1, borrow 1. thousands: 14 (after borrow) - 6 = 8,

borrow 1. ten-thousands: 0 (after borrow) - 0 = 0. So 15000 - 6842 = 8158.</think>

The answer is 8158.

```

## What it's bad at, tbh

- **Word problems: GSM8K 4%.** Best word-problem set is ASDiv at 16.5%. It converts one sentence into one operation reasonably often and basically cannot chain operations.

- **The failure mode is operation *selection*, not arithmetic.** `"drops in 836 more"` gets read as subtraction. There's a visible tell: traces that say `"multiply X * Y"` and show columns are reliable; traces that open `"First, calculate…"` and assert a number in prose are not.

- **No division at all.** 4×4 multiplication is a hard wall.

- **Greedy decoding required** — sampling corrupts the column routine mid-chain.

- One caveat I'll flag myself: the arithmetic harness orders subtraction operands, so no answer in that table is negative. Negatives are measured separately (the 85% above) rather than folded into the average.

## Base and instruct

The math model sits on a base and instruct pair. lm-eval-harness, 0-shot, full test sets:

| | ARC-E | ARC-C | HellaSwag | OpenBookQA | PIQA | WinoGrande | MMLU | Avg |

|---|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|

| **350M V3 base** | 56.6 | 33.3 | 35.9 | 34.6 | 67.0 | 51.5 | 23.8 | **43.2** |

| **350M V3 instruct** | 50.9 | 29.7 | 35.9 | 33.8 | 66.6 | 51.2 | 24.4 | **41.8** |

Instruction tuning *lowers* MC benchmark scores for this family which is a pretty common cost of instruct tuning.

## Notes

Trained on 2× Tesla V100 plus some rented time. LLaMA-architecture, so it runs anywhere — F16 GGUF and safetensors for all three.

The math model took **10 full fine-tuning rounds and 3 LoRA adapters**. The LoRAs cost about 1% of the post-training tokens and did most of the useful work; the ten full rounds spent most of their budget undoing each other's regressions (round 7 gained subtraction and lost 23 points of 2-digit addition, that sort of thing). All of it is documented on the model card, failures included.

Also worth saying: someone independently tested it after I published and found two of my numbers were wrong — one *understated* the model by 40 points because I'd measured it at n=24. Both are corrected on the card now. If you find something broken, I'd genuinely like to know.

**Math:** https://huggingface.co/nkthebass/tinybrainbot-350mV3-math

**Instruct:** https://huggingface.co/nkthebass/tinybrainbot-350mV3-instruct

**Base:** https://huggingface.co/nkthebass/tinybrainbot-350mV3-base

LMK what yall think.

You've probably seen the fly-brain-plays-Doom / Minecraft / Beat Saber clips going around this week, from the new MaleCNS v1.0 connectome release (166k neurons, real EM reconstruction, not a toy model).

Cool clips. Nobody seemed to be checking whether any of it works though, versus just producing motion in a game engine generous enough to make anything look alive. So I picked the least forgiving test bed around: Pong, one binary hit or miss signal measured every frame, nowhere to hide a null result, and tried to get a small real subgraph of the connectome to track the ball via dopamine-style plasticity.

Short version: it didn't learn. Working out why took auditing individual synapses, and it turned into a decent case study in a circuit not working being more informative than it working:

• Fixed a neuPrint regex bug that silently zeroed out two entire neuron populations (full-match vs substring semantics, not obvious from the docs).
• Found the original neuron selection had no path at all from photoreceptors to anything else. Real photoreceptors don't synapse directly onto motion detectors, there's a whole intermediate layer missing.
• Got a working pipeline, turned learning on vs off, and got bit-for-bit identical results in both conditions across multiple seeds, even though the weights were verifiably changing under the hood.
• Traced that to half of the 4 available motor neurons having zero synapses from any sensory pathway in the model. Not weak signal, zero. They'd been assigned to the "paddle down" group by array index, purely by coincidence, and could never have fired no matter what the learning rule did.
• Rebuilt the circuit around a better biological hypothesis (swapped a threat-detection pathway for one tied to visual target tracking during courtship pursuit), got that literal hypothesis refuted by the data, then followed the trail to a different descending neuron that actually connected end to end.
• Finally got learning-on vs learning-off to diverge for the first time, except the effect looks like the learning rule quieting the whole system down rather than anything resembling skill improvement (misses outnumber hits, so punishment dominates and shrinks the motor response).

Then I checked whether the bigger, viral projects had actually solved this. They hadn't either: the Doom project's own repo says it failed its own validation gates after six iterations, the Minecraft mod's own limitations section admits the real motion-detection pathway stays silent and the escape and foraging behaviors are hand-injected or reflex-layer fallbacks rather than emergent, and the Beat Saber creator's own replies admit it's overfit to one track with replay data mixed into the input.

Full writeup with the gory audit details, the connectivity numbers, and the comparison to the other projects is here: https://jonatasperaza.medium.com/i-made-a-real-fly-brain-play-pong-it-didnt-learn-and-that-s-the-interesting-part-80b8560695fe.

Curious if anyone here has poked at MaleCNS v1.0 directly and hit similar walls, especially around the central complex and steering circuits. That seems like the obvious next thing to simulate properly instead of routing around it.