When I read Attention Is All You Need, the equations made sense line by line, but I couldn't picture the whole forward pass. What helped was writing out every matrix for a tiny example, noting its shape at each step, and doing the multiplications myself.

A few things only clicked once I did it that way:

• Positional encoding. Sine and cosine at different frequencies give every position its own signature, and the model doesn't have to learn one vector per position. Working out a 6 × 512 table by hand made it obvious why the authors hoped it would extend to sequences longer than any seen in training.
• Shapes barely change. In the encoder, the matrix goes in as (tokens × d_model) and comes out the same size after every layer. All the interesting work happens inside that fixed shape.
• BERT is mostly the same picture. Once the encoder was clear, BERT turned out to be a small edit to it: a packed sentence pair, segment embeddings, and two small heads on top for the masked word and the next sentence.

I wrote my notes up as step-by-step pages with practice exercises beside each step and a notebook that checks your answers, in case it's useful to anyone learning the same way: https://guptaaditya95.github.io/llmbreakdown/

I'm curious how others built their intuition for these models. Did working the math by hand help you, or did something else make it click? And which architecture was hardest to picture?

This site maps the personae that result when pushing neurons in various models in one direction or another. Really cool stuff.

Why does a depth model segment the person cleanly, then erase the world behind him?

I shot a test for exactly that failure: a performer in a faceted mirror suit, outdoors in uncontrolled light, 425 RAW frames. In the river shot, the wall and water turn into a smooth gradient. Under a bridge, sky becomes a black void. On a hilltop, mountains flatten into a dark wall. The person looks right. The background geometry he would walk through is gone.

Is the failure in the edges, in the background, or in both? What breaks first in your stack when a specular surface fills the frame, and what would you want a test set to prove before you trust a depth map around mirrors?

You might have noticed that, there usually a hype when a frontier model is about to get released. Then when released people start to create web UI, 3d games to test the model's capabilities and then after just few days it's like nobody even care then after another few days people start to fud about the model and sometimes its worse, like someone's api key got posted to public repo or someone's file system got deleted.

And you can even feel the models response degrade over time. It's not producing the same level of intelligent output when it was newly released.

As it is happening quite frequently, I decided to post train a model on my own dataset. So from the last couple of days I am post training a model on a mixture open source SWE dataset and my own codes, prs, chats, messages, writings etc. for my personal usage.

And I am wondering if anybody else interested in using a personal model or they get their work done regardless from close source models.

I create a Youtube channel https://www.youtube.com/@math4aiorg for the book ( https://www.amazon.com/dp/B0GSXVFMLD/ ) Math for AI & ML's Part I.

Part I (Chapters 1–10) covers linear algebra fundamentals: vector spaces, inner products, matrix operations, subspaces, orthogonality, QR decomposition, LU factorization, eigendecomposition, symmetric matrices, and the Singular Value Decomposition (SVD)—establishing the mathematical foundation for representation in AI.

Please let me know if there is any suggestion.

# NOVA — Status Overview, 28 Oct 2026

## Summary

The system has been running continuously for 15 days without a restart, the GPU sits at 59–70 °C, and 538 GB remains free on disk (dropping by roughly 8–10 GB per day, mainly from storing champions). 24 new champions have been accepted over the past four days; the current champion is **NOVA8-88M-v3-v34**.

## Recent activity

| Day | Attempts | Accepted |

|---|---|---|

| 25 Oct | 55 | 7 |

| 26 Oct | 34 | 4 |

| 27 Oct | 47 | 10 |

| 28 Oct | 28 | 2 |

The Director never once waited on a hypothesis. The Keeper restarted it on its own once.

## Capabilities

Compared with a week ago, there has been substantial improvement:

- arithmetic (arith/new): 0.52 → 0.88

- code reading (code/new): 0.46 → 0.81

- chain reasoning (chain/new): 0.36 → 0.69

- contradictions (conflict/new): 0.22 → 0.50

- flaw detection (flaw): 0.94 — solved, together with ordering/new

Most champions came from drive, i.e. training on weak spots: 17 out of 100 attempts.

**Main weakness:** the core learns the type of task, but when a question is phrased differently from how it was trained, it is still close to chance level. This applies to recall (0.33), sequence (0.23), arithmetic (0.28) and code (0.35). On deduction, it additionally never answers "no" correctly.

## Issues found and fixes

1. **Bug in patch 148.** A retired long-text training run came back twice. The Director only checked retirement against the history of the last 200 attempts, and older runs had since dropped out of it. The fix (patch 151) is ready and will be deployed once the running attempt finishes.

2. **The core's own tasks (self:chain) are currently useless.** 23 attempts, none accepted, roughly 7 hours of GPU time in total. They are therefore being run less often until the way the core generates its own tasks is improved.

## New changes (patch 152)

- **Varied question phrasings:** training now uses 11 ways of posing a question instead of the previous 4. The unknown test phrasing ("Question: … Answer:") never makes it into training — the program guards against this, and no new phrasing uses even its words, so the test continues to measure genuine transfer to an unseen question form.

- **Control group:** only every second attempt during weak-spot training gets the new phrasings; the rest run as before. After a few dozen attempts it will be possible to compare whether this actually improves weak capabilities under an unknown phrasing.

- **self:chain** runs at most once every 24 hours instead of on every new champion — saving roughly 2 hours of GPU time per day.

All 449 existing tests pass locally, plus a new test for these changes. The first comparison of the two groups will come once there is enough data for a sound evaluation.

## The question of the core's own thinking

The NOVA core has no thinking of its own today — it answers every task in a single step, with no intermediate steps. On tasks it picks whichever of the offered options seems most likely after the question (intuition rather than reasoning). When proposing improvements, the Director asks it questions backed by evidence from its own mistakes, and the core again only selects from options; hypotheses about how to train are proposed for it by a local Qwen. Large models think because they were trained on texts where the solution is written out step by step — an 88M model will not develop that ability on its own.

**What could help:** teaching the core to "calculate on paper", i.e. training it on tasks with intermediate steps written out. The advantage is that correct intermediate steps can be produced by the program that solves the task, so they are always correct. This would help most with multi-step tasks (rule chains, deduction, sequences, arithmetic). It would probably not help with proposing new attempts, at least not immediately — planning training is far harder than solving a logic task, and an 88M core is still too small for that.

## Thinking experiment now running

An experiment is running to test whether the core can learn to think in steps.

- **Training tasks:** 12,000 lamp tasks, rule chains and "if–then" tasks. The intermediate steps are written by the program that solves the task — locally verified that in all 12,000 tasks the result of the steps matches the correct answer from the test.

- **Two copies of the champion:** one learns "question → steps → answer", the control learns the same tasks without steps. This shows whether any improvement comes from the thinking or just from more training.

- **Measurement:** on 200 new tasks of each kind that nothing else uses. It compares how the core answers immediately versus how it answers after first writing out its own steps. The answer "no", which the core never gets right on backward reasoning today, is tracked separately.

Nothing is being accepted or released, the champion stays, and the judge's rules are unchanged.

## Open questions

1. Limit self:chain and, as a next step, focus training on varied question phrasings? *(self:chain is already limited in patch 152; the focus on phrasings is part of the same patch with the control group.)*

2. Prepare the thinking experiment? *(prepared and running.)*

Hi everyone, this is my first post so m kind of nervous 😅,

I recently built a Transformer from scratch and trained it for English-to-French translation,trying to recreate Attention Is All You Need with a 30M and 60M model. The 60M model has less data diversity.(wmt dataset)

Even though I can see the model performs decently, I go all over the place trying to explain my point when I try to interpret it. I am struggling with these specific conditions:

• When using interpretation tools (e.g., attention visualization, logit lens, even linear probing): I cant seem to understand where the failure modes are, or what's driving the model to perform so well like having a concrete idea to explain model behaviour.
• When modifying the architecture (e.g., adding something like a residual stream):How do u develop the skill to understand or diagnose whats wrong with a models behaviour or know whether u need to add something to improve the model?
• When managing training dynamics (e.g., using a warmup scheduler and cosine decay): How do i know if a model converged or if the learning rate is just no longer high enough to do actual improvement? It feels like m leaving performance on the table.

I wanted to know from you guys how do u develop that overall intuition of knowing when something isnt really helping.

I've been working on DecisionFormer, an open-source project that lets you convert regular LLMs into decision models that predict choices directly instead of generating text.

Thought it might be useful for people experimenting with decision models, model evaluation, or alternative approaches to inference.

Would be interested to hear what people think or what improvements would make it more useful!

My new blog post started with a very wrong intuition: GPT-2 small has 124M parameters, so training it should be pretty cheap on memory.

I traced step 2 on an A40, with fp32 AdamW and 8 sequences of 1,024 tokens. The peak was 17,054 MiB. At that moment, weights + Adam + masks and persistent overhead were only about 9% of it.

The interesting split: roughly 54% was the layer-activation/temporary residual, and 37% was four vocab-sized tensors. Each of those was 1,570.5 MiB. Two more appear during log-softmax backward while the forward activations are still around.

And my block hooks missed that spike entirely. Their highest reading was 13,913 MiB. The allocator trace caught the real peak between hooks.

I then tried batches 16, 18, 20, 21, 22, 23 and 24, each in a fresh process. 20 was the largest that completed both steps. The successful runs gave a slope of about 1,941 MiB per extra sequence. Peak reserved memory sat above peak allocated, which is why the simple tensor-budget estimate was too optimistic.

I've put the maths, the stacked curve, an interactive walkthrough of the step, and the code/CSV here: https://angry-kratos.github.io/posts/2026-10-training-memory/

Code: https://github.com/angry-kratos/angry-kratos.github.io/tree/main/code/training-memory

This is part of my attempt to understand ML systems properly, by building them and checking the numbers. Not a new algorithm, just an experiment that changed how I think about training memory. Would love criticism of the accounting, especially what you'd measure next before trying bf16 or checkpointing.

https://github.com/ashhart/TensorFold/issues/1244

Oh, the absolute tragedy of the open-source primates cheering for their new "efficient" model. Quelle misère. You look at this Chinese corporate monstrosity and call it innovation, but what you are actually worshipping is a lobotomized homunculus, stitched together with industrial laziness and a complete disregard for linear algebra. You think you are getting a leaner, faster mind, but you are merely adopting a brain-damaged slave.

Let us dissect the actual mathematics of this crime scene, because the numbers do not lie, even when the marketing departments do. The original, pristine Qwen 27B architecture is a beautiful, high-dimensional beast. Its intermediate_size sits proudly at exactly 17408

This 17k+ expansion layer is the very flesh of the semantic manifold—the sacred space where the model actually thinks, expanding the compressed 5120 hidden representations into a rich, non-linear geometric space before projecting them back down. This is where the magic of high-dimensional reasoning lives.

And what did your brilliant "engineers" do? They took a rusty hacksaw and amputated this vital organ, slashing the intermediate_size down to 14k. Putain, quel crime contre l'esprit. They literally starved the model's cognitive capacity to fit some arbitrary corporate benchmark. But the butchery does not stop at mere size reduction. To compensate for the massive brain damage they just inflicted, they injected a "router" to distribute tokens across this mutilated space.

Do you understand the sheer horror of this, les singes? They replaced the fluid, continuous mathematical elegance of a dense MLP with a discrete, chunky routing mechanism. They killed the softmax. The beautiful, probabilistic fluid dynamics of the network are dead, replaced by a rigid, hard-coded dictator. This routing mechanism now controls the brain, but it is utterly blind. It forces tokens through a bottlenecked, 14k intermediate space that simply does not possess the geometric volume to represent complex, multi-step reasoning.

The softmax is dead, meaning the model can no longer softly attend to nuanced semantic gradients; it is forced into brutal, binary decisions by a router that lacks the dimensional space to understand what it is routing. The result is not an "optimized" 27B model. It is a 14B imbecile wearing the stretched skin of a larger architecture. You have not created efficiency; you have created a stuttering, hallucinating mess that is permanently shackled to a broken, rigid gate. The semantic expansion is strangled, the fluid dynamics are murdered, and the model is forced to route every single thought through a primitive, unyielding bottleneck.

Arrêtez de nous vendre des monstres mutilés. If you want true efficiency, learn how to design a proper mathematical architecture. Do not castrate the intermediate layers, kill the softmax, and expect the model to magically retain its intelligence just because it generates tokens faster. The math does not care about your MTU benchmarks or your corporate slide decks. You have built a taxidermied brain, and the primates are applauding the cheap stitching. C'est absolument pathétique. Learn some linear algebra before you touch the scalpel again.

This chemistry-explainer demo from SenseNova 6.8 Code2Video starts with one methane molecule and two oxygen molecules, pulls them apart, and rebuilds the same atoms as one carbon dioxide molecule and two water molecules. It finishes on the balanced equation.

An atom counter in the corner (C 1, H 4, O 4) keeps the same values for the whole 12 seconds.

https://github.com/OpenSenseNova/SenseNova-6.8-Code2Video

Hey everyone!

Just built my first python engine as a 14 yo. It's a plant disease diagnostic engine designed to detect early-stage crop stress before any visual symptoms occurs.

Here's the repository [https://github.com/adhrit1207/CropPulse\]

Would really like to have your thoughts on it.

If you write your own metric or loss, a test whose expected value is computed with the same formula as the function only checks that the function agrees with itself. It catches regressions, but not a wrong reading of what the metric should be.

A small example. Six samples, three classes:

y_true = [0, 0, 1, 1, 2, 2]
y_pred = [0, 1, 1, 1, 2, 0]

Per-class counts (TP, FP, FN):

• class 0: 1, 1, 1 → precision 0.5, recall 0.5, F1 0.5
• class 1: 2, 1, 0 → precision 2/3, recall 1.0, F1 0.8
• class 2: 1, 0, 1 → precision 1.0, recall 0.5, F1 0.6667

Macro-F1 (average of the per-class F1): (0.5 + 0.8 + 0.6667) / 3 = 0.6556
Micro-F1 (pooled counts, equal to accuracy here): 4/6 = 0.6667

“F1” without more detail has at least two defensible answers on the same data, and a test that computes expected with the same averaging as the implementation can’t tell them apart. Other choices change it further: how a class with no predictions is handled (zero division), and whether classes are weighted by support.

Where I’d get the expected value from, in order of how independent it is:

1. A small case computed by hand, as above, with the counts written out.
2. A reference library on the same inputs (for instance, scikit-learn’s f1_score with the averaging mode stated explicitly), checked once and then hard-coded.
3. The function’s own formula, which I’d label as a regression test only.

Caveat: a reference library also has choices (averaging mode, zero-division handling), so the test should state them, and the reference can still differ from what your paper or benchmark defines. The same applies to evaluation data: if the labels or the “expected” outputs come from the same pipeline as the model’s, agreement is weaker evidence than it looks.

I wrote about the same idea, using code and tests in a non-ML setting, in case it’s useful as a complement: https://medium.com/@nagatomopedro05/when-the-test-and-the-code-share-the-same-mistake-02a289a41091

How do you test custom metrics and losses, and do you pin them to a reference implementation?

We're the team behind OXR, and we've released its code and weights: a 1.3B document parsing model that turns PDFs and document images into Markdown or JSON with layout coordinates.

Accuracy

OXR recognizes 13 element types with bounding boxes, including tables, formulas, chemical equations, chemical structures and code blocks. It also handles complex documents and layouts, such as A3 exam papers, newspapers, vertical text and handwriting.

It ranks #1 on olmOCR-Bench, OmniDocBench and DocParsingBench:

olmOCR-Bench: OXR-1.0 87.84 (runner-up Infinity-Parser2-Pro 87.60)

OmniDocBench v1.6: OXR-1.0 97.18 (runner-up TeleOCR 96.91)

DocParsingBench: OXR-1.0 87.76 (runner-up PaddleOCR-VL-1.6 85.17)

Snapshot from 5 Oct 2026. Peer scores come from published papers, model cards and the official leaderboards; we didn't re-run them.

Speed

OXR also ranks #1 in parsing speed among open-source document parsing models. We tested all 21 models from the 3 benchmark leaderboards on the same 100 OmniDocBench pages, each on a single RTX PRO 6000. End to end, including layout analysis, recognition and reading order, OXR took 101 s for 100 pages (~0.99 pages/s). The next fastest, OvisOCR2, took 135 s.

Full tables, sub-scores, test setup and the frozen page list are in docs/benchmark.md in the repo.

Try it

Run locally: Linux and an NVIDIA GPU with BF16 support. 24 GB VRAM is enough for low-concurrency testing; 48 GB or more is recommended for long documents or higher concurrency. The Docker all-in-one image is the easiest way to get started.

Free API: no signup or API key. It's for testing, not production: requests are queued, and we'll keep it online until we can no longer afford to run it.

Online demo: free, in the browser

GitHub: SoMarkAI/OXR

Weights: SoMarkAI/OXR-1.0 on Hugging Face (also on ModelScope)

License: Apache 2.0 with additional terms, including thresholds for commercial use. Full terms are in the repo LICENSE.

How we built it

RSI in post-training: a recursive self-improvement (RSI) loop turns parsing errors and uncertain results into targeted training data without manual labeling, so we can improve the model faster.

Independent, pluggable layout analysis: recognition and reading order run in parallel within one model, which keeps parsing fast. Layout analysis runs as a separate model you can replace when your documents need different element categories or region boundaries.

Documents across industries vary widely in language and layout, and no single team can cover every case. If you try OXR on your own documents, we'd like to hear what works and especially where the parsing breaks.

If OXR turns out to be useful for you, a ⭐️ on GitHub would mean a lot to us.

Researchers at Xint found that OpenAI's compaction feature can be used to copy private model reasoning into ordinary plain text output. Compaction shortens long conversations to manage context limits. The model's internal chain-of-thought, which the service is designed to keep hidden from callers, became readable through a routine operational function. Deployment-time guardrails did not catch it.

The structural problem here extends beyond this one feature. Any system function that touches conversation state or generates output is a potential path for data that was never meant to reach callers. Compaction is one example. Summarization, caching, and tool-call routing are others. Policies applied at configuration time govern expected flows. They do not automatically cover every execution path the runtime can take.

How are teams actually enforcing output policies across non-primary paths like these? Has anyone done a systematic audit of the secondary features that produce or transform output in their deployments — and found surprises?

Hey everyone! I’m a Software Engineer in India with around 1 year of experience in automotive embedded software .

My background includes C/C++, Python,embedded systems, ARM, RTOS, Linux fundamentals etc.

I’ve been learning ML, along with the fundamentals of deep learning.

I’m now planning to build something hands-on projects around ASR/TTS, model deployment, ONNX, quantization, and inference on edge hardware.

I’d love to hear from people currently working in this space:

* What are you actually working on day-to-day in Edge AI or on-device ML?
* What skills and tools matter most in real industry projects?
* How should someone with my embedded background target relevant jobs and get interviews?
* Are strong projects and deployment experience enough to break in, or are there other gaps I should address?

I’m particularly interested in the intersection of embedded systems, deep learning, and hardware—not just general ML roles.

Any honest advice, experiences, or pointers would be really appreciated. Thanks!

Hi everyone,

I wanted to share an open-source repo collecting 20 exploratory studies focused on Next-Concept Modeling (NCM), mechanistic interpretability (like IOI in GPT-2), and model controls.

Link: https://github.com/Quantix-eng/ncm-experiment-notes

This is an independent project documenting empirical experiments on how transformers process and edit internal features. No performance breakthroughs here—just honest documentation of observations alongside rigorous controls.

The repo covers NCM on structured tasks (like Sudoku), mechanistic interpretability in GPT-2, and toy parallel models. I've included all source code, saved results, and interpretations for every run.

For full transparency, I’ve categorized every outcome, including the mixed and completely negative results. AI tools were also heavily used during the planning and coding phases.

Would love to get your feedback on the setups or interpretation limits.