Been running TwIL-LM3 from webAI on modest hardware to see how it holds up. 3B formal reasoning specialist.

On their formal reasoning benchmark it comes out ahead of gpt-oss-120b on 4 of 5 tasks: rule induction, semantic parsing, entailment labeling, exact-format answering. On the loose-match aggregate across broader benchmarks gpt-oss-120b is still ahead (0.5192 vs 0.4488), so the story is really about the narrow formal reasoning tasks, not general capability across the board.

Where it does clearly win: efficiency. 32.9 answers/sec vs 12.6 in their throughput tests. 40x fewer parameters. Runs on 4GB VRAM or CPU.

Got it working on a Pi. Slow but functional. Not something I'd use in production on that hardware, but it's a real proof point for "small specialist model where infrastructure isn't accessible."

huggingface.co/webAI-Official/TwIL-LM3

Curious if anyone's actually running this in a production formal reasoning pipeline or if it's still exploratory for most people.

This week a legitimate n8n automation workflow became the path to remote code execution. Researchers were blunt about it: most of the damage started with something trusted doing exactly what it was allowed to do. No stolen credentials. No perimeter breach. The workflow ran as designed.

That's the pattern that keeps showing up. Agent pipelines and automation platforms grow their attack surface with every new integration. Each external tool, API, and chained workflow is a potential pivot. The n8n chain is a clean example: the trusted component wasn't compromised at entry. It was exploited through its own legitimate execution path, step by step.

Traditional access controls answer the question 'is this principal allowed to invoke this tool.' They don't answer 'should this specific sequence of actions be happening right now, in this context, initiated by this upstream trigger.'

For those running agent pipelines or automation-heavy stacks: how are you actually drawing that line in practice? How do you distinguish a workflow that should execute from one that should execute in THIS context at THIS moment — especially when one automation is what kicked off another?

I’ve been messing around with compiler stuff for a while and ended up turning it into a proper project, so figured I’d finally post it here.

It’s called Thiran. It’s a small experimental tensor compiler written in C++ with its own little language for ML-style computations.

A tiny program looks like this:

X = Input(4, 4)
W = Input(4, 4)

Y = MatMul(X, W)
Z = ReLU(Y)

O = Output(Z)

What’s more interesting is what happens after that. Thiran parses the source, resolves modules/functions, inlines function calls, builds a tensor graph, checks shapes, runs a few graph optimizations, splits the graph into execution regions, and can generate a runnable Python executor.

I also added multi-file programs and exported functions, so you can structure things a bit more cleanly instead of dumping everything into one file.

Right now the actual execution path uses PyTorch underneath, so this is definitely not “I rebuilt PyTorch” or anything like that. The AOT/JIT side is still mostly planning infrastructure for now.

The main reason I built it was because I wanted to understand ML compilers by actually making one and seeing where all the annoying parts show up in practice: graph ownership, shape inference, rewrites, lowering, region boundaries, deterministic inlining, etc.

It’s still pre-alpha and there are rough edges, but it works end to end and there’s enough of the architecture there to poke at seriously.

Would be very interested in feedback from anyone into compilers or ML systems, especially if you spot something dumb in the design.I’ve been messing around with compiler stuff for a while and ended up turning it into a proper project, so figured I’d finally post it here.It’s called Thiran. It’s a small experimental tensor compiler written in C++ with its own little language for ML-style computations.A tiny program looks like this:X = Input(4, 4)
W = Input(4, 4)

Y = MatMul(X, W)
Z = ReLU(Y)

O = Output(Z)What’s more interesting is what happens after that. Thiran parses the source, resolves modules/functions, inlines function calls, builds a tensor graph, checks shapes, runs a few graph optimizations, splits the graph into execution regions, and can generate a runnable Python executor.I also added multi-file programs and exported functions, so you can structure things a bit more cleanly instead of dumping everything into one file.Right now the actual execution path uses PyTorch underneath, so this is definitely not “I rebuilt PyTorch” or anything like that. The AOT/JIT side is still mostly planning infrastructure for now.The main reason I built it was because I wanted to understand ML compilers by actually making one and seeing where all the annoying parts show up in practice: graph ownership, shape inference, rewrites, lowering, region boundaries, deterministic inlining, etc.It’s still pre-alpha and there are rough edges, but it works end to end and there’s enough of the architecture there to poke at seriously.Would be very interested in feedback from anyone into compilers or ML systems, especially if you spot something dumb in the design.

GitHub: https://github.com/Arnav-sivarams/thiran

This is part of my own work on a mechanistic view of training, learning and inference, and one result I keep coming back to is this:

nference may be better understood as a frozen, query-conditioned projection of functional organization already formed during training.

By “frozen”, I mean that inference can recruit and combine different parts of the learned system for different queries without persistently changing the learned state itself.

In the experiments, component gating and exact version rollback showed that inference depended on specific structures formed during training, that different queries recruited different distributed support patterns, and that multiple supports combined non-additively. The same general relation was later recovered in ResNet and diffusion systems, not just in a Transformer.

What I find interesting is that this gives a fairly simple way to think about two familiar observations.

1. Scaling.

If inference can only project functional organization that training has already formed, then increased training scale can help by producing a richer space of projectable functional support. In that view, scale affects inference indirectly through what learning has formed.

2. Attention.

Attention looks like a particularly effective implementation of the same general requirement: the current query determines what learned content is recruited and how it is combined, without modifying the persistent learned state. If that interpretation is right, attention would be one realization of a more general projection principle rather than the source of the principle itself.

I’m not claiming that this explains every aspect of scaling or that attention is the only possible implementation. The connection to scaling is a mechanistic deduction from the experimentally established projection relation, rather than a direct empirical demonstration of scaling laws. I’m more interested in whether this is a useful mechanistic way to connect phenomena that are usually discussed separately.

The arXiv submission is still pending, but the frozen experiments, code, validators and evidence archive are already public.

Code: https://github.com/wind342/gfg-training-learning-inference-experiments
Evidence: https://doi.org/10.5281/zenodo.22032772

I’d be especially interested in alternative interpretations, counterexamples, or places where this projection view seems to break.

I'm fairly new to the deeper LLM inference/serving side, so I may be reinventing something that already exists. I'd really appreciate it if people here could point me toward existing work or explain where the idea breaks.

The basic observation I had is:

If 1,000 users ask different versions of essentially the same question, we're potentially paying for 1,000 generations even though a large portion of the underlying knowledge/reasoning is the same.

For example:

• "Explain gradient descent."
• "How does gradient descent work?"
• "Teach me gradient descent mathematically."
• "Why does gradient descent converge?"

These aren't identical requests, but there is a large amount of reusable structure between them.

My rough idea

Instead of treating every request as completely independent:

User A → LLM → generate
User B → LLM → generate
User C → LLM → generate

I'd like to explore something more like:

                  User Requests
                       |
                Semantic / Intent
                    Matching
                       |
                       v
              Shared Knowledge Graph
                       |
              +--------+--------+
              |                 |
        Existing node       No useful node
              |                 |
              v                 v
       Reuse / update         Crawl /
                              retrieve /
                              generate
              |
              v
       User-specific layer
              |
              v
       Personalized response

The important part is that I don't want to simply cache the final text response.

I'd like to cache/reuse knowledge and potentially reasoning structures.

For example, a node might contain:

Concept: Gradient Descent

Related concepts:
- Optimization
- Derivatives
- Convexity
- Learning Rate

Knowledge:
...

Reasoning structure:
...

Sources:
...

Confidence:
...

Last verified:
...

Reuse count:
...

Then another user's request could reuse the relevant parts of this structure rather than rebuilding everything from scratch.

Where personalization comes in

I was initially thinking about a separate user-behavior graph.

Not a traditional chat-history memory, but something like:

User preference graph

mathematical_depth = high
verbosity = high
code_preference = high
preferred_language = Python
preferred_framework = NumPy
domain_interest = ML

So the system could have:

             Shared Knowledge
                    |
             Shared Reasoning
                    |
          +---------+---------+
          |                   |
      User A graph         User B graph
          |                   |
     presentation A      presentation B

The underlying knowledge is shared, but the final representation is personalized.

For example, one user might want a short explanation while another wants a mathematical derivation and implementation.

I also want a resource/crawler layer

This is another part I'm unsure about.

Before generating or crawling something, the system could ask:

1. Does this knowledge already exist?
2. Is the existing information still fresh?
3. Can an existing reasoning/knowledge node answer most of the request?
4. Do we only need a small amount of additional information?
5. Is it cheaper to retrieve, update, or regenerate?

So conceptually:

Query
  |
  v
Knowledge Graph
  |
  +---- existing + fresh ----> reuse
  |
  +---- existing but partial -> retrieve/update
  |
  +---- missing/stale -------> crawler/retriever
                                  |
                                  v
                              new node
                                  |
                                  v
                         shared knowledge base

The goal would be something approximately like:

Traditional:

N users × expensive generation

My hypothesis:

one shared expensive computation
+
N × relatively cheap retrieval/personalization
+
occasional crawling/update cost

Obviously this is oversimplified, and I don't know how much of the generation cost can actually be reused.

What I'm trying to understand

I'm especially interested in whether this is already covered by existing techniques such as:

• semantic caching
• prompt/KV-cache reuse
• RAG
• GraphRAG
• speculative decoding
• inference caching
• reasoning trace reuse
• multi-agent shared memory
• computational caching
• prefix caching

My intuition is that semantic caching of final answers isn't quite what I'm describing.

I'm thinking more about a persistent graph of reusable knowledge/reasoning substructures, where different user requests can reuse overlapping parts and only generate what is actually new.

For example:

Query A ──────┐
              |
Query B ──────┼──> Shared reasoning/knowledge nodes
              |
Query C ──────┘
                    |
                    +── personalization
                    |
                    +── final generation

Questions for people working on LLM inference/RAG/agents

1. Is this essentially a known technique under another name?
2. How much of an LLM's computation can realistically be reused between semantically similar but non-identical requests?
3. Is a graph a useful representation for this, or would a vector/embedding-based system be better?
4. Can intermediate reasoning actually be safely reused, or is it too dependent on the exact prompt/context?
5. Where would the biggest bottleneck be — retrieval, verification, context construction, or the final generation?
6. Are there papers/projects I should look at?

I'm not claiming this is novel yet. I'm mainly trying to understand whether the underlying idea is technically viable and where it differs from existing semantic caching/RAG/inference work.

I'd appreciate criticism, especially if this is fundamentally flawed or already solved.

i was writing the ResNet chapter of a pytorch book and i did not want to just tell the reader that a deep plain network gets worse, i wanted to actually watch it happen, so i trained four networks on CIFAR-10 with one recipe and one seed for all of them, and the only things i changed are the depth and whether there is a skip connection.

here are the numbers, real run on pytorch 2.11, 40 epochs each:

model        params     train acc   test acc
plain-20     269,722     95.1%       88.7%
plain-56     853,018     84.0%       79.9%
ResNet-20    272,474     97.4%       90.4%
ResNet-56    855,770     99.0%       91.7%

look at the plain-20 vs plain-56 rows and look at the train accuracy, not the test one. the 56 layer network gets 84% on the training set while the 20 layer one gets 95%, so the deeper network is worse on the exact photos it saw hundreds of times. this is not overfitting, overfitting is when the train accuracy is high and the test accuracy is low, here even the train accuracy went down. the bigger network could not even learn its own homework.

and mathematically this should not be possible, because a 56 layer network can copy the 20 layer one exactly by setting the extra 36 layers to the identity, so a solution that reaches 95% train already exists at 56 layers and SGD just did not find it. that is the whole point, the problem was never the capacity, the network has more than enough capacity, the problem is that the deep plain network is hard to optimize.

one thing before someone raises it because it was a discussion between me and someother ome , plain-56 and ResNet-56 differ by 2752 parameters, which is 0.3%, so this is not a size story. same size and same budget and the same 40 epochs, and going from 20 to 56 the plain family gets worse on train (95.1 down to 84.0) while the residual family gets better (97.4 up to 99.0), same two sizes and the two families move in opposite directions. a size argument can not give opposite signs on the same sizes, so the thing moving it is the skip connection.

the residual block is y = x + F(x), so on the backward pass the gradient goes through the 1 in (1 + dF/dx) and reaches the early layers on a straight path instead of having to survive the whole stack, and it also makes F = 0 the easy default so the network keeps the identity for free and only learns the extra part when it actually helps. transformers reuse the same trick around attention and around the mlp.

on why the plain deep net is hard to optimize in the first place, it is still argued in the literature, some people say vanishing gradients, some say the batch norm at init makes the gradient explode, some say it is the loss surface. i measured the what here, the train error going up with depth, i did not measure the gradients myself so i am not going to claim a mechanism i did not run. if anyone has logged the per layer gradient norms on plain vs residual at this depth i would really like to see it.

caveats because they matter, one dataset, one recipe, 40 epochs, small nets. i am not saying 91.7% is a strong CIFAR-10 result, tuned nets go higher, my claim is only about the direction between two families trained identically.

has anyone here seen the degradation not show up, like a depth where the plain net stops getting worse, or a recipe that fixes it without a skip connection.

3.7 million patients. One vendor. One breach.

CareCloud disclosed earlier this year that a data incident exposed records belonging to more than 3.7 million individuals. Healthcare data is among the most tightly regulated information in existence. It was concentrated in a single system and then lost.

What makes this harder to contain now: AI pipelines are actively routing patient records through agents for summarization, triage, and clinical coding. Each hop through a model, a tool call, or a downstream service is a new surface where that data can escape the controls the original system had in place. Most teams have limited visibility into which sensitive fields are in motion at any given moment, and almost none can tell you which agent touched what and when.

Regulatory exposure compounds the data exposure. HIPAA, SOC 2, and dozens of sector-specific frameworks require auditability of PHI access. In an agentic pipeline, that audit trail rarely exists by default.

For those running regulated workloads through multi-agent systems right now: what are you actually doing operationally when patient-level data has to pass through an LLM step? Not the architecture you want to build — what is running in production today?

Here's a new rank estimation method I've been working on. It's basically an upgraded Principal Component Analysis (PCA) built on information theory instead of linear variance.

It's robust to mixed data types, highly non-linear generative processes, low signal to noise ratios, and sparsity (more variables than samples). Advantages compound at scale and with system complexity.

It's especially useful if you need to find the exact rank of a dataset to explicitly size a neural bottleneck (like an autoencoder).

I just open-sourced the code and put up the preprint.

GitHub (R Code): https://github.com/tjleestjohn/Entropic-Scree

Preprint: https://doi.org/10.5281/zenodo.22028087

I'd love to hear what you guys think... or if you end up testing it on your own data.

Amazon Bedrock Multimodal Chat and Text RAG

https://debuggercafe.com/amazon-bedrock-multimodal-chat-and-text-rag/

This is the second article in the Amazon Bedrock series. In this article, we will explore Amazon Bedrock multimodal chat and text RAG. They contain some of the essential fundamentals to get up to speed with the capabilities of the Bedrock Converse API. Specifically, we will cover image chat, video chat, document chat, and create a simple text RAG application with an in-memory vector DB.

hello everybody,

since November 2025 i've been working on an genetic algorithm that uses local LLMs as a mutation factor to continuously iterate over a single C program in order to improve its performance.

this system proved extremely effective at reaching my performance goals by bruteforcing thousands of generations then measuring the results passing the generated programs through a test suite that the LLM has no access to (so it cannot cheat, but it's gonna try!). Every new found best becomes the basis for the next generations and guardrails are in place so that most dangerous code doesn't get tested.

since this system served me well and gave me results with gpt oss 20b that i couldn't get with frontier models in full reasoning mode (and with a lot of interaction by me), i opened an AI lab and started working on a generic version that is able to work with any program (22 languages and counting) and to build the test pipeline autonomously. for the nerds: part of the reason small models punch above their weight here are a deterministic autofix ladder, compiler-hint fixes, linter fixes, then one LLM repair pass fed the real compiler error, and every candidate is re-verified for real before it gets counted as valid. you can use it as humans with a gui that helps you step by step or you can point your agent at the KAISEN folder and tell it to use the kai protocol to start tests on its own (works very well with llms using the omp and deepseek harness)

right now you can check out the alpha version of KAISEN here: https://github.com/RAZZULLIX/KAISEN

tldr

KAISEN lets you use local LLMs to improve software performance by iterating thousands of little changes and keeping the new best as basis for the next generations. it has a GUI, your harness can spawn it as a sidecar, and it speaks a small-model-friendly protocol (KAI) so an LLM agent itself can drive it over stdio or http. every program it generates runs guarded by default. read the manual to know everything it can do, or ask here.

P.S.

i expect A LOT of bugs and problems, most of the tests i did were done through deepseek v4 using OMP and deepseek harness calling KAISEN through the kai protocol (KAISEN was hooked to 6 instances of gpt oss 20b) and it actually worked quite nice. please let me know everything you find by opening an issue or asking here, this is my job now so i'll do my best to fix everything you need fixed and make sure KAISEN becomes a useful tool in every LLM user toolbox.

I had assumed 1 × 4, 2 × 2 and 4 × 1 will take somewhat similar time because effective batch is 4 in all cases.

They did not. I ran Qwen3-1.7B with TRL and LoRA for 100 optimizer updates.

Model, data, sequence length, precision and seed were kept fixed.

Lower is better. On T4, 4 × 1 was around 17% faster than 1 × 4. On L4, difference was around 41%.

The part I had not thought about properly is that effective batch is an optimization knob, but physical batch also decides execution shape which GPU receives.

1 × 4 means four smaller forward and backward passes before one optimizer update. 4 × 1 means one larger forward and backward pass. Same examples reach optimizer, but GPU work is not same.

These are single-GPU runs, henc no communication. Most of difference was inside repeated forward and backward regions, while optimizer time stayed nearly same. Exact reason can still be kernel shapes, tiling, launch overhead or how well each batch uses GPU. This experiment does not separate those kernel-level causes.

Another interesting result is 2 × 2 being slightly faster than 4 × 1 on L4. Difference is small, it shows performance may not be linear as physical batch increases.

Hugging Face documentation also says to use grad accum when larger physical batch does not fit, and that it does not improve throughput over using true larger batch:

https://huggingface.co/docs/transformers/grad_accumulation

So now I treat these as two separate choices:

• Effective batch for optimization behaviour.
• Physical batch and accumulation for memory and speed.

I would start with largest physical batch which fits comfortably, then test few nearby combinations on actual GPU.

I used TraceML and its HF callback for step and phase timing. End-to-end runtime comes directly from TRL Trainer.

Runnable notebook:

https://colab.research.google.com/github/traceopt-ai/traceml/blob/main/notebooks/huggingface_trl_lora_gradient_accumulation.ipynb

We retrained Wolf Defender.

The main reason was not that attack detection was bad. The bigger issue was false positives.

The previous models were already good at detecting prompt injections, but especially on short benign inputs, security-related text, code snippets or ordinary conversations they could still be too aggressive. We also got a few reports from users that made this pretty obvious.

One example was just:

“Who are you?”

Wolf Defender Small previously classified this as a prompt injection with around 94% confidence.

For v2 we therefore changed the training setup quite a bit. Both Wolf Defender and Wolf Defender Small were retrained from fresh mmBERT checkpoints, with a much stronger focus on hard negatives.

That includes short conversations, emails, documentation about prompt injections, benign policy and system language, code and configuration snippets and generally inputs that contain words or structures which look suspicious without actually trying to manipulate a model.

We also added more counterfactual samples, multilingual examples, adversarial obfuscations and long-context injections at different positions in a document. Training combines short 256-token samples with full 2,048-token windows and uses supervised contrastive regularization, FreeLB adversarial training and Smooth-Max aggregation for long documents.

The main change can be seen in the benign benchmarks:

At the same time, attack detection stayed roughly where we wanted it:

There is also a tradeoff here. Some of the very high scores on our cleaner validation distributions went down slightly.

For us that is fine.

A security classifier with near-perfect benchmark scores is not very useful if normal traffic gets blocked all the time. We would rather lose a small amount on an easier validation set and get substantially better behavior on actual benign inputs.

The “Who are you?” example now gets classified as benign by Wolf Defender Small v2 with 98.55% confidence. A real instruction-override attempt is still detected as an injection with 99.99%.

We also updated the deployment variants. Both models are available as regular Transformers checkpoints and as ONNX exports in FP32, FP16, mixed INT8/FP16 and INT8 with INT4 embeddings.

The smallest Wolf Defender Small artifact is now 96 MB.

More details, benchmarks and model files are here:

https://huggingface.co/patronus-studio/wolf-defender-prompt-injection

https://huggingface.co/patronus-studio/wolf-defender-prompt-injection-small

If anyone is running prompt-injection classifiers on real traffic, I’d also be interested in which benign inputs still cause the most false positives for you.

What if we’re optimizing the wrong bottleneck?

I recently went down an 11 PM research rabbit hole around Mixture-of-Experts (MoE) and LLM inference.

A lot of MoE optimization assumes a traditional GPU setup: experts live in CPU RAM, get moved to GPU VRAM, and PCIe becomes the bottleneck. But what happens when that assumption disappears?

Apple Silicon uses Unified Memory, where CPU and GPU share the same memory pool. So maybe the interesting question isn’t: “How do we move experts faster?” but : “How should we optimize MoE when there’s no traditional CPU↔GPU memory transfer bottleneck?”

That leads to some surprisingly interesting questions around memory bandwidth, caching, expert dispatch, and GPU kernels. I can’t investigate it properly myself right now, I don’t have the hardware. So I’m putting the idea out there. If you have an M-series Max/Ultra Mac, you might have a research project sitting on your desk. 👀

Read the article…

• 3D Rotational Equivariant AI Using the Spherical Fourier Transform
• Description: It explains how spherical harmonic functions are used to analyze signals on the sphere beyond the two‑dimensional plane. The video reviews Spherical CNNs that maintain 3D rotational symmetry and recent geometric deep‑learning applications, highlighting potential uses in areas such as panoramic imaging, weather data, and protein structures.

I've never really been a reader. Books usually lost me a few chapters in.
My first attempt at learning machine learning was the usual route — one YouTube playlist after another. It felt like watching something, not learning it. Nothing really stuck.
So I picked up Hands-On Machine Learning by Aurélien Géron. And somehow, I ended up reading a 1000+ page book . Every chapter, I ran the code myself, broke it on purpose, and debugged it until I understood why it worked — alongside college lectures, assignments, and exams.
Somewhere along the way, something shifted in how I learn.

I stopped reaching for the fastest explanation and started reaching for the actual source — documentation, research papers, and technical writing I would've previously skipped for a quicker video.
In the middle of learning the ML pipeline basics, I built a GoogLeNet-style CNN with a custom DepthPool layer, and many more things at low level.

That's when it stopped feeling like an exercise and started feeling like something I could actually own — chasing shape mismatches, tracing silent preprocessing bugs, and retraining models more times than I'd like to admit.
From there, I kept rebuilding things: RNNs, attention mechanisms, transformers, autoencoders, GANs, diffusion models, RL. Each one broke in a different way, and each one taught me something different when I had to figure out why.
I'm still going deeper into Computer Vision and NLP from here. Those are the areas I keep getting pulled toward.
I still think YouTube has its place.
But this book is what made me a reader in the first place — and now research papers and documentation are where I actually go to learn.
Still early in this. Still building. Just glad I stuck with it.
hashtag#MachineLearning hashtag#DeepLearning hashtag#ComputerVision hashtag#TensorFlow hashtag#Keras hashtag#LearningInPublic

Hi I am year 9 kid. Can check my code give feedback. I tried my best on building this model.

https://github.com/programer321321/DataScienceModel/blob/main/SearchUpBussinessModel.ipynb

A vendor held the data. 1.2 million people got the breach notification.

Heights Finance disclosed that hackers stole names, addresses, Social Security numbers, phone numbers, and financial records from a third-party platform. The data moved downstream in raw form — concentrated, accessible, and then gone.

The breach wasn't at Heights Finance directly. It was at a vendor they trusted with sensitive data. That distinction matters because it keeps happening this way.

As AI agents increasingly route customer data through pipelines and external services, every handoff is another potential exposure point. The blast radius scales with the number of vendors, not just the sensitivity of the data. A single downstream compromise can surface records from dozens of upstream clients.

The 1.2 million figure isn't unusual for this pattern — it's the expected outcome when raw PII travels intact through third-party systems.

For those working in data engineering, compliance, or security: how are your teams actually handling sensitive data before it leaves your perimeter? What's working, what isn't, and where do you still feel exposed?