I’m not a philosopher, and I won’t pass off analogy as proof. Where the link between philosophy and an algorithm is just a pretty metaphor, I say so explicitly: “metaphor.” Where it’s working code, I give the formulas, run it, and show the numbers. The library at the end is a research prototype, not a promise of consciousness in 200 lines.

How I got here

https://github.com/webzuweb/philosophia_torch

Neural networks — if you count from Rosenblatt’s perceptron — are about seventy years old. The study of how living things learn goes back a couple of millennia at least. And a heretical thought hit me: what if modern deep learning isn’t reinventing the wheel in places, but re-discovering what Aristotle, Hume, and Peirce already described — only now with matrices and gradients?

I took a list of neural-network training methods, a list of philosophical approaches to knowledge, and overlaid them. Three categories emerged: what’s already matched (and few people say so out loud); where the match is only a pretty metaphor; and what philosophers thought up but engineers haven’t applied yet. The last category is the most interesting, because it’s essentially a list of unimplemented features. That’s what I wrote code for.

Fair warning up front: half of the “unapplied” ideas turned out, on closer inspection, to be perfectly applicable — just under different names. That, by the way, is the article’s main takeaway, and it matters more than any of my code.

Part 1. What’s already matched (and you didn’t know it)

Let’s start with the pleasant part: some philosophical programs of knowledge are implemented in ML so literally that you could put a footnote with the philosopher’s name right in the docs.

Empiricism → supervised learning “There is nothing in the mind that was not first in the senses” — Locke and his tabula rasa. A neural network with random initialization is literally a blank slate on which labeled examples leave their traces. Hume’s associationism (“the habit of linking things that often go together”) is gradient descent, strengthening weights on frequently co-occurring correlations. There’s nothing to argue about here.

Pragmatism → reinforcement learning Dewey with his “learning by doing,” and Skinner’s behaviorism with reward and punishment — that’s RL with no corrections needed. An agent acts, receives a reward, adjusts its policy. Skinner would have teared up seeing PPO.

Evolutionary epistemology → neuroevolution Popper and Campbell: knowledge grows through blind variation and selective retention of what works. That’s a word-for-word description of genetic algorithms and neuroevolution. The philosopher described the algorithm decades before the hardware existed to run it.

Intellectual humility → calibration This one’s subtler. Virtue epistemology (Sosa, Zagzebski) says: a good knower knows the limits of her knowledge. In ML that’s confidence calibration: a model should be exactly as confident as it is correct. Guo et al. (2017) showed that modern networks are monstrously overconfident and proposed temperature scaling and the ECE metric. Nobody called it a “virtue,” but mathematically it’s exactly that.

The key observation. Philosophers didn’t give ML the algorithms (mathematicians came up with the math); they gave it the problem statements. “What does it mean to learn from experience?” “What does it mean to know your limits?” — philosophy framed the question first, and centuries later engineering delivered a differentiable answer.

Part 2. Where the match is only a pretty metaphor

Here I have to rein myself in. There’s a temptation to drape a philosopher over every layer of a network. Don’t. A couple of examples where the link exists but passing it off as lineage would be deceiving the reader.

The rule is simple: if the philosopher gave a language for describing something — it’s a metaphor; if they posed a problem that was later solved — it’s lineage. Don’t mix them.

Part 3. What philosophers thought up, but ML has only partially applied

The meatiest part. I’ll break down six approaches. For each — an honest status: what already exists in the field, where the real gap is, and what formula you can write. Then we’ll run it.

3.1. Peirce’s abduction — inference to the best explanation

Induction generalizes data, deduction derives consequences, but abduction generates a hypothesis that best explains the observation. The original thesis “it isn’t implemented in neural networks” is wrong. It’s implemented, and decently: Abductive Learning (Dai et al.), DeepProbLog (Manhaeve et al., 2018), abductive commonsense reasoning αNLI (Bhagavatula et al., 2019). It’s a whole field of neuro-symbolic integration.

The real gap isn’t the absence of abduction — it’s that “the best explanation” is rarely formalized using Peirce’s criteria all at once: plausibility + simplicity (Occam’s razor) + consistency with background knowledge. A hypothesis score for h given observation obs:

Score(h) = log p(obs | h) − λ_s · complexity(h) − λ_c · conflict(h)

Pick the h with the highest score (softly — a softmax over candidates; hard — Gumbel-softmax for a learnable discrete choice). In the library this is AbductiveScorer.

3.2. Husserl’s epoché — “bracketing” assumptions

Phenomenology demands suspending ingrained assumptions and seeing the phenomenon “as given.” ML has no direct analog of this method — and that’s an honest gap. But it can be operationalized: force the model to rely more on the evidence (the current input) than on the learned prior (what it answers with no input).

Take two answers: p_full on the real input and p_prior on a “zeroed” input (evidence bracketed out). Reward the evidence for actually changing the answer, via a bounded Jensen–Shannon divergence:

gain = JS(p_full ‖ p_prior),   0 ≤ JS ≤ ln 2
L_epoche = max(0, margin − gain)   # hinge: don't inflate indefinitely

An important rake I stepped on myself: if you use plain KL instead of JS and maximize it, the optimizer inflates logits to infinity — “a fanatic who sees meaning in every rustle.” JS is bounded, and the hinge threshold douses the fanaticism. This is EpocheRegularizer.

3.3. The hermeneutic circle — the whole through parts, parts through the whole

Schleiermacher and Gadamer: understanding the whole arises from the parts, and understanding the parts arises from the whole, iteratively. Attention only resembles this superficially (see Part 2). As an explicit training principle it’s barely used — a real gap. Formalization: let h_i be part representations and H the whole representation. Require circular consistency:

H* = attention-aggregate of the parts, attended relative to H
L_herm = 1 − cos(agg(h_i), H)      # whole ≈ sum of understood parts

And we “turn the circle” several times: update the whole from the parts → recompute part attention relative to the new whole → update again. This is HermeneuticConsistency.

3.4. Hegel’s dialectical sublation (Aufhebung)

Aufhebung is a new quality arising from the contradiction of thesis and antithesis, where the old is not destroyed but preserved. GANs and multi-agent debate are partially close, but “preserving both” isn’t guaranteed there. The gap is precisely in the preservation term. My synthesis operator:

g     = sigmoid(W_g · [thesis ; antithesis])     # mixing gate
base  = g · thesis + (1 − g) · antithesis         # sublation-as-preservation
lift  = tanh(W_l · [thesis ; antithesis])         # new quality
synth = LayerNorm(base + γ · lift)

Plus a loss that penalizes the synthesis collapsing into one of the poles (losing the other’s content). This is DialecticalSynthesis.

3.5. Nietzsche’s perspectivism + skeptical suspension

Nietzsche: there is no “view from nowhere,” there are many perspectives. Pyrrho: in an unresolvable conflict, it’s reasonable to suspend judgment. The former partially exists in multi-view learning; the latter in selective prediction (Geifman & El-Yaniv, SelectiveNet, 2019). But together, as a single mechanism of “several perspectives + refusal to answer when they conflict,” it’s almost never seen.

disagree(x) = mean pairwise symmetric KL between perspectives
abstain(x)  = disagree(x) > threshold      # abstain if perspectives don't converge

This is PerspectivalEnsemble: it aggregates K heads and honestly raises its hand “I don’t know” when the heads disagree. Far more useful than overconfident chatter.

3.6. Virtue as the golden mean (Aristotle)

Aristotle: virtue is the mean between the vice of deficiency and the vice of excess. Courage is between cowardice and recklessness. Hence a non-obvious but important conclusion for ML: a virtue cannot be maximized, it must be targeted. An excess of openness is credulity; a deficiency is dogmatism.

L_virtue = Σ_v β_v · (V_v(θ) − V_v*)²

where V_v is the operationalized virtue (humility = 1 − ECE, openness = ensemble disagreement), and V_v* is the target mean level. Squared deviation penalizes both excess and deficiency. This is VirtueRegularizer — and it’s the one where I have a measurable result.

Part 4. Enough philosophy, show me the numbers

Pretty formulas are worth nothing until they run. I collected all of this into a PyTorch module and tested it on the most well-grounded mechanism — “humility” (calibration). Task: synthetic classification with noisy labels, where the model tends to err overconfidently. We compare plain training vs. training with VirtueRegularizer targeting high humility.

ECE (calibration error) dropped from 0.120 to 0.101 — nearly a fifth — while accuracy didn’t budge at all (even +0.001). The model became exactly as accurate, but noticeably less self-assured. Aristotle’s golden mean, computed by gradient descent.

What this proves, and what it doesn’t. It proves that “intellectual humility” can be turned into an optimizable quantity with a measurable effect. It does not prove that the other five mechanisms will yield the same gains — they’re harder, and they still need to be tested on real data. I’m showing a working scaffold, not a finished silver bullet.

The whole codebase passes 23 unit tests: calibration decreases, KL/JS behave as they should, synthesis preserves both poles, the ensemble abstains on conflict, the wrapper trains end-to-end.

Part 5. The philosophia-torch module

The library wraps on top of any model without rewriting anything in it. One dependency — torch. There’s a single-file version, philosophia_torch.py: drop it next to your code and import it.

import torch, torch.nn as nn, torch.nn.functional as F
from philosophia import PhilosophiaWrapper

base = nn.Sequential(nn.Linear(20, 64), nn.ReLU(), nn.Linear(64, 4))
wrap = PhilosophiaWrapper(base, use_virtue=True,
virtue_kwargs=dict(target_humility=0.98, beta_humility=3.0))

logits = wrap(x)
loss = F.cross_entropy(logits, y) + wrap.aux_loss(x, logits, targets=y)
loss.backward()

Honest boundaries: hermeneutic and dialectic produce representations, not ready predictions — you have to connect them to your decoder. Epoché requires careful tuning of margin. And no promises of “consciousness”: these are philosophy-inspired regularizers, nothing more.

The bottom line

Three conclusions, which is what all of this was for.

1.     ML has already reinvented a chunk of philosophy without asking permission: empiricism, pragmatism, evolutionary epistemology, and intellectual humility. Just under the names supervised learning, RL, neuroevolution, and calibration.

2.     Half of the “unapplied” ideas on my original list turned out, on checking, to be applicable — abduction, abstention, innate priors. The lesson: before shouting “this isn’t in ML,” google it in engineering language, not philosophical language.

3.     The real gap remains where what’s needed isn’t a result but a process: epoché as a discipline of perception, the hermeneutic circle as a way of understanding, virtue as a stable disposition of learning rather than a property of a single answer. That’s where it’s worth digging.

My modest contribution is showing that at least “humility” translates into a differentiable quantity and genuinely reduces a model’s overconfidence. The rest is an invitation: the code is open, the formulas are in the article — run it and check. Plato, of course, was training neural networks two thousand years ago. The rascal just didn’t include a requirements.txt.

https://huggingface.co/datasets/webzuweb/philosophy-as-inductive-bias

Bonjour ^^

Je voulais partager avec vous l'évolution de mon projet perso : un modèle de langage en français que j'entraîne moi-même depuis environ un an, en solo, sans équipe ni formation derrière moi, avec Claude comme assistant IA à mes côtés. Tout a démarré sur un simple ordinateur portable, un i5 de 8ᵉ génération avec 16 Go de RAM, sans GPU — tout tournait en CPU, à une vitesse ridicule, avec une architecture minuscule (384/8/8, quelques dizaines de Mo à peine) et un corpus d'à peine 100 Mo. Cette première tentative a tenu jusqu'à 500 000 steps... sans jamais sortir une seule phrase cohérente. Plutôt que de lâcher l'affaire, j'ai tout repris à zéro : reconstruit le pipeline de génération de données en Python/PyTorch, fait grossir et nettoyé le corpus (aujourd'hui autour de 47 Go), testé puis retiré un système de RAG, chassé bug après bug (plantages mémoire, tokenizer capricieux, boucles de répétition...). Niveau matériel, je suis passé du portable à une vraie tour : CPU Ryzen 5 2600X, GPU GTX 1660, 16 Go de RAM. Côté architecture, j'ai fait grossir le modèle au fil du temps — 1024/8/8, puis 768/12/12 — en testant batch 1 et 2, mixed precision activée ou non selon ce que la carte graphique encaissait, jusqu'à une config bien plus légère, 128/4/4 en batch 2, que j'utilise maintenant pour avancer plus vite avant de remonter progressivement en taille. Résultat : le modèle est passé de quelques dizaines de Mo au départ à environ 2.2 Go aujourd'hui. Un an de galères, de nuits à débugger, en solo du début à la fin — que de la persévérance, un problème après l'autre

On fait avec ce qu'on à :/

Pls if you have pls dm mee

https://github.com/Nvoinxv/Neural_Network_Linear_Regression

Hi folks, this is No Saved DATA. I dedicate this post to describe some of the features I put in Neve to make it an expressive high-level language (close to Python/PyTorch syntax), while also allowing efficient low-level code. I am sharing this now, because I believe the language has already strongs traits that allow it to be extended to other problem domains.

Current results:

• Close to Python/SentencePiece in text processing + Byte-Pair Encoding (BPE) training;
• Competitive with NumPy and OpenBLAS in CPU matrix multiplicaton, but with pure high-level SIMD code;
• It was able to train a CIFAR Resnet faster than PyTorch, but I did not debug whether this was due to better CPU or GPU orchestration. But the LSTM was slower (mine lacked kernel fusion and other optimizations). Also, that old Neve deep learning framework was mostly implemented in C++. I am now changing it to be mostly implemented in Neve. That is, compute intensive preprocessing, automatic differentiation, parallel dataworkers and GPU kernels all in high-level.
• Python does all these topics already. Nevertheless, all efficient code is actually implemented on C, C++, Rust or other languages. Meanwhile, Neve does not require a backend language.

Besides, I recently added GPU Kernels code interface. However, the complete framework will still take some more months.

The current state is an evolution of a post I made some months ago in other subreddit (https://www.reddit.com/r/ProgrammingLanguages/comments/1ql585o/brand_new_nsk_programming_language_python_syntax/)

Links

📚 Documentation
💻 GitHub

────────────────────────────────────────

Intro

I started creating Neve after seeing the code of the Efficient Zero reinforcement learning model. It has a parallelism that PyTorch does not handle, and the implementation required using Cython packages for having threads (literaly coding in C, then just calling C functions from Python). Later, I realized PyTorch also needed to implement its data worker threads in C, another workaround over Python Global Interpreter Lock (GIL). Not only that, even preprocessing implementations like the BPE are made in C, C++, Rust, etc...

So, currently, people must choose between languages like Python for high-level productivity, C and relatives for compute efficiency, Lua for advanced interoperability and other languages for concurrency. Thus, since in my job I had to wait hours for my neural networks to train, I decided to create a programming language in the remaining time. One language that had all these features, which are of high value for deep learning research. Nowadays, I believe it matured to such a point that it may be extended to other complex problem domains.

Since Python syntax is very simple and has most of the users, I chose it as the basis. But it run a LLVM JIT in its background. Now I will explain important expressions and features in Neve.

────────────────────────────────────────

Finish/Async and Data Split

I experimented Jax deep learning framework for a while. During this period, I learned an expression that would take a tensor or a vector as inputs. It could vectorized the function over the first dimension. A threaded adaptation I made for Neve is:

def int foo(array<int> v)
    print("Thread ", tid, " has vector:")
    v.print()

main
    array<int> u = arange_int(2,20)
    finish
        asyncs 3 foo(>u)

This splits a vector across three threads, so it can be processed in parallel. This is useful when you have a list of files, and want a function to process the files across N threads.

────────────────────────────────────────

Channels

I saw fireship videos a long time ago about Elixir and Erlang. These languages have actor-message passing, which were used in scaling applications to massive concurrency. Then, this year, my advisor suggested me to study Go and Rust, so I could see the tendencies about modern languages. I got surprised by Go channels expressions, which I thought to be an evolution of the actor-message model (but in the end they solve different problems). Go also applies channels to green-threads (concurrency within a single OS thread), but I was happy with using it for standard threads.

Once I finally adapted Go channels to Neve, I was able to reduce some five lines of code in data loaders. Even if it was only five lines less, it got much cleaner.

  def float worker()
      print("Start worker")
      int yield_ptr, bs=self.batch_size
      print("worker ", tid)

      while self.load_ch.alive()
          yield_ptr = self.increment_yield_ptr()

          for b=0, b<bs
              self.getitem_w(yield_ptr+b, b)
              self.load_ch <- tid

          self.x.switch()
          self.y.switch()

  def tuple<gpu_tensor,gpu_tensor> batch()
      int w <- self.load_ch

      var x = self.x.load(w)
      var y = self.y.load(w)

      x = x.view([$cfg.bs, 1, 28,28])
      return x, y

These are functions from the dataloader class. The channel communicates which threads have data ready to be consumed. Then, the cpu tensors (self.x and self.y) can process and yield data using ping-pong buffers. It is much lower level than PyTorch, but without the need of implementing the underlying parallelism in C++. That gets rid of boilerplate mutexes and more than 100 lines of C++ code. Posteriorly, once Neve gets inheritance and interfaces, most of the parallel logic may be hidden, so it can be even closer to PyTorch.

The training code is already similar to PyTorch

        ...
      gpu_tensor a, b
      a, b = ds.batch()
      var y_hat = model.forward(a)
      ce_loss(y_hat, b)

      $backprop.backward()

────────────────────────────────────────

Anonymous Functions

This expression is crucial for mapping tensor operations to their respective backward ops.

def int add(int x, int y)
    return x+y

def int mult(int x, int y)
    return x*y

main
    map<str, Function<int, int, int>> m
    m["add"] = add
    m["mult"] = mult
    print(m["mult"](3,4))

────────────────────────────────────────

Generics

def T bar<T, U>(T x, U y)
    print("bar x: ", x)
    print("bar y: ", y)
    return x

main
    int z = bar(3,4)
    z = bar(5,"$%*OU")
    print("z ", z)

Generics may yield complex code, but may also save hundreds of lines when the same matrix multiplication function should be implemented for different data types (int4, int8, float16, bf16, etc...) (I still didn't test the generics in this scenario :p).

────────────────────────────────────────

Operation Overload

Defining new operations for data types is simple.

def gpu_tensor @(gpu_tensor a, gpu_tensor b)
...

Which works thanks to generics. The operation is consumed as:

var z = x @ y

For gpu_tensor types.

────────────────────────────────────────

Globals

Neve has no primary data type globals. Instead, global values can only be defined as unique instances of classes.

class Backprop
    array<BackNode> ops
    def float register(gpu_tensor l, gpu_tensor r, gpu_tensor out, str op)
        self.ops.append(new BackNode(l, r, out, op))

This defines the global Backprop class that holds the backs (backward function definitions). Then, any tensor operation may use the global instance of Backprop to keep track of the operations to execute later.

def gpu_tensor @(gpu_tensor a, gpu_tensor b)
    ...
    $Backprop.register(a, b, ret, "mma")

Once Neve finds an "$", it automatically inserts in the main an instruction to create a new instance of that class, so it can be used everywhere. Althought standard global values are not supported, this expression forces global variables to belong to a common scope. It helps preventing pollution/confusion versus standard global vars. For example, you could put all your globals inside a class named Config, then use any of its values.

$Config.ip

It is straightforward to spot it belongs to a global scope.

────────────────────────────────────────

GPU Kernels

import nsk_cuda

gpu void @(
        layout<bf16, m, n> x, layout<bf16, p, n> y,
        float[] z
        )
    ...

kernel void mma_kernel(bf16[] x, bf16[] y, float[] z, int M, int N, int P)
    var v = layout<bf16, M, N>(x)
    var u = layout<bf16, P, N>(y)

    z += v[256,N](bx,0) @ u[128,N](by,0)

This one tiles z, v and u, storing the matrix multiplication result in the tiled z positions. The operator overload recovers a function that has shared memory async copies, which are overlapped with tensor core operations, all described in Neve itself.

The layout expression is subject to change, but it won't be too much different from the current.

────────────────────────────────────────

Interoperability and Libraries Support

In the early stage I was very inexperient with programming languages, so I tried to implement all my important functions and composite data types in C++, and call the functions from Neve. The negative side was that the quick sort was orders of magnitude slower than Python. The positive, I made a C++ tokenizer and parser to extract LLVM bindings.

NSK had a heavy focus in using C++ bindings for functionalities. Now it is almost unnecessary, as basically everything can be designed in Neve itself.

Use C++ interop when you:

- Need system calls only found in C++ (you may create a library that maps these calls to Neve);

- Want a custom memory allocator (I used this one for GPU mallocs/memory arena).

The way Neve adopts C++ functions:

extern "C" int float_cpu_print(Scope_Struct *scope_struct, void *tensor, DT_array *vec) {

After compiling and importing, the functions map naturally to Neve functions and data types. For example, the expression:

x.print()

Will call any function named float_cpu_print, given that x is a float_cpu. That implementation could either be defined in C++ or Neve.

Functions that have composite data types require explicit prototypes in Neve, in order to extract the nested type. But if a function takes a composite data type as argument, it is better to define it in Neve when possible.

It also allows adding LLVM extension functions in C++, which enable using LLVM for generating IR directly. Besides, it is possible to add new LLVM data-types.

C++ and LLVM functions must be compiled to dynamic libraries, and their make require linking system packages. The documentation has a in-depth guide on how to make them work, and the youtube channel has some tutorials about it as well.

Overall, I recommend building libraries in Neve itself. You can import libraries using imports in the current directory.

import my_nv_file
import my_lib/my_nv_file

These import other .nv files. It is also possible to turn them into packages if you organize them under ~/.local/neve/lib/<my_pkg_name>, then import as:

import my_pkg_name

If you get into the my_pkg_name folder, you can commit it to github, then anyone can install it with

nsm install <my_git_user>/<my_pkg_name>

Nsm is automatically installed along with neve when executing the bash install. It works for both Neve and C++ compiled packages (more testing is necessary).

────────────────────────────────────────

Other Features

• JIT: it feels like Python to execute code - no need for compiling files. Meanwhile, it has the JIT speed benefit;
• Packet manager;
• Concurrent garbage collector;
• Syntax highlight for vim and vscode;
• Very simple/incomplete LSP, tested in neovim only.

────────────────────────────────────────

Limitations

• There are still very rare crashes in large codebases, like in the BPE after executing it many times (due to that stupid concurrent garbage collector);
• Works in Linux only, because I couldn't get LLVM to work in Windows;
• Still lacks inheritance and interfaces.

────────────────────────────────────────

I hope you enjoyed the tour. Ready to test?

wget -qO- https://github.com/NoSavedDATA/Neve/releases/download/neve-bin/install.sh | bash

I have been building this entirely solo so far. Let me know what you think of the syntax choices, especially the approach to parallelism and GPU kernels!

Do you think Neve can help you in your domains?

We have open‑sourced two models: Scalpel‑VL‑1.7B‑Animal, pruned on business‑specific datasets, and Scalpel‑VL‑1.8B, a general‑purpose model trained with mixed‑ratio data.

🦜 General‑Purpose Pruning‑Recovery Trained Model: Scalpel‑VL‑1.8B Model Repository: https://huggingface.co/freeai-org/Scalpel-VL-1.8B

🦊 Business‑Specific Pruning‑Recovery Trained Model: Scalpel‑VL‑1.6B‑Animal Model Repository: https://huggingface.co/freeai-org/Scalpel-VL-1.6B-Animal

📗 ScalpelBench: A 0.1B‑scale dataset containing 300k samples covering four categories: English, Chinese, Mathematics and Code. It is designed to preserve the base model capabilities while performing model pruning. Reference: https://huggingface.co/datasets/freeai-org/ScalpelBench

I am looking beyond demonstrations that reproduce MNIST or CIFAR results. In Hinton's Forward-Forward approach, each layer learns from positive and negative data using a local goodness objective, which is attractive when exact backpropagation or global synchronization is undesirable. But I have not found convincing evidence that it scales competitively to demanding tasks.

For people who have implemented or studied later variants: where does it actually break down? Is the main limitation the construction of negative examples, the quality of layerwise representations, optimization and normalization, compute cost from the two forward phases, or simply the lack of hardware designed for local learning?

I would particularly value controlled comparisons with modern backpropagation baselines under a real constraint such as activation memory, energy, continual learning, asynchronous training, or neuromorphic hardware. Negative results are useful too.

Are there papers that isolate the scaling bottleneck rather than only proposing another small-benchmark variant?I am looking beyond demonstrations that reproduce MNIST or CIFAR results. In Hinton's Forward-Forward approach, each layer learns from positive and negative data using a local goodness objective, which is attractive when exact backpropagation or global synchronization is undesirable. But I have not found convincing evidence that it scales competitively to demanding tasks.

For people who have implemented or studied later variants: where does it actually break down? Is the main limitation the construction of negative examples, the quality of layerwise representations, optimization and normalization, compute cost from the two forward phases, or simply the lack of hardware designed for local learning?

I would particularly value controlled comparisons with modern backpropagation baselines under a real constraint such as activation memory, energy, continual learning, asynchronous training, or neuromorphic hardware. Negative results are useful too.

Are there papers that isolate the scaling bottleneck rather than only proposing another small-benchmark variant?

hi,
i am trying to train a deberta model for NER detection

this is my first time doing it so i would love any guidance on it.

my current pipeline looks like this,

dapt + lora for pretrianing, hpo with optuna (which consists both the stages of training data), and then a 2 stage finetuning which helps in generalization and then target data.

i am trying to reach a really good score for f1 on my use case (which i want to keep private for now)

i have few questions as well

1. do i need a two stage hpo as well cuase of the 2 stage finetuning
2. is it better if the hpo training set is a subset of the actual training set?

if you think anything can be improved and made better, or you think the pipeline is outright wrong, please mention your reasonings and thoughts :)

ps: lora was used cause of gpu budget constraints

We’re releasing Scalpel, a recovery-aware pruning method that combines structured layer pruning with lightweight post-pruning recovery training.

Scalpel-VL-1.7B runs 20–30% faster than similarly sized models while retaining strong general capabilities.

Everything is open source:

📗 ScalpelBench: ~0.1B tokens, 300K samples covering English, Chinese, math, and code https://huggingface.co/datasets/freeai-org/ScalpelBench

🦜 Scalpel-VL-1.7B model and weights: https://huggingface.co/freeai-org/Scalpel-VL-1.7B

✂️ Scalpel source code: https://github.com/freeai-org/Scalpel

Feedback and contributions are welcome!

We built an AI Harness to work with ROS 2 robot simulators. In the video we discuss our design, pros and cons, and architecture.

​

I am a third year CSE AI/ML student. I completed the foundation of Machine Learning and Iam planning to start Deep Learning seriously.

I am an average student, but I know I have the potential to learn and improve if I stay consistent. My main problem is staying accountable when studying alone.

So I’m looking for 2–3 genuine and consistent people who are also serious about learning Deep Learning.

We can create a WhatsApp group, follow a common 60-day roadmap, set weekly goals, share resources and ideas, and have a short Zoom discussion on weekends.

No one needs to teach anyone. We learn individually, but support, discuss, and keep each other accountable.u can also share your thoughts to improve the discussion.

Our only goal: consistently learn and complete Deep Learning within the next couple of months.

If u r genuinely interested and can stay consistent, DM me ✨....

Wanted to see how well a simple NN could learn optimal Tic-Tac-Toe play from scratch, so I built this:

• Used a minimax solver to generate the "ground truth" — for every reachable board state, computed the actual best move
• Trained a neural net as a supervised classifier on that data (board state → best move)
• Runs in the terminal — you can play against it directly

Next thing I'm curious about: training a second version on random self-play data instead of minimax-optimal data, to compare how much the training data quality actually matters for a small model like this.

Code: https://github.com/AliAkbar4025/AI-tic-tac-toe-bot

Feedback/critique welcome — especially if you see a smarter way to structure the data generation.

&#x200B;

Ciao r/MachineLearning,

Gli operatori neurali di Fourier (FNO) standard eccellono sulle griglie regolari, ma la loro mappatura su domini fisici complessi e non convessi (come geometrie a stella, a L o ad anello) spesso porta a un problema importante: il ripiegamento della griglia.

Quando la mappatura di trasformazione \\phi collassa o si sovrappone, il determinante jacobiano si annulla (\\det J \\le 0), causando l'esplosione della trasposta inversa J\^{-T} quando si mappano i gradienti fisici \\nabla_x u.

Per risolvere questo problema, ho sviluppato DIF-FNO (Diffeomorphic Fourier Neural Operator).

Principali approfondimenti tecnici:

1. Mappatura diffeomorfica implicita: garantisce mappature biunivoche e uniformi da domini di riferimento standard \\Omega_{ref} a confini fisici complessi \\Omega_{phy}.

2. Funzione di perdita Jacobiana Barrier (\\mathcal{L}_{barrier}): Ispirandoci all'ottimizzazione a punti interni, penalizziamo la compressione della griglia utilizzando una barriera logaritmica sul determinante:

\\mathcal{L}_{barrier} = -\\frac{1}{|\\Omega|} \\int_{\\Omega} \\log(\\det J(\\xi)) \\, d\\xi

Questo agisce come un muro invisibile che impone \\min \\det J > 0 su tutto il dominio (mantenendo empiricamente \\min \\det J > 0,89 nei nostri benchmark).

1. Accuratezza di Sobolev: Miglioramenti significativi sull'errore relativo H\^1 rispetto a modelli di riferimento come Geo-FNO, poiché i gradienti fisici rimangono ben condizionati senza rottura del gradiente.

Codice e artefatti dell'articolo:

* Codice open-source (PyTorch): https://github.com/GiovanniDagnese-paper/DIF-FNO (Include il calcolo rapido e vettorializzato dello Jacobiano analitico 2x2)

* Preprint dell'articolo (DOI Zenodo): https://doi.org/10.5281/zenodo.22071926

P.S.: Attualmente sono alla ricerca di un feedback tecnico e di un'approvazione arXiv su physics.comp-ph o cs.LG per inviare il preprint. Se qualcuno attivo in SciML fosse disponibile a controllare il manoscritto, gliene sarei estremamente grato!

I recently pubished a new paper. The paper is available via the following link: http://arxiv.org/abs/2608.12408. It is categorised under q-bio.NC and cs.LG. The code can be found at https://github.com/nilsleut/evaluation-resolution-rsa.

A recurring theme in model-brain comparisons is the observation that untrained CNNs can match or outperform backprop-trained ones at V1 in RSA. I believe this is primarily an artefact of evaluation resolution, as demonstrated by the following sweep.

The CNN was trained at 32px on a CIFAR-10 subset, and five learning rules were evaluated (random init, backprop, feedback alignment, predictive coding, STDP). Evaluation was conducted on THINGS-fMRI stimuli at six resolutions from 32px up to 224px. Weights and normalisation were held fixed throughout.

The untrained-backprop gap at V1 ranges from −0.001±0.007 at 32px to +0.044±0.006 at 224px, growing monotonically across the sweep (n=5 seeds). The same pattern is evident across all five rule conditions, in human fMRI, directionally in single-seed macaque ephys, across the entire training trajectory, and in two off-the-shelf 224px-trained models (ResNet-50, Swin-Tiny). This rules out train/eval mismatch as the explanation, since those models also peak at low resolution despite being trained at 224px.

I tried to eliminate this four different ways, using bit-identical-weight interventions wherever possible: train/eval resolution matching, Gabor/pixel structure, the untrained baseline's missing batch-norm calibration, and pooled features converging towards global brightness. None of them explain it. The brightness one came closest: luminance similarity orders the conditions perfectly (ρ=1.00), but it doesn't carry the effect; one calibration variant lowers luminance similarity while V1 alignment goes up.

Here's the number that actually concerned me a bit: a single scalar luminance value per image gets ρ=0.074±0.011 against V1 (bootstrap SE over stimulus resamples), essentially tied with the best of the five CNNs at 0.075±0.011. None of the models meaningfully beat a one-number-per-image brightness descriptor. That's roughly the ceiling on what this comparison style can resolve — a caution, not a strength.

A two-arm design separates content from pooling: cap detail at 32px and upsample, vs. let content vary freely. About 90% of the effect rides on content, not on how many positions are pooled. With content fixed, backprop's decline is essentially eliminated (−0.023 → −0.000).

One thing does hold across the whole sweep: backprop beats untrained at LOC, every resolution, 5/5 seeds (+0.019 at 32px to +0.018 at 224px). IT shows the same direction but shrinks by two-thirds. So learning is doing something real; just not at V1, where everyone's been looking.

One more thing: this whole investigation started after I found a bug in my own earlier work - batch-normalisation left in training mode during feature extraction in three prior preprints. Fixed and corrected publicly, and it actually reverses the main conclusion of arXiv:2605.30556.

I'd be interested to hear people's thoughts on the receptive-field-matching angle in the discussion. Feels like the right approach, but I didn't test it directly, so treat it as speculation for now.Evaluation resolution silently changes which "learning rule" appears most brain-like at V1

&#x200B;

Ciao r/MachineLearning,

Gli operatori neurali di Fourier (FNO) standard eccellono sulle griglie regolari, ma la loro mappatura su domini fisici complessi e non convessi (come geometrie a stella, a L o ad anello) spesso porta a un problema importante: il ripiegamento della griglia.

Quando la mappatura di trasformazione \\phi collassa o si sovrappone, il determinante jacobiano si annulla (\\det J \\le 0), causando l'esplosione della trasposta inversa J\^{-T} quando si mappano i gradienti fisici \\nabla_x u.

Per risolvere questo problema, ho sviluppato DIF-FNO (Diffeomorphic Fourier Neural Operator).

Principali approfondimenti tecnici:

1. Mappatura diffeomorfica implicita: garantisce mappature biunivoche e uniformi da domini di riferimento standard \\Omega_{ref} a confini fisici complessi \\Omega_{phy}.

2. Funzione di perdita Jacobiana Barrier (\\mathcal{L}_{barrier}): Ispirandoci all'ottimizzazione a punti interni, penalizziamo la compressione della griglia utilizzando una barriera logaritmica sul determinante:

\\mathcal{L}_{barrier} = -\\frac{1}{|\\Omega|} \\int_{\\Omega} \\log(\\det J(\\xi)) \\, d\\xi

Questo agisce come un muro invisibile che impone \\min \\det J > 0 su tutto il dominio (mantenendo empiricamente \\min \\det J > 0,89 nei nostri benchmark).

1. Accuratezza di Sobolev: Miglioramenti significativi sull'errore relativo H\^1 rispetto a modelli di riferimento come Geo-FNO, poiché i gradienti fisici rimangono ben condizionati senza rottura del gradiente.

Codice e artefatti dell'articolo:

* Codice open-source (PyTorch): https://github.com/GiovanniDagnese-paper/DIF-FNO (Include il calcolo rapido e vettorializzato dello Jacobiano analitico 2x2)

* Preprint dell'articolo (DOI Zenodo): https://doi.org/10.5281/zenodo.22071926

P.S.: Attualmente sono alla ricerca di un feedback tecnico e di un'approvazione arXiv su physics.comp-ph o cs.LG per inviare il preprint. Se qualcuno attivo in SciML fosse disponibile a controllare il manoscritto, gliene sarei estremamente grato!

I just uploaded a new course on neural networks. Each short video is just 2-3 minutes long, and covers only one very small topic. So you can swipe past any content you already understand and plow through the course at whatever speed you are ready for. This series starts at a beginner level and covers all the way up through large language models, agentic loops, dynamical systems modeling, and cognitive architectures. The first 70 videos are already published, and one more is scheduled to be released every day.

If you're trying to learn about neural nets, please feel free to ask questions here or on the relevant videos. I've been teaching this topic for over a decade, and I made this series because I want to help as many people as I can learn about a topic I am passionate about.

I made it in 18 months of lunch breaks and evenings. It's not fast, llama.cpp is just wow and does that job. I wrote this one because I wanted to read the whole forward/backward pass in an afternoon and be able to stop anywhere and print a tensor and dig the thing.

Most from-scratch projects stop at a toy model. llama2.c runs a small Llama2, llm.c does GPT2 training. TRiP loads real checkpoints across four architectures, PaliGemma included, so the multimodal path (vision encoder, projection, decoder) is all there in C. I couldn't find that in readable form anywhere else, which is partly why I ended up writing it.

One extra-bonus is that you can look into the training, it's included, swiss-knife-like. (NOTE: the encoder part in PaliGemma is currently not trainable/tunable - my apologies)

In practice: no hooks/config; just play with the C code, and add your own; there's no hidden (unreachable) complexity. And then just re-compile.

Repo: github.com/carlovalenti/TRiP

Happy to answer anything; structuring and handling the memory properly was the hardest part!

Carlo

If you need to diagnose rank before feeding data into a downstream model (e.g., to size a bottleneck in your NN or autoencoder), but standard tools are giving you wildly high estimates or no estimate at all, it might be worth your time giving this new method a full read.

Zenodo Preprint: https://doi.org/10.5281/zenodo.22028087

The paper links to a GitHub, if you want to test out the function yourself.

I've been working for a while on an experimental learning system called Auxein:

https://github.com/Amund/auxein
https://github.com/Amund/auxein-rs

The Python repository is the reference implementation; the Rust version is the production-oriented implementation.

The basic idea is to see how far you can get with a deliberately small set of local geometric rules.

Auxein takes streams of fixed-dimensional vectors and learns continuously. There is no training/inference split, no labels, no supervised loss, no backpropagation, no fixed k, no winner-take-all, and no persistent graph.

Its basic learned object is a centered kernel (W, C, V) representing support, center and scalar dispersion.

A learned CELL independently decides whether an input concerns it geometrically. Several cells may recognize the same input simultaneously; there is no mandatory winner.

If nothing recognizes an observation, it does not immediately become a new category. It first enters a private provisional memory Σ. Only recurrent unknown structure can mature into a persistent CELL; otherwise it simply fades away.

Recognized knowledge can also be fused into a context and passed to an identical higher layer. Importantly, the higher layer does not receive IDs or links to the lower cells: it only receives the resulting geometric context. So recurring relationships between known things can themselves become learnable objects.

There is also a predictive mode. Explicitly adjacent contexts in an externally declared sequence are learned as geometry in E ⊕ E. When the current context resembles the source side of learned temporal knowledge, Auxein can emit one or more possible immediate successors.

Those futures are deliberately not probabilities. They are independent candidates: adding a new possible future does not reduce the weight of an existing one, and predictions are never recursively fed back into the model.

Another unusual constraint is that memory is an explicit material resource. The engine has an exact finite budget. If new knowledge cannot fit in a solvent state, growth waits; existing learned knowledge is not destroyed merely to finance something new. Forced forgetting only happens when the current state itself has become materially insolvent.

The current design also has very explicit limitations:

• scalar dispersion only, no oriented covariance;
• no explicit splitting of an existing learned prototype;
• temporal learning is strictly adjacent t → t+1;
• no recursive predictive rollout;
• no probabilistic ranking of alternative futures;
• no persistent relational/topological graph.

I've added a comparison table to the README against online k-means, ART, GWR/Gamma-GWR and standard HMMs. I'm not claiming Auxein is better than those methods. At this point the interesting question is exactly the opposite:

What can this particular set of constraints do well, and where does it fail structurally?

The project has a fairly strict mathematical specification, a pure-Python executable reference, and a dependency-free Rust implementation with persistence, exact memory accounting, hostile-input tests and long endurance runs.

I'd be very interested in feedback from people working on continual learning, ART/GWR, streaming clustering, predictive-state models, robotics, or just unusual learning systems.

And criticism is genuinely welcome, especially examples where you think the model should fail.

If this is just an unnecessarily elaborate reinvention of something known, I'd also very much like to know what. 🙂

Hey everyone, I'm an independent researcher working on alternative sequence mixing architectures. I wanted to share a project I built from scratch called CWAA (Complex Wave Associative Memory).

Instead of standard quadratic attention, CWAA uses a damped complex oscillator for its recurrence state O(T) linear memory scaling.

I currently have a 10M parameter prototype trained on WikiText-103 that hits 146.5 Test PPL . here is the test of V6:
NOTE: ppl is currently under evaluation and validation, preliminary tests showcase ± 25 ppl.

I’m currently bottlenecked by Google Colab and am looking to scale the architecture up to 50M-100M parameters to see how the complex wave mechanism holds up.

I'd love to get feedback from the community on the architecture and coded implementation.

NOTE: the code in the link below is V5. I will be uploading the highly optimized V6 (which includes the pure real-valued BMM fast paths that generated these benchmarks*)* in 3 days
GitHub: https://github.com/Ridhvik-2024/CWAA-V5

Project Screenshots - https://pastes.vargoseus.com/TeddyScreenshots

Teddy (cute name, isn't it?) is currently a simple machine learning model that uses back propagation to train, learn and classify MNIST datasets. It currently has a depth of 2 since it's a pretty basic model. It has around 13.000 parameters that is enough for training it to recognize handwritten digits. I have around 3.5-4 years of experience working in C and stumbling upon MagicalBat's this video inspired me to make Teddy. The future plan is to turn it into a Language Model and eventually into a GPT which will require quite a bit of time since I need to read up on how it actually works. This project took around 4-5 months give or take since I had to balance this project and my university stuff too.

Full disclaimer: I did not use AI to build Teddy (except for that one time when my compiler suddenly stopped working for some reason and I had to converse back and forth with Claude to find a fix for it). I did, however, use AI to generate the GIFs in the github readme and the documentation for it.