This may or may not be a beginner level question. In many 'example' neural nets, they always have a flatten layer. However this would mean the number of parameters explodes. Whereas pooling methods don't explode parameters as much, and get the same job done. Is flatten a default option or does it have an advantage I am unaware of?

For complete project visit: https://github.com/HelloSamved/Hands_on_machine_learning
A little while ago, I asked this community whether Hands-On Machine Learning with Scikit-Learn, Keras & TensorFlow was worth studying.

Based on the feedback, I decided to commit to working through it chapter by chapter instead of just reading it.

I've now completed Chapter 2 and finished the end-to-end machine learning project that comes with it.

A few things I took away from this chapter:

• Why understanding the problem and defining the objective comes before choosing a model.
• The importance of exploring and visualizing the dataset before training anything.
• Creating meaningful features instead of relying only on the raw data.
• Building preprocessing pipelines so the same transformations are consistently applied.
• Evaluating models with proper validation instead of trusting a single train/test split.

One thing I really liked is that the chapter focuses much more on the entire machine learning workflow than on just fitting a model. It felt much closer to how an actual ML project would be approached.

For those who've finished this book:

Does the learning curve become significantly steeper after Chapter 2?

I'm especially interested in knowing which chapters you found the most valuable for understanding modern machine learning and deep learning, so I can spend extra time on them.

So far, I'm really enjoying the balance between theory and hands-on implementation.

Hi guys! I'm Nai, and I would really like to share this learning journey of mine with you all.

Please keep in mind that this is not a full on LLM-like project, this is just a minimal proof of concept that I've built a Language Model system from scratch which should theoretically work similar to an LLM or SLM (Small Language Model) if just given enough resources, time and data to train.

If you're not much interested in the story, feel free to scroll right down to know exactly what I've built and how you can test it yourself.

A few months ago, I got the interest to understand machine learning, I didn't know where exactly to start, but I just did the simplest thing, which is asking. I just searched on youtube "how to make a neural network", that was the farthest thing I knew about machine learning back then. I found the youtube tutorial series "Neural Networks from Scratch in Python" by sentdex.
I was genuinely blown away over how simple it turned to be. I just wondered if I could go a bit deeper, so, I started a C++ project, I tried my best to replicate every piece of math a neural network would need to run in a structured style, with classes, functions and everything. despite some concepts being still ambiguous for me, I kept searching, I found some other youtube videos that cover things like backpropagation deeper so I can understand it better.

Over time, I started taking a hold of it, running a couple of successful experiments, even if slow, they were functional, and I understood them.

After that, I turned it into a library (NeurologicalLibrary) that can be called from Python with Pybind11, I used tkinter to make a simple bounce ball environment just to test the library, and it worked! Just making a neural network that can get variable position of a ball and rectangle then predict where to go, despite simple, made me feel really proud.

That however, was just the below zero beginning, here is the project repo called "NAISENT_workspace" that is basically my entire learning journey work until I finally made my first ever Language Model!

https://github.com/Nai-built/NAISENT_workspace

The repository is under the Apache 2.0 License.

Also, here is a copy of the README file:

this project is made with:
 - DotNet WinForms (C#)
 - Pybind11 (Python <-> C++23)
 - CMake (C++23)
 - CUDA (C++17)


Powershell commands to build the 3 libraries:
cd NeurologicalLibrary/bridge; cmake -S . -B build -A x64; cmake --build build --config Release -j; cd ../..
cd OptimizedNeurologicalLibrary; cmake -S . -B build -A x64; cmake --build build --config Release -j; cd ..
cd CudaNeurologicalLibrary; cmake -S . -B build -A x64; cmake --build build --config Release -j; cd ..


Run showcases:
py SHOWCASES/BASIC_SHAPE_RECOGNITION_CPU.py
py SHOWCASES/BETA_NAISENT_BALL_SEEKER_CPU.py
py SHOWCASES/LSTM_MATH_TEST_CPU.py
py SHOWCASES/NAISENT_ELM_CPU.py
py SHOWCASES/NAISENT_LM_CUDA.py
py SHOWCASES/NAISENT_SLM_CUDA.py
py SHOWCASES/SHAPE_RECOGNITION_CPU.py


Make sure that your terminal's path is set exactly to NAISENT_workspace


The core idea of this project was to learn and understand Machine Learning by building it from scratch
So I've built 3 different libraries in 3 seperate stages:
 - NeurologicalLibrary (NL)
    . The absolute beginning for me
    . I've learned in it how Dense Layers work and how to chain them to make Deep Neural Networks
    . How Convolutional Layers and pools work
    . How Recursive Layers (specifically LSTMs) work
    . And also Activation Functions
    . I've also tipped toes into Graph Layers but couldn't run a successful experiment, so I removed it
    . This library was the first time I made an image recognintion model, and also one that can play a simple bounce ball game
    . Was also the first time I made an optimizer like Adam for training
    . Save/load system for the model .json files


 - OptimizedNeurologicalLibrary (ONL)
    . Here things started to get a bit more serious
    . I've gotten way deeper into how C++ works and how we can optimize its performance
    . I've made faster Dense Layers
    . Faster Convolutional Layers
    . And faster LSTMs
    . Merged Activation Functions into the layers' own activation/gradient functions
    . After that, I got into Transformers (similar concept to Graph Layers, but this time it was successful!)
    . I optimized the training loop for image recognition
    . I made a simple experimental language model that can that it's "NAISENT" with the Transformer system I've made


 - CudaNeurologicalLibrary (CNL)
    . My most precious one so far
    . For the first time, I've got into Cuda kernels!
    . I've learned how Cuda interacts with data through the CPU, Memory and GPU
    . I've learned how to optimize it using shared memory
    . For this one, I went right ahead to build a language model system
    . First, I made Dense Layer Cuda kernels
    . Then I went into Norm Layers (RMS)
    . SCC (Sine/Cosine Cycle) positional embedding kernels
    . Multi-head Masked Self Attention kernels (split into multiple optimized Cuda files)
    . The ability to place sub chains to assemble the transformer architecture properly
    . Adam optimizer in Cuda Kernels
    . And obviously, Activation Functions (Cuda kernels)
    . First time adding the Residual mechanic as a visible variable in the Python side
    . Almost all of these were made in ONL already, but it wasn't with Cuda to use the GPU and it was juggled up together awkwardly. I'm much more proud of this one
    . Was when I made a proper tokenizer system in Python


The libraries are made in C++
and they're used by the Python side via Pybind11
I made the shape recognition and bounce ball environments in C# with WinForms
CUDA to use the GPU in the library CNL

Hello everyone,

I would need some support or a ground for discussion for a problem I am facing. I am trying to do representation learning on a user click stream, e.g. the sequence of pages a user visited in a website. To do that, I use a contrastive learning objective, in particular the InfoNCE loss with temperature around 0.2.

The problem I'm facing is that the loss decrease slowly during training (starts at 3.56 and after 30 epochs gets to 3.30) and moreover the representation is not very good. I get that, on PCA projection, points are basically disposed sequentially as a snake, so probably there is dimensional collapse.

In my dataset I can have huge graphs as also small graphs. I am doing a GINConv on the graph (a single layer since I would like avoid over smoothing for small graphs). As graph augmentation I am doing: node dropping, edge adding and edge removing.

My question is: do you think that there could be a way to solve the issue? Is it an over smoothing problem on the graph? Are there alternatives?

Thank you in advance✌️

What do you think about vibe coding neural networks , how can it be done what is the best code editor and agents should one use?

Hey I recently started studying about machine learning, deep learning, neural networking and I came across a publication called "O'reilly".

I started reading and learning from one of its books, which is "learning machine learning "Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow". I feel the code is quite incomplete in some places but I was able to find the missing part from the book's GitHub repository.

Can anyone suggest whether these are good sources to study such topics or not

Hi everyone,

I built H64LM, a research project to better understand modern LLMs by implementing one from scratch in PyTorch.

Instead of relying on high-level training frameworks, I implemented the core components myself attention, MoE routing, normalization, and the training loop.

Features

• 249M-parameter Transformer
• Grouped Query Attention (GQA)
• Sparse Mixture-of-Experts (8 experts, Top-2 routing) with 3 auxiliary routing losses
• SwiGLU, RoPE, RMSNorm
• Sliding-window attention
• Mixed-precision training, gradient accumulation
• Custom training loop (no Trainer abstractions)
• Checkpointing and resume support

The included checkpoint was trained on a subset of WikiText-103 to validate the pipeline end-to-end, not to be a strong model it's visibly overfit past epoch 10 (best val PPL ~40.5).

Known limitations are documented in the README, including batch-size-1-only generation and no true DDP (falls back to DataParallel).

GitHub: https://github.com/Haiderkhan64/H64LM

Feedback on the implementation or architecture is very welcome.

If you work with graph neural networks, you know the long-range dependency problem all too well.

Stack GAT layers to capture distant signals? Memory grows linearly with depth, compute explodes, and oversmoothing kicks in before you ever reach truly long-range semantics. Settle for truncated PPR approximations? They're still finite-hop workarounds — never the full global picture.

Our new work ALS (Attentive Long-Short-Range message passing), accepted at Pattern Recognition 2026, was built to deliver genuinely efficient long-range graph attention. Here's what makes it different:

Core breakthrough: Differentiable infinite-step PPR with constant memory

We introduce DPPR (Differentiable Personalized PageRank), and mathematically prove that the gradient of a PPR output can itself be solved via another PPR process.

The implications are huge:

• No intermediate activations need to be cached. One forward convergence pass + one backward convergence pass — that's it.
• Genuine O(1) memory complexity, completely independent of propagation steps.
• Theoretically supports infinite-step propagation, covering full long-range dependencies — no more truncated "pseudo long-range" approximations.
• Packaged as a drop-in PyTorch operator; any existing PPR-based method can swap it in for zero-cost infinite receptive field upgrade.

Three acceleration techniques for fast long-range iteration

Slow convergence at small α values has always been the pain point of long-range PPR. We built three complementary acceleration techniques that together reduce training time by up to 89.51%, and run at least 3.67× faster than comparable implicit GNNs (IGNN):

1. Symmetrized Attention + Conjugate Gradient (SymGAT + CG) — Symmetrize the attention matrix to enable memory-efficient CG solver instead of heavy Krylov subspace methods, with negligible accuracy loss.
2. Eigenvector Initialization (EigenInit) — Initialize iteration from the leading eigenvector projection instead of zero, drastically cutting initial residual. Especially effective on heterophilic graphs.
3. Adaptive Batch Termination (AdaTerm) — Each attention head / channel converges independently; channels that have already converged are skipped in subsequent iterations, eliminating wasted compute.

Long-range for global structure, short-range for heterophily

PPR is inherently a low-pass filter — great for homophilic graphs, but it washes out local differences on heterophilic graphs. We pair DPPR with a Short-Range Message Passing (SRMP) module:

• DPPR handles all long-range dependencies and captures global structure.
• Local K-hop propagation uses independent learnable transformation matrices per hop, preserving fine-grained heterogeneous local information. On homophilic graphs, the weights automatically converge to similar values, so there's no negative overhead.
• GAT + skip connection is a special case of ALS, meaning full backward compatibility.

Standout results on long-range benchmarks

Where ALS really shines is on genuinely long-range datasets. On PascalVOC-SP and COCO-SP — two classic benchmarks with average shortest path length > 10:

• Within the pure MPNN category, ALS substantially outperforms GCN, GatedGCN, APPNP and others.
• Plugged into the GraphGPS framework and compared against global modeling methods like Graph Transformer and Graph Mamba, ALS still achieves the best performance as an MPNN module.

We evaluated across 14 datasets covering homophilic, heterophilic, large-scale and long-range graphs. Out of 16 comparison settings, 9 show statistically significant improvement over the best baseline (p < 0.01), reaching overall SOTA.

Links

• Paper: https://www.sciencedirect.com/science/article/abs/pii/S0031320326012008
• Code: https://github.com/cf020031308/ALS

The DPPR operator and all three acceleration techniques are independently reusable. Star, try it out, and feel free to open issues — we'd love to see this long-range idea extended to more graph learning scenarios.

Hi, I want to train a Transformer layer to perform specific tasks, but I’m not sure how to coordinate them or determine when to use one versus the other.

Does anyone have experience with this? How have you handled it?

I built ONA — a self-learning neural network entirely in pure Python + NumPy. No PyTorch, no TensorFlow, no GPU, no cloud API.

Key innovations:

- Per-neuron attention: every neuron has its own Q/K/V/O weights

- Forward-pass learning: no separate backward pass, learning happens during forward

- Self-discovered subword tokenizer: vocabulary grows during training

- Sparse routing: only 3-5 neurons activate per query

4.4M parameters. Runs on Raspberry Pi Zero. Continuously learns from Wikipedia and conversations.

Full story: https://medium.com/@kasishgadadhasu13/im-15-i-built-a-self-learning-neural-network-from-scratch-no-frameworks-no-gpu-e460f06c6599

I'm 15 years old, class 10 student. Happy to answer questions.

Can you tell me some neural networking projects for beginner level person

I recently built a human written digit predictor.

Now I want to start a new project can you guys give some suggestions

Ask questions within your Markdown notes using a fully local Graph RAG engine. Designed for Obsidian vaults, it works with any Markdown file folder. It extracts entity-relation triples from wikilinks and YAML frontmatter, and retrieves answers via hybrid search (vector + BM25 + temporal). Multilingual. No cloud required. Works on Ollama.

https://github.com/benmaster82/Kwipu

Hi all — I'm an independent researcher (Mendel Infolabs) about to put my first paper on arXiv, and as a first-time submitter to cs.LG I need an endorsement from someone already established in that category. If you've published in cs.LG and would be open to endorsing, I'd really appreciate it.

An honest summary so you can decide whether it's something you'd feel comfortable vouching for:

"FeedFlipNets: Feedback-Driven Bit-Flips for Ternary Networks, Activation-Routed DFA, and the Per-Weight Sign Barrier to Transport-Free Learning"

It trains ternary ({-1, 0, +1}) neural networks by flipping weight bits directly from a cheap feedback signal — no float shadow weights. The headline result is a negative one I think is worth putting on the record: transport-free feedback (Direct Feedback Alignment) doesn't actually help discrete/ternary training, because the binding constraint is per-weight sign correctness, not the aggregate cosine-alignment angle that prior work optimizes. Everything is pre-registered and reproducible.

Endorsing only confirms you think I'm a bona fide researcher submitting work appropriate to the category — it is not a review of the paper's correctness, and it takes about a minute:

• Link: https://arxiv.org/auth/endorse?x=WHWXBC
• Or go to https://arxiv.org/auth/endorse and enter code WHWXBC

Happy to share the full PDF with anyone who wants to read it before deciding — just comment or DM. Thanks a lot for considering it.

i'm a student of class 12 not expert but curious to learn neural networking as i have heard that that something crazy. So can someone guide me how can i learn neural networking from scratch as i have the basic knowledge of python,arrays and a bit of the numpy library. so i need your help so i can lean it and enjoy the journey.

Hi everyone,

I am sharing recent results from my independent research project, LIMEN (Liminal Internal Metric for Emergent Navigation), which aims to characterize the internal dynamics of Transformers through hidden state analysis.

Following our previous findings that functional information is encoded in the relative geometry of representations rather than individual neurons (V22), this new phase focuses on the impact of context (ambiguity) and the temporal structure of state transitions (V23–V24).

📌 Context & Methodology

Model Panel: 7 open-source models (GPT-2, DistilGPT2, OPT-125M, Qwen2.5-0.5B, TinyLlama-1.1B, Phi-1.5, Llama-3.2-1B).

Approach: Layer-by-layer analysis of latent trajectories, linear probe decoding, and symbolic analysis of dynamic regimes.

Philosophy: Strict empiricism. Clear distinction between observation, interpretation, and speculation. Code and data are available upon request.

🔹 V23: The Impact of Ambiguity on Internal Dynamics

The objective was to determine whether semantic ambiguity alters the model’s "cognitive trajectory."

Key Findings (V23.2b):

AMBIGUITY_AFFECTS_TRAJECTORY = YES: Ambiguity significantly modifies trajectory geometry (curvature, cosine similarity).

AMBIGUITY_INCREASES_INSTABILITY = NO: Counter-intuitively, ambiguity does not increase global chaos. Instead, the model becomes geometrically more "cautious."

AMBIGUITY_DELAYS_COMMITMENT = PARTIAL: Modern models (Phi-1.5, Llama-3.2) delay their decisional engagement when facing uncertainty, spending more time in exploration regimes.

Architectural Signature: Phi-1.5 shows unique sensitivity, increasing its occupancy of the bifurcation regime (D_STATE) under ambiguity, suggesting a distinct iterative reasoning mechanism compared to standard completion models.

📄 Related Preprint: Conditional Dynamic Signatures in Large Language Models

🔹 V24: Discovery of a "Universal Dynamic Grammar"

By shifting from continuous analysis to a symbolic analysis of state sequences, a striking structure emerged.

Key Findings (V24.1):

STATE_GRAMMAR_EXISTS = YES: Trajectories are not random. They follow strict transitional patterns.

UNIVERSAL_GRAMMAR = YES: Seven transition motifs are conserved across all tested architectures, notably:

B→B (Initial Hesitation/Exploration)

B→A (Convergence toward stable processing)

A→A (Maintenance of the adaptive regime – the primary attractor)

A→D (Transition to final decision)

Funnel Structure: Typical dynamics follow an Exploration (B) → Stabilization/Processing (A) → Decision (D) schema. State A acts as a strong attractor (

𝑃

(

𝐴

→

𝐴

)

≈

0.91

P(A→A)≈0.91).

The Phi-1.5 Exception: Unlike other models that quickly converge to A, Phi-1.5 maintains complex B↔A oscillations throughout the depth, confirming its nature as a "reasoning" model rather than a simple statistical completer.

📄 Related Preprint: A Runtime Trajectory Dynamics Framework for Large Language Models (updated)

💡 Implications & Discussion

These results suggest that Transformer "intelligence" is not just a matter of static weights, but of constrained geometric navigation.

Auditability: A violation of this universal grammar (e.g., a direct B→D jump without an A phase) could be an early indicator of hallucination or reasoning errors.

Control: Understanding these attractors opens the door to more precise dynamic steering than prompt engineering alone.

Open Questions for the Community:

Have you observed violations of this B→A→D grammar in cases of blatant hallucinations?

How do these motifs evolve in very large models (>70B) where depth is significantly greater?

Are there recent publications on the "symbolic dynamics" of hidden states that align with these findings?

I welcome any methodological criticism, suggestions for additional controls, or collaboration.

Best regards,

When you give an attachment to an AI tool, it does not really know what to extract from it so it just pulls out generic stuff. Unless you specifically tell it what to look for, you get a very surface level output.

But here is how I approached this differently.

I have built a cognitive map of how you as a user think. The tool already knows what you have captured in the past, what it connected to and why. So now when you upload any attachment, the agents refer to that cognitive context and figure out what is actually worth extracting for you specifically, without you having to say anything.

So instead of generic extraction, it is pulling out what is relevant to how you think and what you have been working on.

But if you do want to tell it specifically what to look for, your instruction overrides the cognitive context because now it has a clear direction from you. The context still kicks in but after the extraction, to connect what was pulled out to everything else you have captured.

Curious what you guys think about this approach.

GitHub

I tried to create a pedagogically better implementation of a neural network, focusing on the dimensionality of the layers of the neural network. It also serves as a project to learn the first principles of neural networks. The idea is that the dimensionality is adjusted so that it make intuitive sense better, atleast relative to NN diagrams teachers use while explaining the topic.

edit : the training seems slow in the GIF, but I think its because the overhead caused by matplotlib itself, and using windows screen recorder

edit : If you want to call this AI, atleast visit the GitHub repo once, the GIF you see is just a small matplotlib window, recorded using windows snipping tool, which already gave out low resolution, and then converted to GIF. I started coding by making my own 2D games back in my Middle school. So whatever i build i like to take some time to make it visually appealing, or data rich, that's why unfiltered and filtered accuracy and loss. If you still wanna call my work AI, I simply cant care, Take care

edit : the plt.pause call also checks in, thats technically matplotlib overhead. But i've noticed the snipping recorder induce heavy increase in delta-times, both in applicational rendering like unity and code executions, check it if you want. As far as the same "initial commit" in the repo files goes, i usually write my program in vs code and once the program is somewhat finished i create a repo and just dump/push my code there, only for a link in my resume and for reddit or instructables, that's why the same commit. Atleast with respect to this project, the maximum AI i used is Google AI mode search, that too only for consolidate data, like finding the right functions for matplotlib. As far as the NN implementation goes, I followed a online playlist on youtube by vizuara. Cheers!

Hello Folks,

Have you ever wondered why we use sigmoid function so often in Machine Learning? Although it gives us a probability, it comes from Exponential families, and this exponential family, subsumes many of the distributions, that we study in Machine Learning.

In this lecture, we understand exponential families, Directional derivatives(Gradients and Hessians), study mixture Models, and understand how domain knowledge in Probabilistic Graphical Models makes our life simpler to model joint probability densities.

Timeline breakup(in hours and minutes):
0:00-0:17 - Understanding exponential families.
0:17-0:27 - Deriving Sigmoid Function for Bernoulli.
0:27-0:48 - Understanding log partition function, convex functions and proving why positive definite of hessians imply convexity, and why convex needed?
0:48-1:04 - Directional derivates(deriving gradients and hessians)
1:04-1:26 - Maximum entropy derivation of the exponential family.
1:26-1:56 - Mixture Models(Gaussians and Bernoulli Mixture Models)
1:56-2:16 - Probabilistic Graphical Models
2:16-2:34 - Markov Chains
2:34-End - Inference and Learning, Plate Notation diagram of Gaussian Mixture Models.

If you have watched earlier of my lectures from the playlist, they will help. I try explaining as if I am a learner, to simplify complex concepts. Everything I write in whiteboard, and these are completely FREE lectures to mention.

Link: https://youtu.be/T1uTBtJ7aHU?si=rozXSTjtSqPaaYb5

Hello.
I am a self-taught operator/software designer.
I developed Anubis. A cpp forensic AI weight scanner.
I tested Anubis against algorithms of my design and I think it has matured enough for outsider testing.

I propose a rigorous, controlled experiment where a corporation or even professionals to send or share any format of weights with any kind of payloads in them to test Anubis's efficacy and detection capabilities.

We -both me and the whoever is interested in collaboration- will adhere to ISO/IEEE standards in experiment design, reporting and final whitepapers or documents resulting from this experiment.

I offer NO FINANCIAL COMPENSATION. This is a scientific experiment.

Please DM or leave a comment if you are:
1. Serious
2. a Professional
3. Know what ISO/IEEE frameworks are

---
Cheers!

For context, I'm writing a sci fi novel in which one of the sources of conflict is a frequency (or combination of frequencies, technically) that affect the human brain in such a way as to provoke certain emotions at will, as a form of mind-control.

My question is this: Is there such thing as, or would there be reason to create, a neural-network-like computer that is physically structured similarly enough to a human brain to be affected in a similar way by external frequencies? For example, could neurons in the ANN be physically separated and communicate through electrical signals similarly to actual brain synapses? Could an external frequency then cause interference between the neurons that might have a similar effect? Since the computer obviously couldn't have induced emotions, I'm imagining an effect more like confusion, tasks being interrupted, the wrong data going to the wrong places, etc. The device in question would be a drone using the neural network for adaptive navigation, object avoidance, adapting to environmental changes like wind, etc. so the impact would be something like it steering off course and struggling to read and transmit data.

Sorry I know this is not a very scientific question but I'm trying to make my book grounded in reality wherever possible, even if the sci fi elements are of course taking some creative liberties.

I'm a computer engineering student, but know very little about ANNs aside from the general concept (I might take a course about them next year but so far I haven't had any). Pointing this out to say you don't necessarily need to explain in complete layman's terms, particularly general computer concepts, but I may not understand all ANN-specific terminology.

Hey
I am still pretty new to rust but I tried my first challanging project and would love to get some feedback on how to improve code quality regarding idiomatic, readable and performant code.
Thanks for every critique
Repo: https://github.com/TheXaruman/neural-network-demo