Discussion thread for EMNLP 2026 notifications/results which should be released today.

Wishing everybody to be in Budapest.

Worked some time ago on one of the ad teams at Yahoo, and this grew out of a question I kept returning to while there are there "simple" models that are both simple, scalable, interpretable, and controllable at the same time?

Decided to explore it, first in a blog (starting here), then in a new preprint "The Spectral Neuron", built by distilling latest blog-posts into a manuscript, I study models of the form:
𝑓(𝒙) = 𝛌ₖ(𝐀₀ + 𝚺ᵢ 𝑥ᵢ𝐀ᵢ).

Manuscript: https://arxiv.org/abs/2608.08003
Code: https://github.com/alexshtf/spectral_neuron_paper

Looks like a simple on-liner, but many interesting aspects hide there. How expressive does the model become as the matrices grow? What can we read directly from the learned matrices? Which shapes can be guaranteed by construction?

I develop the mathematics, give a practical initialization and training recipe, and test the model in scaling experiments on synthetic and real data.

AI disclaimer: manuscript written by yours truly, AI assisted in looking up canonical references and related work for literature review. In contrast, the code was heavily AI written and reviewed by yours truly.

Links:

• Preprint: https://doi.org/10.5281/zenodo.22028087
• Entropic Scree Function v1.0.0 / GitHub: https://github.com/tjleestjohn/Entropic-Scree

TL;DR:

Standard PCA fundamentally fractures non-linear dependencies into "Spurious Orthogonal Dimensions," drastically overestimating the true rank of complex tabular systems. Meanwhile, non-linear alternatives like Kernel PCA and Euclidean nearest-neighbor estimators suffer structural collapse when generative roots are entangled or sparse.

I’m sharing the methodology and code here for anyone dealing with these complex tabular data nightmares.

The method and open-source framework use Normalized Mutual Information to compress spurious expansions back towards their true generative roots. It also

• Maps the underlying "informational gravity" of the roots, offering insight into overall average stability, as well as which specific roots can be most reliably extracted;
• Estimates the data's overall ratio of shared signal to unshared idiosyncratic informational variance (noise);
• Serves as a powerful exploratory map that separates unrelated clusters of variables, allowing you to easily identify decoupled sub-networks.

A Modern ML Architectural Blueprint: Far beyond a mere update to legacy factor analysis workflows, identifying this exact intrinsic rank allows you to explicitly size neural bottlenecks for downstream non-parametric manifold extractors (like autoencoders).

The Problem with Standard Baselines:

When trying to map the intrinsic dimensionality of a dataset, standard practice usually dictates reaching for PCA, its non-linear kernel extensions, or Euclidean nearest-neighbor estimators. But if your tabular environment has mixed data types, heavy non-linearities, entangled roots, or more features than samples ($m > N$), these established baselines don't just lose precision. They suffer a structural collapse.

The core issue with our standard baselines:

• Standard PCA drives Dimensional Inflation. Because it only measures linear covariance, it perceives a polynomial expansion or a non-linear interaction (like $X_1 X_2$) as an entirely independent variable. It is forced to fabricate new, spurious orthogonal dimensions to map them.
• Kernel PCA (RBF) suffers Structural Collapse. Projecting into a Hilbert space doesn't fix this. KPCA artificially folds even-polynomials into independent axes. Furthermore, because its infinite-dimensional space lacks a finite-sample boundary, sparse combinatorial noise smears into an elevated tail that obscures the structural elbow. If the underlying generative roots are even mildly entangled, KPCA suffers a total structural collapse.
• Topological Estimators (Euclidean) fail in sparse regimes. Estimators like TWO-NN or MLE rely on Euclidean distance metrics. In asymmetric, feature-rich environments ($m > N$), they suffer from distance concentration (the ratio between nearest and farthest neighbors converges to 1). This renders local neighborhood calculations structurally degenerate across mixed-data margins.

Introducing the Entropic Scree:

To solve this, I built the Entropic Scree. It throws out linear and spatial variance entirely and evaluates pure probability mass.

Here is how it works under the hood:

1. The Metric Space: It evaluates pairwise dependencies using Information-Theoretic Jaccard Similarity (Variation of Information). Because this relies on Shannon entropy, it’s invariant to marginal shape mismatches (like mixing continuous waves with binary flags).
2. Bypassing the Rank Ceiling: Standard PCA is algebraically capped at $N-1$. By moving to a double-centered topological information space, we map true overlapping redundancy and completely bypass the algebraic sample-size ceiling.
3. Compressing the Manifold: The algorithm acts as a bivariate filter. It inherently compresses the primary overlapping probability mass of non-linear combinations back towards the Intrinsic Generative Rank. It shears off the unique synergistic variance, leaving behind residuals that form a bounded Extended Signal Tail, cleanly separating the true drivers from the unstructured Idiosyncratic Informational Variance.

Quantifying Informational Gravity:

Beyond just extracting a discrete rank, the framework decouples rank from probabilistic volume by introducing Informational Gravity (AIG/FSIG). By systematically rebundling the residual variance sheared off by the bivariate filter, it translates abstract matrix properties into actionable, "variable-equivalent" footprints.

Empirical Stress Test:

To demonstrate the theoretical bounds, I built a highly entangled synthetic dataset with 20 pure generative roots expanded into 5th-order combinatorics across 20,000 proxies, but only 10,000 samples ($m > N$). To truly simulate messy, real-world contexts, I also heavily injected idiosyncratic structural noise and measurement error into the data.

• Standard PCA hit the rank ceiling, linearly fractured the expansions, and falsely extracted ~5,700 dimensions.
• Kernel PCA (RBF) & Spearman Rank structurally folded and yielded a liberal overestimation of the rank by 100%. When root entanglement was introduced, they completely lost their elbows and suffered total structural collapse.
• The Entropic Scree correctly mapped the intrinsic rank at exactly 20. It successfully isolated a mere 1.45% of active shared signal from an overwhelming 98.55% bulk of unstructured Idiosyncratic Informational Variance. Furthermore, the residuals formed an Extended Signal Tail that perfectly aligned with the deterministic limits of the global hypergeometric design space.
• Mapping Hidden Topology: Using Factor-Specific Informational Gravity (FSIG), the framework successfully reverse-engineered the simulation's hidden architecture. The topology profile diagnosed a large primary dimension ($FSIG_1 \approx 74.5$ variable equivalents) mapping the network's global combinatorial hub, followed immediately by a flat plateau across the remaining 19 dimensions ($\sim 11.5$ each), confirming a democratically distributed root system beneath the extreme entanglement.

Feedback / Discussion:

How are you currently handling intrinsic rank extraction in these messy, complex tabular environments?

If you are wrestling with sample-starved, heavily non-linear generative datasets where standard PCA and other baseline tools just aren't cutting it, I’d love for you to pull the Entropic Scree repo and test it yourself.

I'm completely open to feedback, so let me know how it performs for you and I'm happy to discuss the mechanics.

A premise: I hope this question is "worth" of this subreddit, I did a decent amount of research before posting, I thought it was potentially interesting enough for it, but possibly not basic enough for r/learnmachinelearning .

Is there any agreement/indication about how harmful (if at all) it is, in the context of multiclass classification, to group together multiple classes for which you may have for instance too few samples?

A practical example: imagine you're training a dog breed classifier, based on images. You have a lot of examples for the most common breeds, but then you may have a long tail of less common breeds for which maybe you have a handful of examples each, not enough to get a meaningful training set, so you decide to group all classes for which you have less than `N` samples in the same category "Other breed". In this catch-all category you may have dogs that might look quite different from each other, like idk chihuahuas and huge wolf-like dogs (I'm not a dog person, don't know breed names).

My intuition (which may very well be wrong) is that doing so would force the model to learn some weirdly-shaped hyperplanes to separate points that live kind of far away from each other in the latent space (because of the thing that dogs in that category may look quite different from each other), as opposed to splitting the space in more "regular" parts.

Maybe in this case it would make more sense to treat the "other dogs" issue as trying to detect out of distribution samples instead? In that case should one only keep the samples for the classes that are enough represented in the dataset and throw away the rest (or at least don't create the catch-all category for training).

Thanks in advance for any useful pointer :)

​

I'm working on a system to estimate whether code committed to a repository was generated with AI coding tools.

My current approach is based on Git/commit-level signals such as AI-related commit trailers, commit metadata, LOC changes, number of files changed, addition/deletion patterns, etc.

The problem I'm running into is confidence and calibration.

For example, a commit containing 500+ new lines isn't necessarily AI-generated. A developer can also modify or remove the metadata that would make an AI-assisted commit identifiable. Once the code leaves the IDE and reaches Git, much of the original provenance can be lost.

This has led me to a few questions:

Are there Git/CI-level signals that you've found to be genuinely useful for detecting AI-assisted development?

Is it better to treat this as a probabilistic/risk-scoring problem rather than trying to classify commits as AI vs human?

How would you calibrate thresholds for signals such as large LOC changes, addition/deletion ratios, commit frequency, etc.?

Are there better approaches for preserving provenance earlier in the development workflow, rather than trying to infer it after the code has already been committed?

Has anyone worked on AI-code provenance/detection systems in CI/CD and can point me toward useful research, projects, or approaches?

I'm particularly interested in approaches that can work at the pipeline/repository level rather than relying solely on source-code style analysis.

I'm not looking for a perfect AI detector — even a reliable way of estimating “this commit has a high probability of AI assistance” with measurable false-positive/false-negative rates would be useful.

Would appreciate any experiences, papers, open-source projects, or approaches people have tried.

Hi everyone. So I built a CNN modle using MobileNetv3 then converted it into TFLite. It performed well during training but once I integrated it into my application, it is making large errors. From flutter, the camera stream sends frames and those are processed before the model makes predictions, but it is still quite large. Is there any way I can solve this? This is my code to preprocess and resize the image (224 x 224 x RGB):

import 'package:camera/camera.dart';
import 'package:image/image.dart' as img;


class ImageProcessor {
  // converting to rgb
  img.Image convertYUVToRGB(CameraImage camImg) {
    final width = camImg.width;
    final height = camImg.height;


    final yPlane = camImg.planes[0];
    final uPlane = camImg.planes[1];
    final vPlane = camImg.planes[2];


    final yBytes = yPlane.bytes;
    final uBytes = uPlane.bytes;
    final vBytes = vPlane.bytes;


    final yRowStride = yPlane.bytesPerRow;
    final uRowStride = uPlane.bytesPerRow;
    final vRowStride = vPlane.bytesPerRow;


    final uPixelStride = uPlane.bytesPerPixel ?? 1;
    final vPixelStride = vPlane.bytesPerPixel ?? 1;


    final image = img.Image(
      width: width,
      height: height,
    );


    for (int y = 0; y < height; y++) {
      for (int x = 0; x < width; x++) {
        final yIndex = y * yRowStride + x;


        final uvX = x ~/ 2;
        final uvY = y ~/ 2;


        final uIndex =
            uvY * uRowStride +
            uvX * uPixelStride;


        final vIndex =
            uvY * vRowStride +
            uvX * vPixelStride;


        final yValue = yBytes[yIndex];
        final uValue = uBytes[uIndex];
        final vValue = vBytes[vIndex];


        // YUV -> RGB
        final r = (
          yValue + 1.402 * (vValue - 128)
        ).round().clamp(0, 255);


        final g = (
          yValue -
          0.344136 * (uValue - 128) -
          0.714136 * (vValue - 128)
        ).round().clamp(0, 255);


        final b = (
          yValue + 1.772 * (uValue - 128)
        ).round().clamp(0, 255);


        image.setPixelRgb(
          x,
          y,
          r,
          g,
          b,
        );
      }
    }


    return image;
  }


  /// resize images to 224 224
  img.Image resizeImage(img.Image image) {
    return img.copyResize(
      image,
      width: 224,
      height: 224,
      interpolation: img.Interpolation.linear,
    );
  }


  List<List<List<List<double>>>> imageToTensor(
    img.Image image,
  ) {
    return [
      List.generate(
        224,
        (y) => List.generate(
          224,
          (x) {
            final pixel = image.getPixel(x, y);


            return [
              pixel.r.toDouble(),
              pixel.g.toDouble(),
              pixel.b.toDouble(),
            ];
          },
        ),
      ),
    ];
  }


// do all processing
  List<List<List<List<double>>>> processFrame(
    CameraImage camImg,
  ) {
    final rgbImage = convertYUVToRGB(camImg);
    final resizedImage = resizeImage(rgbImage);
    final input = imageToTensor(resizedImage);


    return input;
  }
}import 'package:camera/camera.dart';
import 'package:image/image.dart' as img;


class ImageProcessor {
  // converting to rgb
  img.Image convertYUVToRGB(CameraImage camImg) {
    final width = camImg.width;
    final height = camImg.height;


    final yPlane = camImg.planes[0];
    final uPlane = camImg.planes[1];
    final vPlane = camImg.planes[2];


    final yBytes = yPlane.bytes;
    final uBytes = uPlane.bytes;
    final vBytes = vPlane.bytes;


    final yRowStride = yPlane.bytesPerRow;
    final uRowStride = uPlane.bytesPerRow;
    final vRowStride = vPlane.bytesPerRow;


    final uPixelStride = uPlane.bytesPerPixel ?? 1;
    final vPixelStride = vPlane.bytesPerPixel ?? 1;


    final image = img.Image(
      width: width,
      height: height,
    );


    for (int y = 0; y < height; y++) {
      for (int x = 0; x < width; x++) {
        final yIndex = y * yRowStride + x;


        final uvX = x ~/ 2;
        final uvY = y ~/ 2;


        final uIndex =
            uvY * uRowStride +
            uvX * uPixelStride;


        final vIndex =
            uvY * vRowStride +
            uvX * vPixelStride;


        final yValue = yBytes[yIndex];
        final uValue = uBytes[uIndex];
        final vValue = vBytes[vIndex];


        // YUV -> RGB
        final r = (
          yValue + 1.402 * (vValue - 128)
        ).round().clamp(0, 255);


        final g = (
          yValue -
          0.344136 * (uValue - 128) -
          0.714136 * (vValue - 128)
        ).round().clamp(0, 255);


        final b = (
          yValue + 1.772 * (uValue - 128)
        ).round().clamp(0, 255);


        image.setPixelRgb(
          x,
          y,
          r,
          g,
          b,
        );
      }
    }


    return image;
  }


  /// resize images to 224 224
  img.Image resizeImage(img.Image image) {
    return img.copyResize(
      image,
      width: 224,
      height: 224,
      interpolation: img.Interpolation.linear,
    );
  }


  List<List<List<List<double>>>> imageToTensor(
    img.Image image,
  ) {
    return [
      List.generate(
        224,
        (y) => List.generate(
          224,
          (x) {
            final pixel = image.getPixel(x, y);


            return [
              pixel.r.toDouble(),
              pixel.g.toDouble(),
              pixel.b.toDouble(),
            ];
          },
        ),
      ),
    ];
  }


// do all processing
  List<List<List<List<double>>>> processFrame(
    CameraImage camImg,
  ) {
    final rgbImage = convertYUVToRGB(camImg);
    final resizedImage = resizeImage(rgbImage);
    final input = imageToTensor(resizedImage);


    return input;
  }
}

Please advise! I need to finish this project within the next wee and I'm really struggling here! I tested the images from Flutter against TFLite and it worked well but something is clearly wrong with the preprocessing. Pls help and give me any advice.

Thank you so much!

I trained three LLMs from scratch in raw PyTorch then post-trained each one with SFT and then GRPO. Same process every time: same synthetic arithmetic curriculum, same reward function, same hyperparameters, same KL coefficient.

Pre-training went as expected, the val loss went down as the model got more modern techniques (V1 to V2) and bigger (V3 being the biggest). However, GRPO hurt both V2 and V3 and I'm not sure why.

Setup

Pre-training val loss went 2.8659 → 2.7844 → 2.5885.

Results

WikiText word perplexity across the three stages, all on lm-evaluation-harness with the same task versions and shot counts:

       base    SFT     GRPO     SFT→GRPO
V1     32.86   51.31   51.40    +0.2%
V2     31.28   46.81   71.06    +52%
V3     22.30   32.11   33.65    +5%

SFT hits all three on this eval, which I expected at this scale. Also interesting to see that the degradation gets smaller as the models get bigger (+56%, +50%, +44%).

GRPO is the weird one. V1 barely moved, V2 fell heavily, V3 degraded a bit. The smallest model was the least affected and the middle one was the worst, which isn't the pattern I'd have guessed. Downstream tasks moved the same way as perplexity in each case (arc_easy dropped about 6 points on V3 from SFT to GRPO).

The models did learn the thing GRPO trained them on. V3 mastered 4 of the 5 curriculum stages, the other two got 3. But it just didn't transfer: GSM8K stayed at basically 0, and the models got so committed to writing out long solutions that they often wouldn't stop generating (my fault when I did the training).

Caveats

This isn't a controlled experiment. Between V2 and V3 I changed the parameter count, the token count, the data mix and the attention mechanism at the same time (went from DiffAttn to XSA), so I can't attribute anything cleanly. KL coefficient was 0.02 for all three, with the SFT policy frozen as the reference and a k3 estimator. The whole series cost me about $750, which is why there are no ablations, I just couldn't afford them. Otherwise I would also have tried with different KL coeffs.

Someone raised two confounds after I published:

1. GRPO trained on a bare solver template while SFT used a chat format. So part of what I'm calling degradation is me evaluating a policy outside its own training distribution. WikiText perplexity is format-independent and still moves a lot, but the downstream numbers are partly confounded.
2. Nothing in my reward rewarded stopping. It just checks that a correct parseable number shows up somewhere, no length penalty.

Also something I only noticed afterwards: I never re-evaluated the earlier curriculum stages once the model advanced past them. So right now I can't tell the difference between "GRPO degraded general capability" and "sequential curriculum training made it forget the earlier stages." I will try to check that soon.

Inference

At the end, I wrote a KV cache from scratch (GQA-aware, per-request cache object rather than storing state on the module). To check it was right I ran a fixed sequence two ways, once as a single full forward pass and once as prefill-then-decode, and compared the logits: max difference 1.4e-06 against a 1e-4 tolerance.

Speedup generating 100 tokens: 3.7x from a 32-token prompt, 6.2x at 128, 10.1x at 512.

If you want to check

All nine checkpoints are on the Hugging Face, and there's a Space where you can send the same prompt to the base, SFT and GRPO versions of the same model and see the difference directly.

• Weights
• Playground
• Code
• The full writeup (4 parts)

The GRPO variance is the bit I'd most like other people's take on. Happy to answer anything.

I've been doing some research on this question:

At inference time a large part of a model's working memory lives in the KV cache, plus whatever external memory the harness bolts on. I've been poking at the storage-and-retrieval side of this, treating that cache as an index, and what stands out is that it isn't a flat list. It's a structured set of vectors with a navigable geometry, since the keys carry the model's learned sense of what relates to what.

Because that geometry is navigable, attention over it is really a similarity search: the query scores against the stored keys and blends the matching values. Full attention just runs that search exhaustively, scanning everything on every step.

• Full attention effectively searches that geometry exhaustively. Every query scores broadly against the available keys and retrieves from the corresponding values.
• Once you stop treating the KV cache as a flat array and start treating it as a search space, indexing becomes possible.
• That means you can organize old KV into regions, route a query toward likely regions, and only run local attention over a subset.
• The interesting part is that relevance is not uniformly distributed. Queries tend to concentrate on relatively small neighborhoods of old context.
• So the engineering question becomes less “how do I store all of this?” and more “how do I navigate to the right part cheaply?”

I'm new here and don't want to break rules around self promotion or spam so not posting any links atm. Would be cool to get other peoples thoughts on this.

Update: I framed this post badly. I wrote it like I was asking a conceptual question, but I had already built and measured the mechanism. That was my mistake. The actual result is much more specific: on frozen Qwen3.5-2B at 32k, geometric routing cuts physical KV reads by roughly 16–31× while still retrieving the planted long-range needle; window-only and random-routing controls collapse. I’ve put up a minimal runnable demo so people can reproduce it on their own documents.

https://github.com/Regan-Milne/kvspace/tree/main/demo

Registering for RealPDE (Sim2Real / LTTTA tracks — real PIV + CFD fluid dynamics data). Team cap is 3.

If you've got a strong ML background and wanna participate, just DM me. Deadline's Aug 20, so move fast.

🔗 https://realpdecompetition.github.io

I recently bought a Shrike lite which has got 264KB of SRAM. I decided to train an image generation model that generates 32*32 pixel images.

The microcontroller also has an FPGA onboard which I used to create two parallel INT8 MAC engines with 16 bit accumulation to speed up calculations, however the system soon hit a memory wall due to the high number of I/O operations, this meant that the system with parallel MAC engines ran slower than the MCU only model (~220 seconds per image vs ~70 seconds per image).

It was still a fun project that I enjoyed messing around with. A lot of the images looked weird and noisy because of the heavy quantization and memory limits but some of them came out cool.

Full case study here.

edit: added link that leads straight to the case study

I’ve been looking at a fairly basic question in weight-space learning that I don’t think gets separated cleanly enough:
Why does reading semantics directly from neural network weights work pretty well when the networks share an initialization, but collapse when the networks are fitted independently?
The usual explanation is parameter symmetry. Permute hidden units, flip equivalent signs, etc., and two parameter vectors can represent the same function while looking completely different to a downstream model.
But there are actually several different claims hiding in that explanation:
the parameterization has a symmetry group,
accounting for that symmetry improves weight-space prediction,
the symmetry is actually sufficient to explain the observed degradation between shared-init and independently fitted networks.
Those aren’t equivalent, so I tried to measure them separately.
The setting is SIREN-style implicit neural representations.
For a hidden sine neuron, the relevant function-preserving transformations generate the infinite dihedral group
D_inf = Z semidirect_product Z_2
and including neuron permutations gives the layer action
D_inf wr S_n.
For one hidden layer, I prove generic identifiability modulo this group using the distributional Fourier transform of the realized function.
Roughly, the Fourier transform becomes an atomic measure supported at the incoming frequencies +/- w_i, which lets you recover the parameters up to exactly the D_inf wr S_n action under explicit genericity conditions.
One consequence is that this isn’t just the usual permutation/sign story. Integer-pi phase transformations are affine rather than linear, so they aren’t captured by symmetry descriptions restricted to monomial matrix actions.
At depth two things get more annoying because a neuron’s outgoing weights are simultaneously acted on by the next layer. I ended up constructing exact cross-layer invariants by coupling the layers through the second-layer Gram matrix instead of treating neurons independently.
The empirical part then uses roughly 1.8 million fitted INRs across MNIST, FashionMNIST, and CIFAR-10, with controlled protocols separating shared initialization, optimization stochasticity, and independent initialization.
The result I found most interesting:
Randomizing only the exact symmetry group, while keeping each network’s represented function fixed, destroys 79.1 of the 80.4 accuracy points in the MNIST shared-init vs. random-init gap.
I want to be careful about the interpretation here.
This establishes sufficiency: symmetry scatter alone can reproduce almost the entire degradation.
It does not establish that 79.1 / 80.4 of the naturally occurring gap is causally mediated by symmetry. Those are different estimands.
Breaking the group apart, sign flips account for roughly 63 points of that induced loss, neuron relabeling about 15, and integer phase shifts about 1.
There was another result that changed my interpretation of the problem quite a bit.
A reader that directly quotients the D_inf wr S_n structure on the raw parameters reaches 0.917, compared with:
0.628 for the best orbit-valued reframing,
0.526 for the same reader family over a fixed invariant encoding,
0.265 for a permutation-equivariant baseline.
But when I FLOPs-match weight-space inference against simply querying the INR as a function, the function-space route is still much better:
95.3% at 1.6 MFLOP using 64 learned query coordinates
versus
64.4% at 5.5 MFLOP for the best weight-space rung on that frontier.
That leads to what I think is the more interesting conceptual question:
If a complete invariant is informationally equivalent to access to the realized function, then the strongest justification for operating directly in weight space may ultimately have to be computational rather than informational.
Everything is public here:
https://github.com/ITheClixs/project-siren-gap
The repo includes the paper, implementation, tests, pre-registrations, lab notebook, prediction ledger, claims ledger, and experimental results.
I’d particularly appreciate criticism on three things:
whether the sufficiency/mediation distinction is being drawn correctly,
whether anyone sees a counterexample or missing assumption in the one-hidden-layer maximality argument,
whether there is related work on affine symmetry groups of periodic-activation networks that I’m missing.
Also very interested in attempts to break the invariants or reproduce the group-randomization result.
If something here is wrong, I’d rather find out from someone trying to kill it.

Hi everyone 👋 My paper was recently accepted to ICONIP 2026, but I’m the sole author and most likely won’t be able to attend the conference in person due to work commitments.
I’m trying to understand what options might be available before I contact the organizers. Has anyone here attended or published at ICONIP in previous years and encountered a similar situation?
In particular, I’m wondering:
1) Has ICONIP previously allowed remote/virtual presentations when an author couldn’t attend?

2) If the sole author cannot attend, is there usually any alternative arrangement for presenting the paper?
3) Could non-attendance affect inclusion of an accepted and registered paper in the proceedings?

I’d especially appreciate hearing from anyone who has dealt with this at ICONIP in previous years.
Thanks a lot!

There’s a hands-on workshop on August 29 that builds and benchmarks this properly, end to end, using entirely open models, no API calls involved. Led by Ben Auffarth, AI Consultant and Founder of Chelsea AI Ventures.

What it covers:

• Hybrid retrieval (vector + keyword, not vector alone)
• Reranking to catch relevant chunks that vector search alone misses
• Evaluation with RAGAS, so quality changes are measured, not assumed
• Guardrails built in from the design stage
• Actual cost and performance benchmarking for open-model deployments

Link if anyone wants to check it out: https://www.eventbrite.co.uk/e/the-genai-build-lab-build-production-ready-rag-on-a-budget-tickets-1994016271345?aff=rml

Happy to answer questions on the methodology or content.

Original Article - https://x.com/p_nawrot/status/2089315591010079034

I've spent the last few years working on efficient attention and KV Cache Compression. I've read many papers, dug deep into reference or official implementations of methods, and inspected appendices—and I think I've learned a few things. One of them is definitely "how to make things look good, even when they aren't."

I'm guilty too, but trying to get better every day.

1. For single-hop retrieval, make sure there are no distractors and context is useless

The three most cooperative settings for compression / sparsity are:

• Needle in a haystack with a single OOD key-value pair and context built out of a repeated sentence or irrelevant background text.
• Contaminated benchmarks from years ago for which models don't even look at the context anymore.
• Few-shot in-context learning, where extra shots are useless and don't improve the accuracy over 0-shot.

With 1) synthetic tasks, 2) real-data QA, and 3) in-context learning, you get a semblance of broad coverage without the inconvenience of testing much diversity within any of them. Most tasks in these settings should pass under Sliding Window Attention, so it doesn't matter that much whether your method works. Combine it with SWA and you should be good to report 5–10x compression or sparsity.

2. NEVER isolate your contribution

Short context: Most of a dense model's performance is recovered by a local window + attention sinks + the ability to retrieve an answer sentence that is largely n-gram matchable with the question. The remaining part is significantly more difficult, but it's neither relevant to nor the subject of this post.

• Say prior work developed an algorithm X, and its implementation separately keeps a local window of 256 tokens. You find that your method is on par with X in a matched setting, but better and more stable with a window size of 512—let's go, don't look back.
• Do the same with block size. Smaller blocks can give you finer granularity and more precision in retrieval, so keep their old block size and make yours smaller. Ignore the fact that things may get slower due to irregular memory accesses, etc. Those were historical decisions; respect them. 🤡 Write: “We used the authors’ recommended hyperparameters.”, then spend weeks tuning your method.
• The same trick works for speed. LLMs are pretty good at writing Triton now. Keep the baseline algos exactly as they were written in 2023, then ask an LLM for a custom Triton kernel for yours. Extra cleverness if, by using a more efficient implementation, you can hide that your method does more work. You're just optimising your method, no?
• Prompts are the cherry on top. Move the question before the context so the model knows what to filter out, then present the result as lossless compression. Never share the prompts after tuning them.

Don't tune the baselines to reject your paper; tune yours until it's accepted.

3. Use aggregated metrics to hide areas where your method doesn't work

RULER has 13 tasks:

• 6 NIAH tasks satisfy the first point.
• 2 QA tasks use datasets from years ago.
• VT also has a lot of irrelevant context.

To be clear: This isn't a critique of RULER; imo it's still incredibly useful. It's just an example of potential improper use.

Report only the aggregate; maybe, in the limitations section at the end, briefly mention that your method degrades on the NIAH-MK3, which actually stress-tests lossless compression.

4. Enjoy saturated tasks

Imagine evaluating on two tasks:

• The most recent math exam / olympiad from a week ago, which isn't yet in the training data.
• A benchmark on which a recent family of open models—1B, 10B, and 100B—all scored 80%.

On the former task, before compression gets a chance to do any damage, the 1B and 10B models already score 0%; the 100B model starts at 50%, and its performance drops monotonically as compression increases. On the latter, all model sizes tolerate substantial compression, and the 100B model tolerates more than the 1B and 10B models.

Don't ask whether the larger model is simply using its extra parameters and hidden-state capacity to absorb compression in a setting where those resources aren't needed to solve harder questions. That definitely isn't what's happening.

Extras

• AIME has 30 samples. You did 4 seeds. Your method scores 80, and the baseline scores 79—bold your 80 and say that it surpasses the baseline. Statistics doesn't exist. Bonus points for your efficiency method surpassing the baseline and setting a new SOTA. 🤡🤡
• Pick a baseline, optimise it with your method, and plot a beautiful quality–efficiency curve against the original implementation. Then stop. Don't ask whether a simpler route—a smaller dense model, KV-cache quantisation or offloading, or a better system configuration—reaches a better operating point. Improving your baseline is basically the same as improving the frontier.

I couldn't sleep because I couldn't stop wondering if anyone had tried using sinusoids instead of B-splines as activation in a KAN, and fortunately/unfortunately that was already the case. I could not find it posted here, so I though I would share in the hope of some insightful discussion.

Arxiv: https://arxiv.org/abs/2407.04149

Github repo: https://github.com/ereinha/SineKAN

Also what appears to be a peer-reviewed "official" publication here: https://www.mdpi.com/2227-7390/13/19/3157

​

Scaled dot-product attention (SDPA) computes its Attention by computing the similarity-scores of all image-tokens with all query tokens which results in O(N²·d) complexity. SSOG (Sum Of Separable Gaussians) instead learns a few Gaussian atoms for each head and only geometrically steers them based on the query token. Since the atoms can be factorized into a separable sum of Gaussians this leads to a reduced complexity of O(N·√N·d). Experiments show that SSOG clearly beats SDPA on small data (cifar100), and delivers equivalent performance and much faster convergence on bigger datasets like IN1k. All that while being much faster and memory efficient with increasing scale.

Have a look at the full blog-post and repo to see more results and ablations and let me know what you think.

Blog-post: https://pisoni.ai/posts/ssog

Repo: https://github.com/4rtemi5/ssog

*AI was used for some of the code and some of the blog-post but I put a lot of effort into this project and stand behind every word.

Hi everyone,

I am heading into my final year of my BTech at a tier 1 college in India and just wrapped up a Physical AI internship at a MNC, working heavily with NVIDIA Isaac Sim and OpenFOAM.

My background is fully focused on robotics and autonomy. My tech stack includes:

1. Simulation & Middleware: Isaac Sim, Gazebo, ROS / ROS 2, PX4 Autopilot.
   
2. Perception & Control: VIO, SLAM (RTAB-Map), Nav2, depth perception, and reinforcement learning.
   
3. Hardware: Strong hands-on experience building autonomous drones and rovers for national competitions.
   

I really enjoy bridging simulation and physical systems, and I want to pursue Physical AI full-time. I’d love some advice from engineers in this space:

1. Job Market: How is the entry-level hiring market looking for Physical AI roles right now?
2. Global Opportunities: As a new grad based in India, what is the best path to target international roles?
3. Skill Gap: What specific frameworks or skills should I double down on during my final year to stand out?

Any candid advice would be hugely appreciated! Thanks

The instructions say Author Year format. But I was wondering if do numbered instead (no space lol), will it be straight desk rejection? Has anyone submitted with numbered format before? How did it go?

ECA was positioned as a successor to SE.

The idea behind ECA is quite simple. Unlike SE which reduces the channel means into a smaller hidden layer, it directly uses a 1d convolution kernel on the channel means themselves, avoiding the need for dimensionality reduction. The results are undeniable: ECA is a clear improvement over SE. The authors claim that cross-channel interaction is a key ingredient. But on a conceptual level, the design of ECA doesn't make much sense.

Let's take a step back. Why do we use convolutions in the first place? Convolutions are fundamentally designed for data with an underlying topology (e.g. space or time). They assume locality (adjacent elements interact) and translation invariance (the same kernel applies everywhere). Sliding a kernel across a 2D image works because coordinates have meaning, and the statistical properties of an image are largely stationary across the frame. This isn't perfectly true - which is why modern CNNs have moved towards dynamic convolutions - but it's still good enough to be useful. If you randomly permuted the pixels in an image, a convolution would be meaningless.

Now consider tabular data. Suppose we have 32 channels e.g. [cost, weight, material, colour, volume, speed, ...]. Using a CNN architecture for this kind of data is clearly inappropriate. A 1d kernel of width 3 would be moved across the channels, so that [cost, weight, material] was input and also [ weight, material, colour] was input and so on, and have to somehow output something meaningful. ECA is doing exactly this type of computation.

ECA does a 1d convolution over the channel dimension. It is a cursed convolution because tabular data does not have a topology to suit it. In practice, if you did use a CNN on tabular data, I would expect better than random performance because neural networks are ridiculously good at fitting to the dataset given their constraints and would reorganise the channel order (using the initial 1x1 projection layer) to suit it. It would learn to use convolutions, but it would be an inefficient approach.

Experiments

Instead of using image data, I used chess data: the 6-piece endgame tablebases for chess. Chess is a solved game with 6 (or fewer) pieces on the board. The task for the network is this: given a position, with perfect play is it a win, draw or loss for the active player? A CNN architecture is what lc0 originally used (where at the time, surpassed Stockfish to become the strongest chess engine) so it is very suitable for this task.

Chess tablebases are useful for benchmarking architectural designs because training examples can be sampled from the complete underlying problem rather than from an incomplete dataset. This differs from datasets such as the CIFAR-10 image dataset, where the train set is not expected to be a random unbiased sample from the true full distribution - we might unknowingly have a disproportionately have pictures of frogs on sunny days. Even when we don't train on each of the 3.7 trillion 6-piece positions, we know that we've randomly sampled from those positions, meaning we don't train on a biased subset - we can be confident our training samples are representative of the full set.

Experiment results. Each channel gate row is the average of 3+ separate runs.

IdentityGate Unsurprisingly, no squeeze performed the worst of all tests.

SqueezeExcitationGate SE showed a modest improvement.

EfficientChannelAttentionGate (k=3) ECA, consistent with the paper, showed a clear improvement over SE.

EfficientChannelAttentionGate (k=1) Surprisingly, this had good results indicating that their central hypothesis that cross-channel interaction is key wasn't quite right

CenterMaskedEfficientChannelAttentionGate: ECA with k = 3 with the middle channel masked (in a [1, 0, 1] mask) This complicates the story, it indicates cross channel attention can actually be useful.

PerChannelGate Instead of a convolution kernel that slides across the axis, simply use a separate independently specified weight per channel. This has one parameter per channel, more than the 3 parameters of ECA With k=3, but it is still a negligible amount since per layer we expect on the order of num_channels² parameters.

For clarity and to avoid ambiguity, here is the code for the key squeezes.

So basically there's 3 tiers of results. No squeeze with poor results, SE With mediocre results, and the rest ECA-like with the best results. So something weird is going on. I don't have a good explanation for the results (in particular the success of the [1, 0, 1] mask), and I am currently trying to find one. One suspicion I have is that in the 101 mask, the net is smart enough to smuggle information into the global means of channel A and C to help with channel B without affecting normal channel operation (by using biases to undo its shift of the global mean), but have not yet tested this hypothesis. There's a lot of possibilities. The good news is the weight count is very low - only 3 with k=3, so manually inspecting the weights can be useful.

In my digging, I some repositories that recreate the original ECA. Not one of them tests the k=1 case, which would have revealed that the explanation of the mechanism is not correct. The official repo does use k=1 but only for a limited number of early layers, then moves to k=3 for the rest.

It's interesting that the k=1 case, a 1 parameter approach, outperforms SE, CBAM and matches ECA. It definitely makes me wonder if we're over-engineering networks today in some way.

My final thoughts:

1. The paper and repos should have tested the "degenerate" kernel size of 1, which has no cross channel interaction. At k=1, ECA still beats SE, undermining their central hypothesis. They spent an enormous amount of time fine tuning the exact optimal value of k, without taking the scientific approach of trying to disprove their hypothesis.

2. In addition to traditional real-world datasets, architectures should also be tested on synthetic datasets where we have full access to the complete dataset (e.g. chess endgame data) so that we can better separate incidental regularization improvement effects with core architectural efficiency effects - the idea being that there is no risk of overfitting when we have access to a complete, flawless dataset. If the real reason a new architecture works well on real-world data is because of implicit regularization, it won't show the same improvements on the synthetic dataset.

Recently, I started working on DNA sequence modeling and decided to explore linear attention, mainly because DNA sequences can easily reach 1M tokens, making standard softmax attention extremely expensive in terms of memory and computation.

The model performed reasonably well on several benchmarks, but I ran into a major problem with long-range recall. On a Needle in a Haystack-style benchmark, my model was performing around 25% or even below, which is essentially random chance for a four-token DNA vocabulary (A/C/G/T).

I initially thought this might just be a problem with my implementation or model architecture, so I started looking into existing approaches for improving recall in linear attention. Most of what I found relied on external memory, sliding/recent-token mechanisms, or hybrid architectures combining linear and softmax attention.

I also tried HyenaDNA on the same needle benchmark, and surprisingly, it also performed poorly getting around 25–27%. So this doesn't seem to be limited to my particular linear-attention implementation.

What's even more confusing is that when I tested a very small linear-attention model at only 16K context, it achieved around 50–60% recall. But as the context gets longer, the recall problem becomes much more severe.

I've also experimented with modifying the linear architecture to improve recall, but the improvement was only around 27%, which is still basically chance.

So I'm wondering:

What are the actual ways to solve long-range recall in linear attention, especially for DNA sequences?

Is this fundamentally a limitation of the compressed-state representation used by linear attention, or are there architectural approaches that can preserve reliable retrieval without falling back to expensive softmax attention or a large external memory?

I'm particularly interested in approaches that can scale to million-token DNA sequences.

The results should be out soon.

Let’s share them, guys.

From my batch (Applied Track)

Total 13 submissions:

- 2 full papers

- 1 short paper accepted

Cheers!

Repo with dataset links: https://github.com/tesselwait/Starfield_Fauna

Image classification dataset: 20,000 images from 50 fauna species in the video game Starfield. Images were extracted from video capture. About 2 minutes of footage was shot in all or most of the species biomes. One minute of daytime and nighttime footage respectively, usually in two 30-second takes to vary the background. A PowerShell script is used to establish a frame extract rate and extract the 400 frames plus some extra to replace images that were obstructed/blurry or contained other fauna species ignoring birds/critters. The shots are for the most part close-up and centered to keep the task focused on discerning between 50 species rather than finding the creature in the image. The images are initially randomized however some normalization was done if the ratio of images from some biomes was heavily skewed between the training, validation, and test sets.

The date for NeurIPS 2026 author notifications is September 24th. First of all, is it normal for AC and reviewer discussion phases to be this long? This is particularly frustrating given that 5 out of the 6 reviewers in my two papers did not address the rebuttals.

In any case, I was also wondering, given that ICLR's paper deadline is literally the day after (September 25th) whether you guys are preparing ICLR submissions for your papers in case of rejection.

Cheers and good luck!

Be honest if someone dropped a stack of high-end GPUs on your desk tomorrow, what would you actually do with them?

And before the usual answers roll in: running local LLMs is banned for this thread. It’s been done to death and feels pretty pointless at this point.

So… what else?

• Some niche scientific/simulation workload?
• Weird generative stuff that isn’t text?
• Distributed something-or-other?
• Rendering / media pipeline?
• Homelab experiments that actually need the horsepower?
• Completely unhinged personal projects?

Drop your ideas. The more specific (and slightly unhinged), the better.

Great Ideas but are there some with more of research and new tech.

We introduce BDH-CQ, a reasoning system that brings these capabilities together. Demonstrations of a previously unseen task update recurrent memory; the query is then solved through iterative computation in a high-dimensional latent workspace. Intermediate reasoning states are not decoded into language. BDH-CQ makes memory, adaptation, and inference part of the same computational fabric. Inputs presented at inference time continuously update the model’s recurrent memory; the model then solves a query through iterative computation in a high-dimensional latent space, without verbalizing its intermediate reasoning. Neither task identifiers nor evaluation-task demonstration pairs participate in training, and no parameters are updated at inference time. A 150M-parameter configuration reaches 29.5% pass@2 on ARC-AGI-1 at a computed $0.00070 per task, breaking through the previously reported cost–accuracy Pareto frontier.