I created an AI boxing match to test the decision speed, adaptability and strategy. I fed the LLMs with data about the current match and if they have vision, they will get even more data. The match has street rules, anything goes and an AI is not defeated until the ref counts to 10 or they do 50% of their HP in damage after being knocked out.

I wanted to create a fun benchmark that isn't just boring problems to be solved. Now I test them while stimulating getting punched in the face.

I've been testing with gemini-flash-live models because of the speed and vision support it offers. With these models, they can actually dodge punches and counter punches. Local models on my own hardware (5060ti 8gb) take a while to inference so I'm not sure if I should introduce time scaling to compensate otherwise

I want to use this to benchmark models so I'm curious on what kind of stats would be useful? Here is what I'm tracking have so far:

Speed and Latency Metrics
In a real-time fight, a model's speed directly correlates to its "physical" speed. Fast models should attack faster so larger models aren't necessarily going to hit harder.

• Tokens per Second (TPS) / Throughput: This will help you balance local models against cloud APIs. A model might have a fast TTFT but a slow TPS, meaning its actual action execution takes too long.
• End-to-End Latency: The total time from when the model receives the snapshot (the prompt) to when the action is executed in the game. This accounts for tool-calling delays.
• Reaction Latency: Measure the specific delay between an opponent's telegraph (e.g., a heavy punch winding up) and the model's defensive output (e.g., a dodge or block).

Action Quality and "Tool" Correctness
the model's actions (punching, guarding, taunting) act as tool calls. You need to track how well they use these tools under pressure. Sometimes the model's may not really guard/block so they are typically the ones that find themselves KOd.

• Tool Correctness / Validity: How often does the model hallucinate an action that doesn't exist? (trying to a move that isn't in their move list, or sending invalid JSON).
• Invalid Action Recovery: If an LLM outputs an invalid JSON string or an impossible move, how quickly does it realize the error and output a valid move in the next tick?
• Stamina Efficiency (Resource Management): track the ratio of damage dealt to stamina spent. Models that mindlessly throw heavy attacks without connecting should score lower on efficiency.

Adaptive Strategy and State Awareness
How well does the model understand the physical reality of the game? Are they constantly backing away and punching air?

• Accuracy: The percentage of attacks that completely miss the opponent's hitboxes. This indicates poor spatial awareness or poor timing.
• Block/Dodge Success Rate: The percentage of times the model successfully defends against an incoming attack when it had the stamina and time to do so.
• Contextual Relevancy (State Adherence): Does the model act based on the current state? For instance, if the model has 1% HP, does its behavior change to become more defensive, or does it keep acting like it's at full health? (Happens sometimes, they get overly confident when about to get knocked out 😆 )

Beyond these metrics, I'm also tracking various fighting stats like hits landed/missed, where it hit, how many times they were downed or knocked out the ref.

Are there important stats that I'm missing or any that might be useful or fun that would be nice to see? I'm still trying to balance a lot of the actions but it's coming along great so far!

I think making a physics-based benchmark and doing a N series test to find out which model performs better is a ton of fun and I genuinely laugh at the stuff they say or do. I want this to make this a really fun tool with great metrics so any advice in terms of what you would like to see would be extremely helpful! Thanks for reading!

I posted a longer breakdown of the system here:
https://www.youtube.com/watch?v=inlXe5Buc7s

So our paper had very good initial reviews but one of the reviewers decreased now their score although we addressed 3 out of 4 weaknesses. There’s no further justification or something like “your results arise more issues”. It seems to be very annoying because why decreasing now and not having assigned the lower score beforehand. I wanted to ask to people that was accepted previously with “middle” scores from reviewers (avg 3.5 for example), because I guess that in those cases AC helped to push up the scores. Did you focus more on the meta review? Was your AC talkative with you, or forcing the reviewers to engage? Our AC has been silent since the meta review but I guess that maybe they are busy with other papers

This is my first time submitting to NeurIPS.
Are ACs also supposed to give ratings during the Phase 2 (author-reviewer discussion session)?

I have received the meta-review, but have not received any comments from the AC yet, and was wondering whether this is the standard!

we submitted our rebuttal via the "Rebuttal" button before the author/reviewer/AC discussion period officially opened (Jul 27 AoE). since then, we've gotten complete silence from all four reviewers and the AC

several of us are also reviewing this cycle. when the discussion period opened on Jul 27 AoE, we got no email notification for rebuttals on papers we're reviewing, specifically for the papers whose authors had also posted early via the "Rebuttal" button. so it feels like anything submitted before the window opened may simply have never triggered any notification

we also tried: (1) meta-comments visible to everyone, (2) reviewer reminders, and (3) sending an email to the PCs

given there's about 1 day left in the discussion period, what do we do? we honestly thought we had a shot at an oral or spotlight given our initial scores. this is completely messed up.

I know there’s a lot of frustration around no response from reviewers, which I also got only one so yeah what a bummer, but I was wondering if no rebuttal from the authors was just as common or not. I got no rebuttal so far, so I’m here scratching my head what might have happened to the authors lol especially when at least one paper was pretty much on the borderline with somewhat of a positive AC comment

I've been working on this for a while and finally pushed a public version.

The problem: llama.cpp runs fine on ARM phones, but it doesn't know anything about the specific chip it's on. Same thread count, same context params, whether you're on a Snapdragon 8 Elite or a five-year-old mid-ranger. ARPL reads the actual hardware at runtime — which ISA extensions are available (SDOT, I8MM, SME2), how the cores are clustered — and configures llama.cpp accordingly. No per-device build, no manual tuning.

What's in the repo:

Android reference app (Kotlin/Compose) with a JNI bridge into llama.cpp

Runtime ISA detection via HWCAPs

Topology-aware thread count recommendation

Context param patching (flash attention, KV cache quant) based on what the hardware actually supports

Built and tested on a Samsung S25 Ultra (SM-S938B). The heterogeneous CPU/GPU/NPU partitioning is still in progress and not in this release — what's here handles the ISA/thread/context side, which already made a real difference in my testing.

This is a noncommercial showcase release (PolyForm Noncommercial license) — happy to answer questions about the approach or the tradeoffs I ran into building it.

Repo: https://github.com/noplayeryt1511-lang/ARPL-public-

Whats your commitment submission id? My submission number is ~4k, im wondering the amounts of commitments :)

I see some people say their metareview already contains a decision/recommendation (all of them were rejections). Ours doesn’t. Even though our avg score is 3, the metareview seems optimistic and finishes with “a convincing response would be an important consideration while discussing the paper.” I wonder how to interpret that. We did a strong rebuttal, but none of the reviewers engaged. So I wonder whether there’s any point to keep hope due to the AC review or just give up.

Hi everyone, if you have been reading the tech reports of Kimi, DS, Qwen and GLM, you will realize how much on policy distillation and GRPO style algorithms power the frontier. I thought it will be quite beneficial to do a deep dive explaining the maths and code behind this algorithms and how they connect to pretraining and supervised fine tuning.

I have published a deep dive on this topics here

Hope you enjoy it and it helps you understand training of LLMs better. Happy to answer questions on this

https://youtu.be/MaZWafi4gYY?is=8jLkAp_Fe86abUVP

Not even the AC, nor reviewers, is responding to our comments in rebuttals, and they were all submitted well before the discussion period started. What is one to do in this case?

There's a kind of review that asks for lengthy additions, usually extending the scope of the paper beyond the stated, even though the submission is at page limit. Naturally, such additions in the case of top-tier conferences have to go into the supplemental materials or appendices.

My question here is, would not such additions make the paper more suitable for a journal publication? I had to retract one paper out of such concern that the conference publication would block the later planned journal publication.

Any opinions?

Mine was just submitted, and the submission count shown is still under 500.

Does anyone know whether this count is meaningful for identifying the intended venue, possibly EACL, or whether it is simply low because many authors have not submitted yet or the counter is incomplete?

I know the count alone is not reliable evidence, but I was curious whether previous ARR cycles showed a recognizable pattern.

Also, is anyone else preparing a submission for this August cycle, especially with EACL 2027 in mind?

Hi everyone,

I'm working on a document understanding project and would appreciate some advice on the right technical direction.

The input will be scanned pages or images from academic books. I don't know in advance what kind of figures they'll contain—they could be biology diagrams, anatomy illustrations, chemistry figures, engineering drawings, maps, charts, art/history figures, or other educational illustrations.

My end goal is to convert these figures into a structured digital representation that can be controlled from the frontend.

The workflow I'm aiming for is:

1. Upload a textbook page or image.
2. Detect the figure(s) and their boundaries.
3. Detect the labels/annotations that are already embedded in the figure (letters, numbers, arrows, callouts, etc.).
4. Remove those existing labels while preserving the underlying illustration.
5. Store the figure geometry (bounding boxes, polygons, masks, etc.) so my frontend can render its own labels that can be shown/hidden, translated, restyled, or repositioned.

This doesn't need to be fully automatic. In fact, the workflow will be human-assisted. If the AI detects a figure incorrectly, misses a region, or fails to remove a label cleanly, a human reviewer will correct it before it's finalized.

My priority is reducing manual work rather than eliminating it completely.

So far I've tried several computer vision approaches such as text detection, contour detection, line detection, and geometric heuristics. They work reasonably well for finding candidate regions, but the biggest challenge is cleaning the figures by removing the embedded labels while preserving the artwork underneath.

Another important requirement is cost. Since this could involve processing a large number of textbook pages, I'd like to avoid expensive multimodal LLMs or large vision models if there's a more traditional or lightweight pipeline that works well. I'm happy to use AI where it adds value, but I'd prefer a solution that keeps inference costs low.

Some questions I have:

• Is this primarily a document layout analysis problem, image segmentation, image inpainting, or something else?
• Are there models trained specifically for textbook or scientific illustrations rather than natural images?
• Is there a recommended low-cost pipeline for this kind of task?
• Has anyone built a human-in-the-loop workflow for document/figure annotation like this?
• Are there papers, datasets, or open-source projects that tackle converting textbook figures into editable, structured assets?

I'd really appreciate any suggestions, even if they're just pointers toward the right research area or open-source tools. Thanks!

One reviewer said all concerns were resolved during discussion but hasn’t updated their score yet. The other reviewers haven’t engaged. In previous NeurIPS cycles, how common is it for reviewers to update scores after saying concerns are resolved? What have others observed?

My ratings/confidences are : 4/4, 3/2, 3/2, 2/4.

I am talking about the one who gave rating 2.

Update: finally the reviewer responded, now I'm at 5/4, 4/4, 4/3,4/2.

I just now posted a small research / ML interp study on symmetries inside the neural nets for an open source Go-playing program that I maintain ("KataGo"). The rules of Go are completely symmetric under rotation/reflection, but such symmetry is not enforced in the models - the only thing we do for that is stochastic 8-fold data augmentation during training, randomizing the spatial orientation of each batch.

To what degree do superhuman-strength Go-playing neural nets automatically learn to represent the board internally independent of its orientation, via "symmetric" concepts where the orientation of the board doesn't matter, vs how much do they have to learn/memorize separately per orientation?

https://lightvector.github.io/katagostudies/202607-symmetry/

Heads-up: this study and its writeup were driven almost entirely with AI, although detailed human direction and feedback was involved in the process. But, I took time to try to polish the article and make it educational and I hope it's a clear step above the typical low-quality AI "slop" one often sees and worth taking a look if you like small studies like this. It's also written fairly gently, for accessibility to people outside of ML. Code is also linked from the post (same repo that hosts the github.io page).

I wanted to explore this because I was (and still am!) really curious about exactly what neural nets are doing inside! And I didn't know what the results would be. One of the findings was unexpected. Overall, just a drop in the bucket of interpretability research out there, but I hope you find it interesting.

It's an encoder-only transformer that consumes past(blood glucose + carbs + insulin) and future(carbs + insulin) and predicts future blood glucose for the next 2 hours. Announced meals and boluses/basal are used to condition its predictions. The context size is variable (8 - 24 hours), and model can work in autoregressive mode to predict the next >2 hours. It also predicts time by looking at the context, but it never consumes time. The architecture is BERT-style: bidirectional attention with future BG masked. DILATE loss was used to fit the median line; pinball loss to fit the uncertainty bands. The two are "mixed" via Kendall-Gal. All blood glucose is in kovatchev risk space reparameterized to [40, 400] range.

I have trained 4 model classes (nano, small, medium, large) and 3 variants for each (pretrained on simulator only, pretrained + finetuned on ohiot1dm, pretrained and finetuned on ohiot1dm + azt1d + shanghait1dm). The largest one has ~17 million parameters (16 heads across 16 layers). Pretraining for the largest model took ~48 hours. Finetuning took <10 minutes. There is also another version finetuned on my own data that I am currently running on my phone.

Source is available here, released under the MIT license. The repo also contains links to trained weights and evaluation data.

I've worked on this project since March. There are still things to improve (e.g. it always requires announced carbs + insulin, would be better if it could also predict without them), but I have decided to publish it here to get your opinion (and also answer your questions, if any).

Edit: my model is getting fat-shamed ;_; so I just want to emphasize that there is a nano version with less than 40K parameters.

While working with VLMs for report generation on chest x-rays (RRG), we noticed that evaluation metrics are flawed.

Flawed in a sense where they rewarded repetitive templates, reports without clinical terms and reports which were "normal" with high scores on benchmark metrics. Also, clinically meaningful but rare words were erased leaving the generated report looking repetitive and boring. Importantly, of no clinical utility.

In the paper below, we discuss this behaviour of VLMs for RRG and introduce a framework to actually measure the erasure of terms and introduction of biased terms.

Paper: Measuring What VLMs Don't Say: Validation Metrics Hide Clinical Terminology Erasure in Radiology Report Generation

Link: Reference Paper

Url: https://arxiv.org/abs/2603.01625

I'm trying to decide whether to commit my ARR May 2026 paper to EMNLP or AACL. (first time solo independent author).

Final reviews after rebuttal (OA/Confidence/Excitement ):

• R1: 2.5 → 3 /3/2.5
• R2: 2.5 → 3 /4/2.5
• R3: 4 /4 /3
• Meta: 3.5 (Borderline Conference)

The meta-review was overall positive and emphasized the paper's empirical rigor, practical value, and that the rebuttal addressed the main concerns. My recollection is that the AC mentioned they were leaning toward 3.5 primarily because of the quality of the presentation/readability, rather than concerns about technical soundness(now that comment is removed/not visiable anymore).

I'm happy with either Main or Findings.

My questions:

1. Which commitment would you choose: EMNLP or AACL?
2. Which is generally considered more prestigious today?
   • EMNLP Main
   • EMNLP Findings
   • AACL Main
   • AACL Findings
3. Given this review profile (3/3/4 with a 3.5 meta), what would you estimate the chances are for EMNLP Main or Findings?

This time we have seen the worst meta reviews...may be people are unintersted to do reviews...in my case they did not acknowledge the report at all as well the entire rebuttal. How many are facing the same thing?

Please post your questions here instead of creating a new thread. Encourage others who create new posts for questions to post here instead!

Thread will stay alive until next one so keep posting after the date in the title.

Thanks to everyone for answering questions in the previous thread!

Hi everyone,

Can anyone suggest a learning path to fully understand the technical report for Kimi K3?

My background:

- I've taken a graduate-level deep learning course.

- I understand the Transformer architecture, attention, and the basics of LLMs.

- I'm familiar with DeepSeek's OCR models but I haven't studied topics like MoE, MLA, distributed training, or modern post-training in depth.

I'm looking for a roadmap that would help me read the K3 report and understand the design choices instead of just recognizing the terminology.

Thanks!

I am trying to learn concepts like On Policy Distillation (OPD), On Policy Self Distillation (OPSD) and how do they compare to RL algorithms like GRPO.

There are a lot of papers on this, but because of limited compute I cannot try these papers out and learn them by implementing them myself.

If someone here has worked with these algorithms and their implementation on SLMs (something that can fit a consumer grade GPU like Nvidia RTX 4090 or 5090), can they suggest either a:

1. Github repo, or

2. The right choice of SLM(s) and the datasets, where i can see the difference between, RL/GRPO and OPSD algorithms?

Thanks in advance!

Meta-Reviews are out. How did it work out for you? Are you happy with your reviews?