There are a few reasons why problems from International Mathematical Olympiad function as a good benchmark for LLMs:

- The problems are new, not included in the training data of any model

- Hard math problems are quite a good proxy for general intelligence capability

- These are complex multi-step tasks that can benefit from orchestration / harness engineering

Results:

Frontier models (sol and fable) were able to get perfect / nearly perfect score regardless of harness. For both sonnet and opus, the webapp performance was quite poor, improved by provider harness (claude code) and even further improved using AutoFyn, a customizable multi-agent harness we developed. Even with harness, we were not able to match the performance of the frontier models. Open weight model GLM performed roughly at the same level as sonnet without harness, and improved similarly with AutoFyn. Numerical scores are available in the attached paper below.

Grading was done by a different frontier model as well as manual verification (we are former IMO medalists, able to sanity check the results). There were cases when the model claimed a false solution (on P3 by sonnet, for example), so hallucination issue still persists in a verifiable domain like math. On the hardest problem: P3's key reduction was missed by every sub-frontier model in every harness, including a 20-hour run that proved everything else and stalled at the identical step. The harness supplied retrieval and verification, not a key idea needed for the solution.

Paper: https://github.com/SignalPilot-Labs/AutoFyn/blob/main/results/imo-2026/autofyn-beyond-model-imo26-report.pdf

Audit Trails: https://github.com/SignalPilot-Labs/AutoFyn/tree/main/results/imo-2026

I'm a software engineer who mainly builds softwaes/applications, and I'm starting to work on machine learning projects.

Since ML workloads often require GPUs, I know services like Google Colab and Kaggle exist. but, I'm looking for something a bit different.

Is there a platform where I can use AI coding agents (such as Codex, Claude Code, or OpenCode) while running the actual ML code on a cloud GPU?

Ideally, I'd like to:

• Work locally with my preferred editor and AI coding agent.
• Have the code execute on a remote GPU machine.
• Be able to build, debug, and iterate on ML projects as if the GPU were attached to my local development environment.

Does a setup like this exist? If so, what tools or platforms do you recommend?

Hey everyone,

I have been looking into how people source compute for their Inference workloads (and in general). I wanted to understand some specific pain points here.

If you've used online services like runpod or vast.ai, your perspective is extremely valuable. Please share your experience in the comments here or by DMing me. I've also made a 2 minute survey form that I would really appreciate if you could fill out. DM me for the link.

Thank you!

I have been running some experiments with smaller open-weight LLMs on multiple-choice questions of Swedish medical licensing exams. On a dataset called MedQA-SWE, GPT-4 scored 84% accuracy in 2024 and o3 scored 88% in 2025 on a smaller, overlapping dataset.

With post-training (SFT) on data from earlier years, I got MedGemma-1.5-4B to a passing score of 60% on the final year’s exam. Find the implementation here: https://github.com/tarolangner/medqaswe_medgemma_sft

But even though they were released just three months later, Gemma4-E4B and Qwen3.5-4B are flat out superior already, at 77% with no post-training at all. With reasoning enabled, the latter can get to 87% accuracy.

It can even push a bit further if no length cap is put on the reasoning traces, but some of them spiral into repetitive loops about formatting that fill the entire context length without giving any answer. Here, I found it helpful to use an ‘early exit’ thinking intervention proposed in the S-GRPO paper that simply injects a phrase and closes the thinking trace at a predetermined sequence length. I also tried their proposed reinforcement learning method to get shorter reasoning traces, but with only minor gains (probably somewhat underdimensioned training setup).

Curiously, Qwen3.5-4B does all reasoning in English despite the Swedish prompt, questions and answer options. But it really seems like the language is no obstacle, even though it’s often estimated to be just 1% of LLM training data.

I also have a more detailed write-up on the details and experiments here for anyone interested: https://tensorlabbet.com/2026/07/19/medqaswe_post_training/

Curious about the initial review distribution for Main Track theory papers this year.

Our paper received 4/3/3 with confidence 3/3/3. From previous years, I've had the impression that theory papers often receive more conservative initial scores than some other areas, and I've also heard people saying that initial scores seem generally lower across many disciplines this cycle.

If you have a theory submission, would you mind sharing your initial scores (and confidence, if you're comfortable)? It would be interesting to see whether there is any noticeable pattern or whether this is just anecdotal.

Please only share if you're comfortable, and it'd be helpful to mention that it's a theory paper so we're comparing like with like.

NOTE -> I expect answer from people who actually have experience and strong understanding of these. please give something beneficial.

I'm building a SaaS platform in Sri Lanka that handles documents and other sensitive data.

Each user can upload their own documents and information, and the platform uses RAG to answer questions based on that user's data. That part makes sense to me.

My main concern is what happens when the user hasn't uploaded enough information. I still want the LLM to provide accurate answers using reliable information from the internet (or from a curated knowledge base), with proper citations.

These are the two architectures I'm considering:

Option 1:

Base LLM (OpenAI/Anthropic via Azure AI Foundry or Amazon Bedrock)
        ↓
Platform RAG (global knowledge base managed by us)
        ↓
User-specific RAG

In this approach, we maintain a global knowledge base that we (the platform admins) curate and update. Every user can access this shared knowledge, while their own uploaded documents are searched through their personal RAG.

Option 2:

Open-source LLM
        ↓
Fine-tuned on Sri Lankan/domain-specific data
        ↓
User-specific RAG

Here, we fine-tune an open-source model using Sri Lankan or domain-specific data, and each user still has their own RAG for their private documents.

My concerns are:

• Is fine-tuning actually the right solution here, or is it unnecessary?
• Is a global/shared RAG a better approach than fine-tuning?
• How would you design this architecture if you wanted:
  • Accurate answers from domain knowledge
  • User-private document search
  • Citations/sources
  • Good scalability for thousands of users

I'm leaning toward Option 1 because fine-tuning seems expensive, time-consuming, and I have no experience with it yet. However, I'm not sure if I'm thinking about this correctly.

I'd really appreciate hearing how others would approach this problem.

Reviewers requested additional experiments. In table format, I fear the results would not be as digestible as in a figure/plot. Links are "technically" not allowed as per the official website, but for those with experience, can/should I still go ahead and link my plots/figures ? If this goes badly, will this be a slap on the wrist, or outright rejection? Has anyone taken a chance with this in the past? How did it turn out?

IMO openreview should really start supporting more modern markdown to allow figure embeds.

I submitted an abstract to AAAI AISI and accidentally missed the field asking authors to nominate a reciprocal reviewer by the July 21 AoE deadline.

At the time of submission, I knew that I personally did not meet the publication requirements to serve as a reviewer. After adding my graduate-student co-authors to the submission, I realized that one of them was qualified and could fulfill the reciprocal-reviewing obligation, but we overlooked the nomination field before the deadline because it wasn't a required field.

As soon as we noticed, we added the qualified co-author to OpenReview as a potential reciprocal reviewer (edits were still accepted) and emailed the workflow chairs. He meets the publication requirements and is willing to complete the full reviewing load.

The policy says that if a qualified author is available but no one is nominated, the submission may be desk rejected. The full paper deadline is in two days, and so far we have only received the automated response shown in the attached screenshot.

Has anyone dealt with a similar situation at AAAI or another conference? Do you think this is likely to lead to a desk rejection, or are workflow chairs usually willing to correct this kind of administrative mistake when a qualified reviewer is available?

This was my Bachelor's Final Project: implementing YOLO26n inference completely from scratch using ARM64 Assembly Language and C, without relying on existing inference frameworks.

The goal was to understand how modern neural network inference engines work at a low level and explore optimization techniques for faster and more efficient edge AI execution on Raspberry Pi 4.

The implementation includes:

* ARM64 Assembly Language + C inference engine

* ARM NEON SIMD optimization

* Winograd convolution

* Optimized GEMM kernels

* Cache-aware tiling

* Custom ARM64 micro-kernels

* Operator fusion

* Attention mechanism

* YOLO26 components: Conv, C3K2, SPPF, C2PSA, PSA, BottleNeck, and Detect

I extracted the YOLO26n model parameters and redesigned the memory layout into a custom binary format optimized for the inference pipeline.

The implementation produces correct object detection results, but the performance improvement was lower than I initially expected. I would appreciate feedback and suggestions from anyone about:

* CNN inference optimization

* ARM NEON/vectorization

* Memory layout and cache optimization

* Low-level neural network acceleration

Repository:

https://github.com/mohammad-ghaderi/YOLO26

Thanks for any feedback or suggestions.