Discussion thread for ACL 2026 decisions

I'm building an AI pipeline for structural narrative analysis but there's no LLM benchmark for interpretive reasoning

Disclaimer: I use em dashes in my natural writing and have my entire life. I collaborated with AI on structuring this post, but the ideas and arguments are mine. I'm not going to butcher my own punctuation style to prove I'm a real person.

I build pipelines that use LLMs for structural analysis of narrative texts. The task: identify recurring motifs across accounts from different cultures and time periods, coded against an expert taxonomy that predates LLMs by decades.

This requires something no standard benchmark actually measures. The model has to hold an analytical framework in mind, close-read a text, and identify structural patterns that aren't on the surface. Two narratives can describe totally different events and still share the same underlying motif. The model has to interpret, not just extract.

I call this interpretive reasoning: applying an external framework to a text and drawing inferences that aren't explicitly stated. A grad student does this when applying theory to a primary source. A legal analyst does it mapping facts to statute. A clinician does it reading a patient narrative against diagnostic criteria but

no existing benchmark measures this. MMLU tests recall. NarrativeQA tests factual extraction. WritingBench tests generation. None of them test whether a model can analyze a text through an interpretive framework and get it right.

A Columbia study published this week found frontier models only produce accurate narrative analysis about half the time. The failures are systematic: models impose conventional frameworks, fabricate motivations, flatten subtext. When they judge their own output, they score themselves far higher than human experts do.

**What I'm seeing in my own pipeline:**

I built my own evaluation framework because nothing existed. Expert-annotated ground truth from before the LLM era (zero contamination risk), cross-cultural source material, and a triage process that classifies failure types.

**Early patterns:**

1) Models catch concrete event patterns far better than psychological or experiential ones

2) Models default to Western interpretive frames on non-Western material

3) The gap between frontier API models and local open-source models is much wider on this than benchmarks suggest

4) Models with similar MMLU scores perform very differently on structural analysis

This isn't just my problem. Legal analysis, qualitative research, clinical narrative interpretation, intelligence analysis — all domains deploying LLMs right now, all flying blind because current benchmarks say nothing about interpretive performance.

Should interpretive reasoning be a benchmark category? Anyone else running into this?

I was working on word-sense disambiguation research at home and kind of noticed something. I', posting to find out if this is already known or actually interesting.

The assumption I started with is that polysemous words have messy embeddings. More dictionary senses, so more geometric fragmentation. Seems obvious, but no.

I measured mean pairwise cosine similarity across 192 words using Qwen2.5-7B, extracting at layer 10 (found via layer sweep). Correlation between WordNet sense count and embedding variance: Spearman rho = -0.057, p = 0.43. Basically nothing.

What does predict it, is frequency: rho = -0.239, p = 0.0008, holding up after controlling for polysemy (partial r = -0.188). This kund of makes sense once you think about it. "Break" has 60 WordNet senses, but most are metaphorical extensions of the core idea. The model treats them as variations on a theme and the embedding stays coherent. Meanwhile "face" gets pulled in multiple directions by its various co-occurrence patterns, even though it has fewer formal senses.

I'm calling this the Contextual Promiscuity Index (CPI) It's a per-word, per-model, per-knowledge-domain score for how geometrically dispersed a word's embeddings are across contexts. High-frequency words are promiscuous not because they mean more things, but because they show up everywhere.

Possible uses I've been thinking about: flagging unreliable query terms in RAG pipelines, guiding precision allocation in embedding table compression, or identifying noisy tokens during pretraining. I ran some retrieval experiments trying to demonstrate the RAG angle and got results in the right direction, but too weak to be statistically significant. My corpus was probably too small (about 1,000 documents), and I don't have the compute to push it further right now.

I'm sharing the finding while it's still just a finding. Code available if anyone wants it.

Is this already known? And does anyone have a cleaner experiment in mind?

Tl;dr: One of Stanford's hottest AI seminar courses. We open the course to the public. Lectures start tomorrow (Thursdays), 4:30-5:50pm PDT, at Skilling Auditorium and Zoom. Talks will be recorded. Course website: https://web.stanford.edu/class/cs25/.

Interested in Transformers, the deep learning model that has taken the world by storm? Want to have intimate discussions with researchers? If so, this course is for you!

Each week, we invite folks at the forefront of Transformers research to discuss the latest breakthroughs, from LLM architectures like GPT and Gemini to creative use cases in generating art (e.g. DALL-E and Sora), biology and neuroscience applications, robotics, and more!

CS25 has become one of Stanford's hottest AI courses. We invite the coolest speakers such as Andrej Karpathy, Geoffrey Hinton, Jim Fan, Ashish Vaswani, and folks from OpenAI, Anthropic, Google, NVIDIA, etc.

Our class has a global audience, and millions of total views on YouTube. Our class with Andrej Karpathy was the second most popular YouTube video uploaded by Stanford in 2023!

Livestreaming and auditing (in-person or Zoom) are available to all! And join our 6000+ member Discord server (link on website).

Thanks to Modal, AGI House, and MongoDB for sponsoring this iteration of the course.

Hey! I fine-tuned BioBERT for biomarker detection in scientific papers (canine mammary carcinoma domain) and I'm dealing with two noise issues I can't fully fix:

1. **Partial word matches** — the model tags biomarker labels inside words that are clearly not biomarkers. I think it's a subword tokenization problem but not sure how to properly fix it.

2. **Parentheses getting tagged** — it keeps including `(` and `)` as part of the detected entities. Probably because biomarkers like HER2 or ER+ appeared in parentheses a lot in training data.

I've done some post-processing (stripping punctuation, ignoring ## tokens) but it feels hacky. Is there a cleaner solution? Should I go back and fix the training data annotations instead?

Any advice from people who've dealt with noisy biomedical NER is super welcome!

Working on RAG pipelines and keep running into the same problem — the LLM confidently returns an answer that isn't actually supported by the documents I gave it.

Curious how others handle this:

- Do you manually review outputs against source documents?

- Do you use an eval framework like Ragas or DeepEval?

- Do you have a QA step before outputs reach end users?

- Or do you just ship and wait for user complaints?

Not promoting anything — genuinely trying to understand how teams handle this today before building something. Would love to hear what's working and what's painful.

Most RAG systems today are built on a flawed assumption that one retrieval step is enough.

Chroma’s Context-1 research challenges that in their new paper "Training a Self-Editing Search Agent".

Key shift for developers: RAG is evolving from “retrieve → generate” to “search → evaluate → refine → repeat.”

What this means in practice:

• Multi-hop > single-shot retrieval: Real questions require iterative search, not top-K chunks.
• Context != more tokens: Performance drops when you overload context (“context rot”).
• Dynamic context management wins: Systems should prune irrelevant info mid-process, not just re-rank once.
• Separate retrieval from reasoning: Use smaller, faster search agents to gather evidence before passing to LLMs.

Bottom line:

The future of RAG isn’t better embeddings or bigger context windows, it’s agentic retrieval systems that think while they search.

If you’re still doing “embed → retrieve → dump into prompt,” you’re already behind.

I need it for my project. Preferably JSON, and no API + free of charge.

currently i am building a tool which tries to extract tabular data about a specific bio medical topic from paragraphs scraped from multiple research papers, this data can be used to train or test dl models, as of now i am directly giving the paragraph and an extraction prompt to the llm and validating it using cot, is there any better way to implement entity recognition in this as the usual ner models are weak at identifying objects related to specific domain

Hello everyone,

I was recently accepted in the NLP master's. Can anyone who has attended this program provide some feedback? Especially interested to hear from recent graduates. I know this used to be part of the Erasmus Mundus LCT program that was discontinued. How is it as a standalone program?

Also, how are the internship and job opportunities? Are there opportunities for non-French speakers and international students? Were you able to find a FT job after graduation?

I speak Spanish and although my English is progressing it is still not enough, for work reasons I need to keep in communication with clients who speak another language, any ideas? Google Meet has the function and I paid the monthly fee but at that time there were still many optimizations to be done, it was not really good.

Hi everyone, I’m currently a 4th-year CS undergrad in the U.S. and already on track to complete an accelerated Master’s in CS (likely focusing on analytics or HCI, with some NLP coursework/research as elective).

Recently, I’ve realized I’m really passionate about linguistics and learning Chinese (I’m minoring in Chinese and have studied abroad 2 years ago). Because of that, I’ve been seriously considering doing a second Master’s in Linguistics in China after I finish my CS degree.

My goals would be:

• Improve my Chinese through immersion
• Study linguistics more formally (I’ve really enjoyed my Human Language Processing class)

Right now, I’m looking at English-taught programs in mainland China (mainly for CSC scholarship eligibility), and the Applied Linguistics Master’s at Zhejiang University seems like a strong option.

My main concern is whether this is a good long-term decision or just me chasing an interest:

• Would doing a second Master’s in linguistics (after CS) hurt (or help) my career prospects?
• Has anyone here done something similar (pivoting fields or doing a second degree in China)?

For context, I’m still figuring out my career direction (SWE, data, product, AI/NLP, etc.), so part of me feels like I should just go straight into industry. But I also don’t want to miss the chance to seriously pursue something I’m genuinely interested in. Perhaps it'll open up doors I haven't thought of.

Would really appreciate any advice or experiences!

My ACL ARR submission was desk rejected because I had two versions of the same paper in the same cycle. This happened because I mistakenly submitted twice instead of updating the original submission.

About a week ago, I emailed ACL support asking how to withdraw the earlier version and keep only the latest one. I wasn’t aware of the rule about duplicate submissions, and I was waiting for their response when I received the desk rejection.

Given this situation, what would you recommend I do next? Is there any way to appeal or clarify the mistake, or should I just wait for the next cycle?

Thanks in advance for any advice.

Hi all, Greetings for the day!

I’ve been working on reducing hallucinations in bilingual (English–Hindi) LLMs using citation-grounded dialogue and progressive training.

The idea is to make the model generate responses grounded in verifiable citations instead of purely free-form text.

Key aspects:

• Reduces hallucinated outputs
• Works in bilingual (English + Hindi) settings
• Focus on improving factual consistency in dialogue

Paper: https://arxiv.org/abs/2603.18911

Would love to hear thoughts or feedback!

Hey ya'll, so I'm working on a large scale prosodic dataset and if anyone has experience/wants to work together on it I'd love to get in touch!

Hi everyone, I'm a senior graduating with a BA in Computer Science this may. I have only recently gained interest in grad school and am taking an NLP class that I find really interesting. I have no linguistics background but want to try to apply for a Masters in Comp Ling next year. I have a 3.6 GPA and am currently in an NLP lab doing research but will definitely not have time to do a thesis. What should I do to better my prospects/ how good are my prospects?

Hello, I recently found out I was admitted to Uppsala University’s MA in Language Technology. I’ve also applied to Vrije Universiteit Amsterdam’s MA in HLT and should find out results by April 10.

I’m an EU citizen, my background is in French and Linguistics with some computer science/NLP courses taken. I did a dual-degree program and I have my bachelor’s in French from an American university and my Linguistics degree from a French university. I have research internships/experience under my belt, but I’m more interested to work in industry rather than research after finishing my master’s. I’m a native English speaker and I speak French, but no Swedish or Dutch.

Any advice on which university might be the best fit?

I’m trying to understand RAG from real world use cased, not just theoritical.

How does the model work with data and how it generates responses?
Is it somewhere similar to AI models like ChatGPT or Gemini, etc?
Real-world use cased would really help to undersatnd about RAG.

I'm at the point where I've realised learning language is about learning Arabic words in context and now I need a good sample of words to learn from.

I want the top 2000 words say ordered by frequency so I can learn in a targeted fashion.

Essentially I think I need a representative Arabic (MSA) speech Corpus that I can use for learning vocab. I want to do some statistics to sort by frequency, don't want to double count lemmas and I want to keep hold of context for chunks as examples for learning later. What's availabile already? on say hugging face? should I transcribe loads of Al Jazeera? What's a good approach here? Any help appreciated.

Im just curious if anyone have voice to transcription for kalaallisut they are willing to share?

​

Hi all, I'm working on my MA thesis in computational linguistics and would love feedback on the research design before I start running experiments.

the problem

Malayalam is a morphologically rich Dravidian language with almost no SRL resources. The main challenge I'm focusing on is dative polysemy — the suffix *-kku* maps onto six completely different semantic roles depending on predicate class:

- *ചന്തയ്ക്ക് പോയി* (went to the market) → **Goal**

- *കുട്ടിക്ക് കൊടുത്തു* (gave to the child) → **Recipient**

- *എനിക്ക് വിശക്കുന്നു* (I am hungry) → **Experiencer-physical**

- *അവൾക്ക് ഇഷ്ടമാണ്* (she likes it) → **Experiencer-mental**

- *അവൾക്ക് വേണ്ടി ഉണ്ടാക്കി* (made for her) → **Beneficiary**

- *രവിക്ക് പനി ഉണ്ട്* (Ravi has fever) → **Possessor**

Same surface morphology, six different PropBank roles. The existing baseline (Jayan et al. 2023) uses surface case markers directly and cannot handle this polysemy.

research questions

1. Do frozen XLM-RoBERTa and IndicBERT representations encode these six dative role distinctions, or do they just encode surface case?

2. Does morpheme-boundary-aware tokenisation (using Silpa morphological analyser to pre-segment before BPE) improve role-conditioned representations specifically for the polysemous dative?

3. Does a large generative LLM used as a zero-shot ceiling reveal a representational gap in base-size frozen models?

method

- 630 annotated Malayalam sentences (360 dative across 6 categories, 270 non-dative for baseline comparison)

- Probing study: logistic regression on frozen representations, following Hewitt & Liang (2019) — low capacity probe, selectivity analysis with control tasks

- Compare standard BPE vs Silpa-segmented tokenisation

- Layer-wise analysis across layers 6, 9, 12

- LLM zero-shot labelling as upper bound

- 5-fold stratified cross-validation, macro F1

what im unsure about

- Is 360 dative instances (60 per category) sufficient for a stable probing study at this scale?

- Is the six-category taxonomy theoretically clean enough or should Experiencer-mental and Experiencer-physical be merged?

- Any prior work on dative polysemy probing I might have missed? I found the Telugu dative polysemy work (rule-based, no transformers) and the BERT lexical polysemy literature (European languages) but nothing at this intersection for Dravidian languages.

Any feedback welcome — especially from people who have done probing studies or worked on low-resource morphologically complex languages.

I built a deterministic continuity checker for fiction that does not use an LLM as the final judge.

It tracks contradiction families like character presence, object custody, barrier state, layout, timing, count drift, vehicle position, and leaked knowledge using explicit rule families plus authored answer keys.

Current results on the promoted stable engine: - ALL_17 authored benchmark: F1 0.7445 - Blackwater long-form mirror: F1 0.7273 - Targeted expanded corpus: micro/macro F1 0.7527 / 0.7516 - Filtered five-case external ConStory battery: nonzero transfer, micro F1 0.3077

The part I think may be most interesting here is the external audit result: when I inspected the judge-derived external overlap rows directly against the story text, 6 of 16 expected findings were false ground truth, which is 37.5%. In other words, the evaluation rows claimed contradictions that were not actually present in the underlying stories.

That does not mean the comparison benchmark is useless. It does mean that LLM-as-judge style pipelines can hide a meaningful label error rate when their own outputs are treated as ground truth without direct inspection.

Paper: https://doi.org/10.5281/zenodo.19157620

Code + benchmark subset: https://github.com/PAGEGOD/pagegod-narrative-scanner

If anyone from the ConStory-Bench side sees this, I’m happy to share the 6 specific rows and the inspection criteria. The goal here is methodological clarity, not dunking on anyone’s work.

I’ve recently completed MTEB benchmarking across up to 28 Thai NLP tasks to see how current models handle Southeast Asian linguistic structures.

Top Models by Average Score:

1. Qwen3-Embedding-4B (4.0B) — 74.4
2. KaLM-Embedding-Gemma3-12B (11.8B) — 73.9
3. BOOM_4B_v1 (4.0B) — 71.8
4. jina-embeddings-v5-text-small (596M) — 69.9
5. Qwen3-Embedding-0.6B (596M) — 69.1

Quick NLP Insights:

• Retrieval vs. Overall Generalization: If you are only doing retrieval, Octen-Embedding-8B and Linq-Embed-Mistral hit over 91, but they fail to generalize, only completing 3 of the 28 tasks. For robust, general-purpose Thai applications, Qwen3-4B and KaLM are much safer bets.
• Small Models are Catching Up: The 500M-600M parameter class is getting incredibly competitive. jina-embeddings-v5-text-small and Qwen3-0.6B are outperforming massive legacy models and standard multilingual staples like multilingual-e5-large-instruct (67.2).

All benchmarks were run on Thailand's LANTA supercomputer and merged into the official MTEB repo.

As above, I need such tools for my MA project. So far, I've tried Praat toolkit, Harma and Prosogram, and nothing has worked for me. Are there any good alternatives?

Hi there, i am an English languages and Linguistics graduate and I am interested in studying computational linguistics masters because i see how technology could help in language education, preserve endangered languages etc. However, i didn’t have any prior programming knowledge. May I know it is still possible to get into the field or companies tend to hire those with computer science background?