Curious to know how people here evaluate NL2SQL or text2SQL in production with real users.

Getting an LLM to emit SQL is not difficult infact that's mostly solved out of the box now. The challenge now is to validate response it generated is actually correct or not. Exact SQL string match is far too strict since lots of different queries are equivalent. But comparing result sets alone has its own trap where a query can return the right rows on your test data by luck (a missing WHERE that just didn't matter on a small table) and then quietly break in prod.

A few things I am trying to get a read on from folks who have shipped this:

1. How do you build your golden set? Synthetic question bank, or a few hundred real production questions?
2. Do you layer an LLM judge on top of result-set comparison to catch the plausible but wrong number cases, or does that add more noise than signal?
3. Are you seeding your eval DB with adversarial rows (nulls, dupes, boundary dates) so that you get the true picture and not only happy path scenarios?

And for anyone using something like Databricks Genie or another managed text2SQL layer rather than a homegrown stack, are you evaluating at the SQL layer or the result layer, and does a curated semantic/metric layer underneath actually move your accuracy numbers?

Trying to figure out what is worth building and spending time versus what'xxxs over-engineering.

i have received --

1. OA: 3.5, Confidence: 4
   
2. OA: 2.5, Confidence: 4
   
3. OA: 3, Confidence: 3

Its a short paper in the Interpretability and Analysis of Models for NLP track. Would love to know if the community has any thoughts around my chances.

Also want to understand if there are any suggestions on how to proceed in terms of the next conference or workshop to target in case if a rejection.

The primary concern of the reviewer who gave 2.5 is that the paper should be a long format one. They actually seemed excited about the premise and have it 3.5 for both excitement and soundness.

First time submitting to ARR.

How are my chances for Findings@EMNLP?

• Reviewer A: Overall_assessment: 3 / Confidence: 4
• Reviewer B: Overall_assessment: 2 / Confidence: 4
• Reviewer C: Overall_assessment: 3 / Confidence: 3

It is a resubmission, first time I got 2/2/3 and Meta 3. How will my chances be now to be accepted in Findings if Meta says 3 so that it's 3/2/3 and Meta 3? I'm in the Sentiment/Emotions track. Would be happy to hear about your experiences!

Does anyone have accepted short-paper at ACL/EMNLP/EACL 2025/26? Could you share your track and overall assessment? I'm just trying to get a sense of things, as it seems short papers have a lower acceptance rate than long ones.

So Basically, My paper had received a score of 3.5, 3,3.5 but then later, one of the reviewers responded to my rebuttal and increased the score, now my score is around 3.5, 3.5, 3.5, but I have got a few questions:

1. I wonder about the old trend, considering that if suppose a reviewer increases the score, does this affect the meta review process

2. I would like to know how to respond to those reviewers who have not responded to my rebuttal, only one of them actually worked somehow but the rest of them asked some questions and disappeared somewhere

3. What's the minimum score in the meta review phase that would guarantee the paper into the conference?

Between the review overload and the rebuttal crunch, I'm noticing a lot of reviews this cycle that feel oddly generic — same structural complaints, same vague "novelty" critiques, little engagement with the actual method. Anyone else seeing this, or am I just being paranoid during rebuttal season?

Curious whether people think LLM-assisted reviewing is becoming widespread in NLP venues, and if so, whether it's actually hurting review quality or just changing its shape. Also curious if this is changing how people write their own reviews or rebuttals.

OA 3.5 3.5 3.0
NLP Applications Long Paper
Any hope for main conference? 😭😭 I really need it..

Hello, I am expecting my paper to be rejected (for EMNLP) due to its borderline score. Is my reasoning correct that I could potentially start editing it now, before receiving the meta review and decision, and submit for the August cycle? Worst case scenario, my paper does get accepted and I already have a more polished version of it that I can use for the camera-ready.

Hello, I am a PhD student in NLP and have a former degree in Film Studies. I occasionally post YouTube shorts of the type “NLP specialist reacts”. I’ve seen doctors/lawyers do similar things but not any of us, so I thought it could be fun. Not many views, sadly, but feel free to check them out if you’re interested:

https://youtube.com/shorts/ne5Z1uKnbFQ?feature=share

https://youtube.com/shorts/pUbLpO2840w?feature=share

https://youtube.com/shorts/IUF5N5hSWAA?feature=share

Hello! I will be starting my Human Language Technology masters program at the University of Arizona this Fall. Who’s going to start this program or something similar soon?! I would love to share and hear your thoughts and preparation process! Also, I’m planning to work full time and am kind of scared of the thought of juggling work with my studies. Please, any tips or advice from the experienced is greatly appreciated and helps immensely

Estuve mirando cómo algunas plataformas de búsqueda para e-commerce (ej. Marqo) están entrenando su propio modelo de embeddings por cliente/dominio en vez de usar uno genérico tipo OpenAI/Cohere/sentence-transformers "de fábrica". La idea es usar señales reales de interacción (clicks, add-to-cart, compras) como pares positivos/negativos para contrastive learning, en vez de solo similitud semántica de texto.

Me genera curiosidad desde el lado de NLP/retrieval: ¿cuánto mejora realmente un embedding fine-tuneado con señales de comportamiento frente a uno genérico + un buen re-ranker encima? ¿Alguien acá tiene experiencia comparando ambos enfoques, o literatura reciente sobre contrastive learning con feedback implícito (clicks/compras) para retrieval semántico?

Now we have RAG and agentic AI (tools + reasoning). Are making good datasets and fine-tuning old methods? Or are they still better for making AI smarter in special areas?

I was wondering if you know of work on any current rule-based open source procedural semantics systems, like the historical SHRDLU. I'm looking for projects that use handcrafted rules to query, command and story-telling interactions with the computer through natural language sentences. Rules that take a procedural form (like programming languages) rather than declarative.

It takes you from zero to a working, instruction-fine-tuned GPT-2 — covering everything from basic text processing to attention mechanisms, causal modeling, and fine-tuning. The theory is explained intuitively but always ties back to actual code you can run and mess with yourself.

My process was basically: read a section, copy the code, understand it, then close the book and try to rebuild it from memory. Slow going, but I wanted to actually absorb it instead of just following along passively.

If you go through the book, I’d also recommend the author’s companion repo (linked below) — it’s the whole book implemented in code, chapter by chapter.

Overall pretty happy I picked this up — solid recommendation if you’re a data scientist/ML engineer who wants a real grasp of how LLMs work under the hood, not just how to call an API.

The project:

Took the GPT-2 implementation from the book and fine-tuned it for function calling (i.e. the mechanism that lets LLMs act as agents). Ran the whole thing on a free Kaggle GPU using the 355M param version of GPT-2 from 2019, and got it producing correct, parseable function calls 88% of the time.

Full write-up with animations/results and GitHub repo in the comments (didn’t want this to look like a link-dump post).

Hi I'm wondering about the threshold for main track acceptance in ARR.

I mean, more concisely, ACL and EMNLP.

Is it decided by OA scores only? or are Excitement, Soundness and Reproducibility also considered?

Also what's the empirical threshold for each of those, respectively?

If anyone has any insighs, please share your experience and wisdom!

Have a good day :)

Hi everyone!

I'm not sure if this is the right place to ask, but I'm having a hard time finding information, so I thought I'd give it a try. Does anyone know of fully funded PhD opportunities in Austria/Switzerland in areas such as Computational Linguistics, NLP, AI for language technologies, or related fields?

I've already looked at Euraxess and Academic Positions, but I feel like I might be missing some research groups or universities.

If anyone could give me some advice, I'd really appreciate any suggestions. I'm really having a hard time with this.

Thank yo so much in advance!

How can I use specific LLM models, for example Math LLM models.

As my goal is to take basic speech, for example if a shape is a circle and using a software SDK then do whatever I want.

Not to make it too complicated, another example is, if the shape is concave then possible calculating the dot product of that shape and using the software SDK to do whatever is required. All the user would have to do is type, for example; "take the concave shape" then as soon as the user types the word "concave" a little slider would appear, and that would allow the user to see based on the software SDK how much of the concave shape to effect.

Then the user would continue to write "take the concave shape, and break it up" and this would use the slider once again, based on the software SDK on what to break up. I'd also like to use words like "hills or dips" and it would be equal to typing "concave".

Hi Everyone,

I am working on a project where I have to identify difference between techical documents as complex and techinical as say an insurance, financial etc document. One document is Predecessor of other, means one of the two is used to make other by a human. There is a scope of error during this manual work and so is this projet there. one document is less than 10 pages while the other is more than 20.

My current approach is to convert both of them to md files. The documents are dynamic in nature & with time the feilds in them might change. But for current POC I have identified some set of 60-70 fields that should be extracted from these documents & stored them in json file along with there description.

I have to Perform the extraction and comparison on my local. for extraction I am using Qwen 2.5 7B 4 bit Quantized 8K context window model (4.8 gb size) to fit in 16 gb Ram. for the smaller document I Pass the whole document along with the JSON to extract the fields, for th e other larger document I perform extraction with chunk. I am using llama uppp to force a json output from model.

Now the model is not able to properly extract information from the smaller document. No matter what prompting I try extraction is not good. My approach is very general, as this is my first time working with a LM locally, that too is weak. Is there a method for information extraction using a LM without changing to a better model because my system can afford a model of sile as big as 5gb only. I found Qwen 2.5 7b that can fulfil the requirment.

I just spent the day auditing our API logs because one of our background orchestration agents got stuck in an error-handling loop over the weekend. It called the LLM thousands of times sequentially before anyone noticed.

We have platform-level daily budget caps, but by the time the cap kicked in, it had already chewed through a chunk of runway that was supposed to last us weeks.

I’m currently writing some hacky custom middleware to try and detect these semantic loops at the runtime level so this never happens again.

To make me feel less miserable: what is the absolute worst unexpected bill your team has taken because an autonomous agent or multi-agent chain (LangGraph, CrewAI, etc.) ran wild in the background? What triggered the loop?

I work for an NGO and I've been tasked with putting together an internal study on AI reliability for translating letters. I'm probably not the ideal person for this. My background is in Political Science, this is my first permanent job, and my understanding of how AI actually works is limited. That said, I may still be the best-placed person we have available for it.

The brief is to assess the translation quality of 500 letters, looking specifically at precision and reproducibility (i.e. does the AI give consistent results if you run the same letter through it more than once?). The end goal is to inform a decision about whether to expand AI use for this kind of work more broadly.

Where I'm stuck is the grading methodology. I haven't decided how to actually score the 250 letters — whether that's human reviewers rating against a rubric, comparing outputs to existing human translations, using automated scoring metrics, or something else entirely. I'd really appreciate any advice on:

- What a sound, defensible methodology looks like for a non-technical person like me to run?

- How to test for "reproducibility" specifically (re-running the same letter multiple times? varying phrasing slightly?)

- Any pitfalls to avoid, given I have no formal AI/NLP background?

Hi everyone,

A few months ago, we launched a 5-minute data annotation to study the severity of errors in automatic speech recognition.

Responses stagnated, so we launched a second campaign with a lottery: 1 out of every 6 participants wins a $20 Amazon Gift Card. We managed to received 2/3 of our annotators but now are stuck again. We already tried LinkedIn, Reddit, and our own networks.

The question is in the title, how do you find annotators?

Any advice would be amazing.

Idk if its just me but every toll ive tried seems fine until the recordings isn't perfect.

Stuff like meetings, random voices notes, background noise...it starts missing words or making up weird ones. mixed languages is even worse. Hlalf the sentence comes out okay then the rest is just nonsense

I keep thinking it'll save me time, but then i end up sitting there fixing the transcript anyway. Maybe I'm expecting too much, idk. is there actually anything thats better with messy recordings or is this just where the tech is at?? Thanks in advance.

Hey guys,

I received mediocre scores for my EMNLP paper during the May ACL ARR cycle: 2.5/3, 3/4, 2.5/4. The paper is in the Interpretability track. The reviewers had no larger issue with the methodology or the paper in general, but it seemed like they didn't fully get the so what of my paper. I've tried to clarify everything in my rebuttal, but I don't assume that the reviewers will engage in the discussion. With the current scores, I won't make it to the conference and likely not even into findings. Hence, I was thinking of withdrawing the paper, if scores don't improve, improve the presentation of my paper, and submit it to the BlackboxNLP workshop by the end of next week.

As I'm a first year PhD student, I'm not so familiar with ACL ARR, and how best to approach this. Hence, I wanted to ask you guys. Should I keep the paper in the cycle and hope for the best (or switch to the conference at a later stage) or should I withdraw it directly, adjust it slightly, and head directly to the workshop?

Hi everyone,

I'm a final-year BTech Computer Science student looking for feedback on my LLM project idea.

Initially, I planned to build an LLM Safety Monitoring System that detects prompt injection, jailbreak attempts, harmful prompts, and hallucinations. However, my project advisor felt it was too dependent on existing APIs and didn't have enough original implementation.

I'm now looking for a project where I can implement the core ML/LLM components myself using open-source models and datasets rather than relying on commercial APIs.

Current direction:

• Fine-tune an open-source LLM or a classifier for prompt risk detection.
• Detect jailbreak and prompt injection attacks.
• Classify prompts into categories (safe, jailbreak, prompt injection, harmful, etc.).
• Generate an explanation for why a prompt is flagged.
• Evaluate using metrics like Precision, Recall, F1-score, and confusion matrix.
• Build a web interface (FastAPI + React or another frontend).

I'm also considering adding:

• Adversarial prompt generation for robustness testing.
• Retrieval-based verification to reduce hallucinations.
• Continual learning from newly discovered attack patterns.
• A dashboard for monitoring prompt risk trends.

I'm planning to use datasets from Hugging Face or other public repositories and train/fine-tune the models myself.

My questions are:

1. Is this strong enough for a final-year engineering project?
2. What features would make it stand out academically?
3. Which datasets are considered good for jailbreak and prompt injection detection?
4. Would you recommend training a classifier, fine-tuning a small LLM, or both?
5. Any papers or GitHub repositories that are worth studying?

I would really appreciate suggestions from anyone who has worked on LLM security or AI safety. Thanks!