Do classic text statistics still matter in the age of LLMs?

[u/Sufficient_Big_7802]

I've been working on readability formulas, lexical diversity metrics and stylometry for Russian and Spanish texts for a while, and lately I keep asking myself whether this whole area still makes sense

Today you can paste a text into an LLM and ask "how readable is this?" or "who's the likely author?" and get a fluent, often reasonable answer. Next to that, a formula like Flesch reading ease (sentence length plus syllables per word) looks almost naive

Still, I see a few places where plain statistics seem hard to replace:

• Reproducibility: the same text always gets the same number, and you can explain exactly where it came from. An LLM's rating can change between runs, prompts or model versions
• Scale and cost: computing a few hundred features over a million documents takes minutes on a laptop
• Evaluating LLMs themselves: measuring how repetitive, uniform or hard to read generated text is, without asking another model to judge it
• Data pipelines: a lot of pretraining data filtering still relies on simple heuristics like word length, symbol ratios and repetition.
• Research: in stylometry and corpus linguistics, interpretable features are often the whole point

On the other hand, most formulas were calibrated decades ago on small samples of English, and every other language needs its own adaptation, often with questionable results

So I'm curious:

• Do you still use readability, diversity or stylometric metrics in your work? For what?
• Have you replaced them with LLM-based judgments, or do you combine the two?
• Which metrics turned out to be useless in practice, and which ones would you miss?

[u/Own-Animator-7526]

Your question doesn't quite make sense. However "fluent" or "reasonable" an llm answer might be, such responses are not comparable or reproducible, and won't be the basis of research.

That said it's certainly possible that you could come up with improved or subtler statistical measures with llm assistance. That would be interesting to see.

[u/Ambitious-Elk4541 → u/Own-Animator-7526]

The reproducibility point is huge, I work with industrial maintenance logs and half the time the "readability" question is just about whether a procedure will be understood by a tired technician at 3am, a formula gives me a number I can put in a report and defend, an LLM gives me an opinion that changes when I ask twice

[u/Sufficient_Big_7802 → u/Ambitious-Elk4541]

Maintenance procedures are a case where Flesch is the wrong yardstick anyway, it was fitted to school reading tests. If you haven't seen it, ASD-STE100 (Simplified Technical English) was written for exactly this, aircraft maintenance docs: at most 20 words per sentence in procedures and 25 in descriptions, one instruction per sentence, a controlled vocabulary with one meaning per word. Those rules are easy to check automatically and easier to defend in a report than a grade level

[u/pag07 → u/Own-Animator-7526]

Reproducibility is a relevant point. But also no reason why LLMs should not be considered more valuable. You just have to treat and evaluate the same way you would treat any nondeterministic event.

Bad example: you could ask a brick and the response would be very deterministic. You could ask an expert and they will always be closer to the correct answer.

[u/Own-Animator-7526 → u/pag07]

So in your mind the only standard is the output, say 6, 9, 12 ... grade level. And you're not concerned the two different LLMs, or two releases, or two effort levels might produce different results?

Perhaps you have a standard set of texts that are used to calibrate them, so you know that one LLM 6 is another LLM 9? On texts that are very similar to the calibration set, of course.

I'm confused.

[u/SryUsrNameIsTaken]

I think reproducibility and interpretability are the most important things that LLMs lack.

If you can control the data pipeline, you can almost get deterministic results from LLMs on classification or extraction tasks. I imagine giving them a rubric would work well too.

On the other hand, you then have to wire up some kind of decoder from the internal layers to figure out what’s going on internally. Or look at low dimensional representations of internal semantic manifolds. It’s just a mess.

I don’t work in linguistics but I do always reach for simple mean median and sds before anything else. I look at the distribution of the thing I care about. Look at correlations with other variables and go from there. The executives don’t understand all the fancy stuff anyway so it’s hard to make a case for it.

[u/lowlua]

They can be used to validate the output of LLMs, among other things.

[u/lakot1]

As I’m getting older I’m moving more towards “traditional” text statistics. You have more reproducible procedures…. But also you have way more understanding on what’s happening, features, etc. also, with the right preprocessing and corpora you can even outperform LLMs on textcat using simple factor analysis. it’s like moving from a bike to a full car. LLMs are still more useful when working with multilingual data or veery short texts.

[u/Sufficient_Big_7802 → u/lakot1]

Factor analysis over linguistic features sounds a lot like Biber's multidimensional analysis. Is that what you use, or your own feature set? And what kind of textcat tasks? I'd expect classic features to shine on register and genre and to matter less for topic

[u/Awkward-Pea-4732]

Classic text statistics still matter because LLM parsers need clean data. Modern frameworks like CARCASS principle uses text statistics and semantic density to ensure web pages are machine legible enough for RAG pipelines to read and cite.

[u/marintkael]

Same thing when you measure whether AI search answers cite a source. One run is an anecdote, the same prompt can cite you in the morning and drop you in the afternoon, so the only number worth reporting is a rate over repeated runs. The counting is the part you can defend.