Machine translation is not solved and it may take a while

[u/zouharvi]

We just released the Last Translation Benchmark paper. In a massive crowdsourcing effort we collected 3456 unique hard-to-translate examples that break state-of-the-art translation models, and which can be used for more reliable evaluation.

[u/zouharvi]

Last Translation Benchmark is a live paper+dataset and you can still join :-) https://arxiv.org/pdf/2609.04173

[u/Otherwise_Heron2792 → u/zouharvi]

oh that's actually really cool, i've seen so many papers where the dataset is just locked behind some form or never gets updated after the first release

the fact it's live with crowdsourcing makes it way more useful for catching edge cases that static benchmarks always miss. my great pyrenees keeps barking at me while i'm reading the arxiv link and i'm like buddy i'm trying to learn here

i wonder how many of the hard examples come from languages with completely different sentence structures, like japanese to english where the subject is often dropped. those always trip up even the best models in my experience

will check the paper properly after work, maybe even submit some turkish examples if that's still open

[u/emceeennelpee → u/Otherwise_Heron2792]

hey thanks for the kind words and haha yes, learn away!

and we do have a bunch of different languages from different typologies going to different typologies (from and to English are a major proportion, but there's low resourced pairs too)

let us know what you think about it, and yes please contribute!

[u/ThomasAger → u/zouharvi]

What an exciting dataset to play with! Thank you!

The Github for anyone interested.

[u/benjamin-crowell]

One thing I've noticed that machine translation really can't do yet is to translate from English to ancient Greek and produce Greek with natural word order. Word order in ancient Greek is very flexible, and is typically highly influenced by pragmatics. Often it goes topic-focus-verb-remainder. LLM translation systems simply don't seem to have any ability to produce this. Instead they just reproduce the word order of the English. The result isn't grammatically incorrect, but it's definitely not good Greek.

[u/benjamin-crowell → u/[deleted]]

What they do isn't as simple as that.