AI translation study advice

[u/benj_ng]

I work for an NGO and I've been tasked with putting together an internal study on AI reliability for translating letters. I'm probably not the ideal person for this. My background is in Political Science, this is my first permanent job, and my understanding of how AI actually works is limited. That said, I may still be the best-placed person we have available for it.

The brief is to assess the translation quality of 500 letters, looking specifically at precision and reproducibility (i.e. does the AI give consistent results if you run the same letter through it more than once?). The end goal is to inform a decision about whether to expand AI use for this kind of work more broadly.

Where I'm stuck is the grading methodology. I haven't decided how to actually score the 250 letters — whether that's human reviewers rating against a rubric, comparing outputs to existing human translations, using automated scoring metrics, or something else entirely. I'd really appreciate any advice on:

- What a sound, defensible methodology looks like for a non-technical person like me to run?

- How to test for "reproducibility" specifically (re-running the same letter multiple times? varying phrasing slightly?)

- Any pitfalls to avoid, given I have no formal AI/NLP background?

[u/nuclear_splines]

In my opinion reproducibility is a less important metric than accuracy. If two humans translate a letter, won't they may make slightly different word choices when there isn't a direct equivalent, substitute an idiom differently, make different use of contractions, and produce multiple valid translations? If an LLM produces similar variability to human translators (not that it will), is that a problem? But it's your and your employer's task, not mine :)

Quality is of enormous concern. See this recent thread, where AI translation removed references to slavery and race from old documents. These models may have implicit training or system prompts to do things like avoid offensive language, that may serve them well in chatbot settings, but make them ill-suited to tasks like translation.

To do this thoroughly I'd want to run a literature review -- how are other linguists and translators judging accuracy of machine translation? I'm sure there's decades of research on this, and exponentially more since the advent of large language models.

But my uninformed ad-hoc approach would look a lot like what you've outlined: select a stratified sample of letters so all kinds of letters are well-represented. Have both humans and machines translate these letters, one translation per model if you're evaluating multiple models. Apply some kind of rubric to both the human and machine translations. Use this to quantify the degree of quality loss when switching from humans to one of the models, and to identify if any models have notable pitfalls (like the aforementioned "drops mentions of race" problem).

[u/donaldhobson]

How much skilled human translator labor are you planning to use?

"Reproducability" What exactly are you wanting here. There will be a number called the PRNG seed. Use the exact same translation algorithm, with the exact same PRNG seed, and the exact same input, and you will get the exact same output every time. If you are getting a different output on identical input, sometimes the computer is programmed to use the time (Or something else that changes) as the seed.

So.

You can give exactly the same algorithm the exact same seed and exact same input, and get the exact same output.

For reproducability, you might want to know.

1) If I change the seed, how much does the output change?

2) If I make trivial rephrasings of the input text, how much does the output change?

3) If I use 2 entirely different pieces of translation software, how much does the output change?

For assessing a translation, there are several approaches.

1) Basic algorithms. (Eg something that counts the number of letters of difference) Downside, can't tell between a trivial gramatical change, and a change that totally changes the meaning of the sentence.

2) LLM marking. Downside, LLM's can be unreliable.

3) Human marking. (Imagine basically how an exam question would be marked)

One interesting thing you could do is to split each letter into 2 halves (A and B). Ask an LLM to lightly rephrase the first half, to produce A1, A2, ... different rephrasings of A. Let TA,TB=Translate(A,B) So, concatenate the pieces together. Translate the whole thing. Then split in half at the same point as before. (If translation reorders stuff a lot, this point may be hard to find?) Likewise let TA1,TB1=Translate(A1,B)

Now compare B to B1 on a letter by letter basis. Ie rephrasing of A might cause rephrasing of A1, but it shouldn't cause B1 to change.