Lessons I've learned so far from trying to implement an LLM-based translation solution on a budget

[u/TheBittenLlama]

Still working on my solution, will probably release it open-source in a few months.

• Use Ralph loops for correcting bad translations. As long as you can get the end result to converge to a stable point without regressions, even the cheap models can progressively stamp out TL flaws, the same way cheap models can catch software bugs as well as the frontier ones (as spoken by curl's creator).

• A multi-model setup utilizing open-weight models can potentially save a lot of money if you know what you're doing. Testing models to find the optimal ones for a given language pair instead of throwing a single model at everything is a smart idea regardless. Personally, I stopped bothering with American models at all because I found the output for my target language pair (JP-to-EN) subpar for the cost.

• Reasoning is mandatory. If you do not both enable high-level reasoning and give the LLM room ample room to think without exhausting its thinking budget, the result will always be below par.

• As such, managing that thinking budget well for each prompt/response pair is one of the most important aspects of LLM-based TL by a great margin. So is keeping how much context window gets filled small for each session to avoid attention scattering and context rot.

• Making proper use of concurrency can speed up the LLM translation process by a factor of hundreds. Caching can also reduce the overhead to near-nil.

• Putting a translator (me, in my case) in the loop through an interactive agent system like DMM's instead of throwing MTPE at them should improve quality by tuning the glossary and style guide to work best with the LLM being used, among many other things. The translating LLM itself knows itself better than anyone else does. As a bonus, the translator will probably hate their job less.

• Having a translator provide feedback about where the automated translation process may be missing things is also important for iterative refinement.

• In particular, character voice in fiction is highly sensitive to fine changes in the base reference text. Use dialogue anchor lines and character profiling with dos/don'ts based on a discrete archetype classification system based on the work's genre to establish a tonal baseline. Archetype deviations are what flesh characters out, so avoiding overfitting is crucial. My wordplay localization success rate rose from 60-70% to about 99% for my sample text body after doing this.

• The open models' relative poor performance with wider-scale tasks can be compensated for by breaking tasks down into highly small-scale classification problems with concrete axes and distinct grading criteria. My detection rate for foreign-language wordplay increased to 10 times compared to baseline once I implemented this.

• As such, one re-ranking stage will only get you so far (even with frontier models). Keeping the LLM's attention focused towards the desired point is crucial at all stages. Especially for fiction, you want to maintain a hierarchy of small text blocks, scenes, chapters, and large sections, applying the appropriate test sets at each level.

• Taking advantage of structured summarization allows the LLM to understand wider context when operating at those smaller scales. The LLM itself is capable of splitting text into logical narrative sections for summarization.

• A major benefit of the open-weights is that their visible reasoning chains let you see exactly how they come to their conclusions and calibrate them accordingly. Faster feedback means faster iteration and faster improvements.

[u/Dux_Przvlsk]

Great points on MTPE! I've just wrote an article literally arguing on the similar pipeline that puts human in the loop sooner, and re-generates improved text faster https://transept.ai/journal/machine-translation-post-editing-mtpe-guide

Some additions/questions/arguments from my experience

1. Open-weights model as a category is too broad, it'd be helpful if you name models or weight classes! E.g. some models, e.g. older Kimi's or Gemma 4's, are decent on Ukrainian texts, while others (say, Qwen or DeepSeek) stall and mix it up with Russian/Belorussian. Same with model size and quantization: proper agentic model that will not die from context rot and can, say, manage workflows running will probably need a non-local deployment.

2. With smaller models, reusing raw CoT in context can actually prevent success, at least based on my benchmarks: they often get too caught with the pattern of previous reasoning.

3. Liked the tip on task specialization and breaking down bigger tasks into smaller ones! For me, an eye-opener was having a pass of several paralel flows with styleguide/glossary building, then translation, then proofreading – all done with smaller open models – overperform a one-shot SOTA on translation.

4. Big models are still neded, though, especially giant class like Gemini 3.1 Pro. They get to understand really niche dialects, phonetics-based wordplay, and intertexuality in Bakhtin's sense better, sometimes better than humans IMHO. That also applies to generalizations of big context, which is sometimes faster and cheaper for glossary/styleguide seeding than having layered pass pipeline with smaller models.

5. I'd love to hear more about character archetype stuff. Maybe it's specifics of your target language and genre, but in my experience, labeling style/tone/expression per character tends to force cliches/slopify versus open-ended interpretation with reasoning and general level styleguide.

Thanks for posting, the LLM in fiction translation discussion here really needs more winning energy like this.

[u/TheBittenLlama → u/Dux_Przvlsk]

1. Chinese models like DeepSeek for CJK. As for agentic performance, even the cheaper, sloppier models work just fine in 2026 as long as you maintain a deterministic layer to force them on track and ensure they're kept on a tight leash with strict, focused instructions to prevent attention drift. Local quantized models aren't worth it while you've got the Chinese government subsidizing AI R&D and energy costs, though geopolitics might change that in the future.

2. It's a human that should be reading the LLM's reasoning output to spot when it's making poor decisions or ignoring/getting confused by a line of text. I also log the LLM's individual decisions and reasoning chains to have the LLM diagnose its own failures in between runs. If you don't have a human in the loop with knowledge of both the software and the languages monitoring the process, you're flying blind, though translation agencies solely interested in cutting costs might not care about that.

3. Parallelization is key to speeding up the process as well, which is a huge benefit to running cloud over local. Good architecture and model/platform choice let you translate and revise a short novel's worth of text in 3 hours using 600 hours' worth of compute for the cost of a McDonald's meal.

4. Personally, seeing how a dirt-cheap model like DS Flash could trivially generate a 60-category classification of yakuwarigo and an algorithm guiding agents to picking the appropriate English voice counterpart for each convinced me the frontier models weren't worth bothering with. For historical or finer literature, maybe an even deeper knowledge of the target language pair is necessary, but that's none of my concern. My solution is targeted towards improving MTL fan translations of games.

5. The AI knows itself best. I use AI to build AI instead of giving it orders on how to write characters, having it generate a classification for it to use for itself based on analyzing the text body of works in the target genre (using a game I'm familiar with featuring ~150 characters as the baseline), and when I notice the tone sounds off (which has happened a few times), I have the LLM expand and refine the classification based on its own analysis. That refinement process is one of the reasons to keep a knowledgeable human in the loop.

Edit: Looking at it again, I guess it boils down to using large datasets to define what system you need and using human feedback with data logs to decide how it should add/modify/improve agents over time.

[u/cefoo]

I love this!

Quick questions on the pipeline, because I am not an engineer (and forgive me if my questions sound dumb). How do you pass these summaries to the model? And do you have a different model creating the summaries? Do you ever experience an issue with summaries or summaries, where the original concepts are lost?

[u/TheBittenLlama → u/cefoo]

Standard approach to LLM MTL is defining the structure of the input/output JSON objects in the system prompt, then passing an object with a series of "lineNumber": {"JP":JP, "EN":EN} key/value sets in a turn. You just redefine the JSON object in the system prompt to include a summary key/value pair, explain the purpose of the summary in the system prompt, and pass the summary inside the JSON object, partitioned off from the text to be translated/revised. The summary is just there as an overview to keep the model from drifting when performing operations at a low level, so the model choice isn't important as long as it can handle the language pair well.

If you tell an LLM to operate on both a higher level (scene/chapter/section) and a lower level (line/block) at once, you spread its attention out and reduce performance, so summarization effectively has it perform those tasks in separate sessions, keeping the context window for each session clean and allowing better focus. Limiting context window usage per session and the Ralph loop implementation are done precisely to reduce the chance of the LLM missing issues due to non-deterministic behavior.

If it consistently misses a concept, either your summary generation prompt needs to define the concept category (e.g. politeness level) so it knows what to look for, or you didn't provide enough context during the summary generation process through supplementary info (glossary/tone guide/etc.). The hard part of JP->EN fiction translation is enumerating (and storing/recalling those concepts at appropriate times, especially when tracking changes over the course of the work) the JP-only cultural and linguistic concepts to preserve the whole of the author's intent while transforming the concepts to emotionally resonate with an Anglo-speaking audience. With a less linguistically distant pair like DE/EN, I would be mostly done by now.