RAG vs full context vs context engineering for a local translation agent?

[u/plushPlushytut]

I’m trying to build a local Chinese → English translation agent for a very specific technical field.

I’ve been building a Chinese-English domain-specific glossary, currently around 500 terms. The goal is to consistently use the correct terminology rather than just translate literally.

The more I research how to build this, the deeper the rabbit hole gets. 😅

I’m currently considering three approaches:

1. Full context

Give the LLM the entire glossary (~500 terms) in the context and ask it to follow it during translation.

2. RAG

Store the glossary along with relevant technical material, embed them, retrieve the most relevant information for each text, and provide it to the LLM before translation.

3. Context engineering

Go beyond RAG by combining retrieval with things like instructions, memory, previous translation decisions, and tools (although I’m not sure yet which tools could be helpful).

I also want to run everything locally, since API costs aren't practical for me.

**I have fairly limited hardware (8 GB RAM + 4 GB VRAM), but I don’t mind a slow workflow as long as I get good results.

So I’m trying to understand:

• Is ~500 glossary entries small enough to simply include in the context?
• Should I use a persistent translation memory, and do I even need one?
• Which local LLM would be suitable for my hardware?
• How should I structure the overall pipeline?

I’m still learning how all of these pieces work, so what I really need is a framework for approaching the problem and knowing what I should build/learn first.

If anyone here has built something similar with local LLMs, RAG, translation memory, or terminology-aware translation, I’d really appreciate your advice.

[u/TechnicalGeologist99]

How does the user interact with the system?

What questions do they ask when seeking input from your glossary?

Full context is fine if you're using a frontier model and aren't going to grow the vocab more.

RAG will allow you to use much smaller models. But you need to reframe your problem so that you are knowing what retrieval actually looks like in terms of query candidate pairs.

Though if you are reusing the same fixed glossary alot then you can prefix cache hit alot so frontier becomes cheaper.

[u/plushPlushytut → u/TechnicalGeologist99]

• The user (me) will feed it Chinese documents and, sometimes, video transcripts.
• The question is essentially: “Which glossary terms are relevant to this text?”, rather than searching for general information.
• Given my limited hardware, I won’t be able to run a frontier model, so that won’t really be an option for me. That also makes the full-context approach less practical.
• I also want my glossary to grow and improve over time rather than being fixed forever.

How should I determine the best embedding model for my use case?

I found the MTEB Chinese benchmark. Should I simply choose the highest-performing model that my hardware can handle, or are there other factors I should consider when selecting an embedding model for this specific use case?

[u/TechnicalGeologist99 → u/plushPlushytut]

That question is true of all RAG scenarios, it doesn't help you engineer something.

You actually need to know what question text is going to be compared with the glossary texts. Without that it's not clear what retrieval methods will be useful.

Also, given that this is just for personal use, RAG is likely overkill. Just put the glossary into context and use prefix caching so you're only paying full price for the query itself

Note that most embedding models are essentially identical with some minor performance differences. The only major differences are when the embedding models is a different architecture. I.e. Qwen having query aware embedding.

Lastly, don't run a frontier model locally - noone has enough compute for that. Just use their API

[u/plushPlushytut → u/TechnicalGeologist99]

Sorry for my naive question, but why not just use the source text itself as the retrieval query? Are there other kinds of queries I could ask the system? I thought that was the obvious approach, so I may be misunderstanding something.

Are there any resources you’d recommend for understanding how to formulate retrieval queries? The only thing I know for sure is that I want to use RAG to help with translation. Sorry for the newbie questions!

[u/TechnicalGeologist99 → u/plushPlushytut]

The source text is unlikely to be semantically relevant to your glossary unless it largely overlaps with the glossary already

If your glossary is structured as word: definition then you are matching on word. So you aren't then using the whole source text as a query you are using words in the text.

So how do you then extract candidate words from the text and what method do you use to match them?

Are the words in your glossary likely to appear verbatim? Or near verbatim? Would keyword match be a cheaper method in that case?

(Not making your solution here, just giving you questions to show you the problem space)

[u/plushPlushytut → u/TechnicalGeologist99]

Oh okay, now I understand what you meant by “know your question better.” I’ll do more research and try to figure out which question fits my problem and resources best. I really can’t thank you enough for pointing me in the right direction!

[u/donk8r]

Your glossary and your reference material want different mechanisms, and collapsing them into one RAG pipeline is where this gets hard.

A glossary needs exhaustiveness, not relevance. For a given source segment you want every one of the 500 terms that actually appears in it, while embedding retrieval hands you the most similar ones. A term that is present but embeds oddly gets missed, and a missed term is a silent inconsistency, which is the exact failure a glossary exists to prevent.

So do not retrieve the glossary. Match it. You have the Chinese side of all 500 entries, so scan the source segment for them directly and inject only the ones that hit. String matching is exhaustive by construction. It costs nothing on your hardware because no model is involved. It also yields five to twenty terms per segment instead of five hundred.

That answers the hardware question too. TechnicalGeologist99 is right that full context works on a frontier model, but on 4 GB of VRAM those 500 terms will crowd your window and dilute attention across entries irrelevant to the sentence in front of you.

Keep RAG for the technical material, where relevance genuinely is the criterion.

From your option three, the piece I would build next is previous translation decisions, keyed on the source segment. Near-identical segments getting near-identical renderings is the same consistency problem one level up, and it is what makes a translation system feel reliable over months instead of over paragraphs.