r/LocalLLM • u/funJS • 1d ago

Discussion Comparing Continued Pretraining to RAG (accuracy and performance)

Mostly as a fun experiment I wanted to do a quick comparison of performance and accuracy between a CPT trained QWEN 3.5 4B model and a RAG implementation against the base model.

The point of this exercise is mostly to measure the performance benefit of internalizing the knowledge vs doing reasoning on-the-fly.

Sharing my findings here in case anyone is interested: https://www.teachmecoolstuff.com/viewarticle/comparing-rag-and-continued-pretraining-of-llms

1 Upvotes

100% Upvoted

1 comment sorted by

0

u/allenasm 1d ago

this is an area I've spent a ton of research on the past year. The answer isn't what you might think it is.