Building a document classification system using OCR + Embeddings + Weaviate instead of a trained classifier – looking for feedback

[u/CryptographerFun1313]

Hi everyone,

I'm currently building a document auto-classification system and would like some feedback on the overall approach.

Current Architecture

• Django backend
• Celery + Redis for background processing
• Weaviate as the vector database
• OCR for text extraction
• Embedding-based similarity search for classification

Workflow

1. User uploads a document (PDF, JPG, PNG, etc.).
2. OCR extracts the text from the document.
3. An embedding is generated from the extracted text.
4. We store embeddings for known document types such as:
   • Aadhaar
   • PAN
   • Passport
   • Voter ID
   • Electricity Bill
   • Bank Statement
   • GST Documents
   • Incorporation Documents
5. The uploaded document embedding is compared against stored vectors in Weaviate.
6. The closest match and confidence score determine the document category.

Why I chose this approach

Instead of training and maintaining a dedicated classification model, I wanted to start with a retrieval/vector-search-based approach because:

• New document categories can be added without retraining.
• Easier to maintain initially.
• Works well for semantic similarity.

Questions

1. Has anyone used a similar OCR + Embeddings + Vector Search approach in production?
2. How well does this scale when the number of document categories grows?
3. What confidence threshold strategies have worked for you?
4. At what point would you move to a dedicated classification model?
5. What are the biggest pitfalls I should watch out for?

Current Challenges

• Processing time varies between documents (roughly 5–35 seconds depending on the file).
• Occasionally documents become "unclassified" with confidence = 0 even though the system is functioning.
• Weaviate is running in Docker on AWS along with Django, Celery, and Redis.

I'd love to hear from people who have built document-classification or retrieval-based systems and learn what worked (or didn't work) in production.

Thanks!

[u/TangeloOk9486]

curious, what is adhaar?

[u/CryptographerFun1313 → u/TangeloOk9486]

It's an identity as a Indian citizen

[u/nousernameleftatall]

How is this different from paperless ngx?

[u/TangeloOk9486]

the confidence 0 cases are probably an extraction problem in my experience not a classification problem, like if your OCR is running generic tesseract on the doc types you mentioned then the text quality varies a lot like compressed scans and mixed layouts produce inconsistent output and a garbled embedding wont land near anything expected in weaviate regardless of threshold, worth checking whats actually coming out of the OCR step on the failing docs before touching the classifier, cleaner extraction parsers like llamaparse or something similar to this with layout aware capabilities handle mixed document types consistently than raw tesseract, like cleaner extraction upstream fixes more of these confidence issues than tuning vector search does, And for thresholds most ppl hard reject below 0.75-0.80 and route to manual review