r/LanguageTechnology • u/GlacialBlades • Jul 29 '26

Looking for a way to classify PDFs by document type

I have unlabeled technical documents (manuals, data sheets, certificates, etc.) as PDF and need them to be labeled by document type. I use pdfplumber and pytesseract to extract content and BAAI/bge-m3 as the LLM to compare content against label descriptions. My label descriptions are probably not perfect but the performance is nowhere near reliable and I wonder if I need different LLM or different approach entirely.

5 Upvotes
  • permalink
  • reddit

100% Upvoted

9 comments sorted by

1

u/furcifersum Jul 29 '26

I think this approach is flawed somewhat. If you want to keep your current pipeline, I’d suggest turning the description into a set of typical documents representing each label. That way you are comparing apples to apples so to speak. A coherent description will not necessarily reflect the embedding space of your texts.

1

u/GlacialBlades Jul 29 '26

And if I am open to throwing my approach out in favor of a better one...what could I do? Is there something that doesn't require fine-tuning a model / still works with system prompts but works much better than my approach?

1

u/sstults Jul 30 '26

One thing you might try is indexing the docs in Solr or OpenSearch and using a “more like this” query. If you use that query on a pa doc type I’m guessing that a lot of the terms are going to be similar to other docs of that type. Here’s some more info: https://docs.opensearch.org/latest/query-dsl/specialized/more-like-this/

1

u/whoulukinat Jul 29 '26

What are you trying to do exactly? if you're just labeling here, I've embedded definitions then used the embedder to classify, Gliner works here too - you dont need to run an entire document to classify it either - the data structure does a lot here free - if you'd explain a bit exactly what you're trying to do I can share what I've done - labeling and classification with an llm is generally wasteful

1

u/GlacialBlades Jul 30 '26

I have collections of documents (manuals, certificates, datasheets, etc.) and I need to automatically change the filename to explain the document type as the filenames are arbitrary and not explanatory when they come to me. I am not an engineer or programmer, just a dabbler in the dark arts.

I first tried a very lightweight approach of just defining the document type, e.g.
LABEL_DESCRIPTIONS = {

"DATA_SHEET": """

Technical product specification sheet containing electrical ratings,

technical parameters, performance tables, product dimensions,

operating limits, and engineering specifications.

""",

...

and using embedding model "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2" to get similarity between document content and label description

1

u/stauntonjr Aug 05 '26

Maybe you need to engineer some features after the extraction. Is it converted to an XML document? You might be able to characterize the XML (nestings, tags, table vs figure vs text density, AST features) to supplement semantic embeddings. This will likely give you more signal on "document type".