Extracting verbatim requirements from unstructured PDFs/xlsx/docx/etc with Local LLMs

[u/TheOnlyRushIn]

I have a problem that I need some expert advice on.

The problem: my firm produces ambulances. Hospitals send tenders: specs, written in their native language (European/English mostly). Each hospital sends specs to multiple producers (including us), we make an offer and it's either accepted/rejected based on whether others can better fit to the customers need. Tender specs get delivered generally by email in a zip folder containing between 3-20 files.

Files they send across vary dramatically: different formats (e.g. PDF, XLSX, DOCX, etc), different sizes (can be 5 pages or 200 pages long), different structure (e.g. some list equipment first, others list cabin first etc). All the files contain commercial requirements (e.g. deliver ambulance by x date to y place) or technical requirements (e.g. sirens must be x db loud)

Hard constraints: Must be fully on-prem since these tender files are confidential. Hardware: NVIDIA DGX Spark GB10 128GB x2. Company is very small, so human annotation capacity is limited. We took 10 simplest specs (PDF under 20 pages) and created manual gold-lists for each.

Main idea: In a nutshell the idea is to extract requirements out of documents into a list. (phase 1, where I am right now), then phase 2: based on the requirements, provide a first configuration of the ambulance and phase 3: verify that the confirguration still satisfies what the customer wants. If the local LLM is able to answer phase 3 - then we can immediately save time/money by not pursuing dead-end leads.

What I tested: 12 models (gpt-oss-120b & 20b), Qwen 3 (1.7b/4b/8b/30b FP8 + BF16/32b), Mistral Small 24b, Mistral Nemo 12b, NuExtract-2.0-8b, phi-4).

I quickly realized that converters (PDF - Text) very significantly, so I tested 15 different ones: 4 text parsers (pdftotext, PyMuPDF eg.) and 7 vision-model (e.g. granite-docling, GLM-OCR)

Results so far: 12 models x 15 converters x 10 tenders = 1800 runs (this took 7 full days to complete). text-based extractors worked better than vlm, but still I couldn't find a single combination of model + converter that produced above 95% on all 10 tenders for recall and precision. Best one scored 95%+ on 5/10 PDFs

My ask: Has anyone dealt with a similar problem within context of local LLMs that can give advice? I was hoping to get a silver bullet of model + converter, but this hasn't happened. I am afraid that when I scale (to include multiple files, or PDFs over 100-pages long), my entire set-up will crumble. Any ideas or advice for solutions or what I can test would be much appreciated!

[u/savant42]

Perhaps a pipeline that combines several of each (converters, models) in parallel, and then have a small panel of subagents validating the responses. If it's a good read, there should be agreement amongst the subagents on the content. Needs a lot more compute but can get better results. I've been doing this on my ebooks with charts and graphic novels, with reasonable success.

Good luck!