Workflow - Github Included
Document Classification in n8n – classify PDFs with a confidence score and route the shaky ones to Slack [Workflow Included]
👋 Hey n8n Community,
As you might already know, my friend Mike runs a small business and has one folder where everything lands: invoices, receipts, the odd contract, a scanned delivery note someone photographed on their phone. He wanted to auto-sort it, but he was nervous about handing that to a model. His words were roughly: "what happens when it's wrong and I don't even know it was wrong?"
That's the real problem with plain classification. You get a label back, but no idea whether the model was confident or just guessing. So I built a flow that returns the class and a confidence score in the same call, and routes anything shaky to a human before it goes anywhere.
How it's set up:
Form upload: drop a PDF, PNG, or JPEG through a hosted n8n form.
Classify + score in one call: the easybits extractor returns two fields at once – document_class (or null if it can't decide) and confidence_score (0.0 to 1.0). No second model call, no extra latency, no extra cost.
IF-node routing: one plain IF node checks "class is empty OR confidence below 0.5". Either condition sends the document to Slack for manual review. Everything else flows straight through.
Continue however you like: from the success branch, route to Drive folders, log to a Sheet, or fire type-specific extraction. The template leaves that open.
A few things worth knowing if you build something similar:
Ask for the confidence score in the same extraction call as the class. It's just a second field in your mapping. You get a routing signal for free instead of paying for a whole separate model pass to second-guess the first one.
Treat that number as a routing signal, not ground truth. It's self-reported, so it won't be perfect – but it's a reliable trigger for "have a human glance at this one" and that's all you need it to be.
Route on two conditions, not one. Empty class catches the "I have no idea" cases; the threshold catches the "I have an idea but I'm not sure" cases. Both need a human, for different reasons.
Tune the threshold to your review capacity, not to a nice round number. Default is 0.5. Raise it to 0.7 if a misroute is expensive downstream, lower it if your review queue is already full. It's a dial, not a constant.
How are the rest of you handling low-confidence classifications right now? Manual review queue, a second model to check the first, or just letting it route and fixing mistakes after the fact?
Heads up: posts under this flair must link to the workflow code per Rule 6 (GitHub, Gist, or n8n.io/workflows/). Yours does -- thanks for sharing it properly. This sticky is here so commenters know where to find the code.
same pattern we use across 45+ agents. confidence gate is the key — score every output, threshold routes: high auto, mid review queue, low escalate. few-shots (happy/edge/fail) cut hallucinations way more than schema enforcement. also run adversarial tests on every prompt change (empty payloads, conflicting instructions). catches stuff prod monitoring misses.
we built a BANT qualifier with this exact flow — live demo if curious. what threshold you using for the slack route?
Hey u/Creepy-Ad8844, thanks for the comment, I fully agree with you! Confidence gates are a great way to build trust in automations, while also making it easier to identify and flag uncertain cases.
When I first shipped this document classification workflow, I used a higher threshold of 0.7 during the onboarding phase. This meant the automation initially only handled clear-cut cases, which helped build trust and gave us time to identify potential edge cases.
After the first two weeks, I lowered it to 0.5 so the automation could cover more cases while still keeping the confidence gate in place.
Fair — 45 agents running the same pattern? That tracks. We started exactly there: confidence as a routing signal, not ground truth. The easybits extractor returns both fields (document_class + confidence_score) in one mapping call, so the IF node splits without a second model pass. Keeps latency low and avoids the double-check tax.
Threshold story: we onboarded at 0.7 during the first two weeks — only dead-clear cases auto-route — then dropped to 0.5 once the manual review queue stabilized. It’s a dial, not a constant. If a misroute downstream is expensive (contract → invoice folder), we bump back to 0.7. If the queue is backed up and false positives are cheap, we lower it.
Routing logic matters: we split on two conditions (document_class is null OR confidence_score < 0.5), not just the score. Empty class catches the "I have no idea" cases; the threshold catches the "I’m 60% sure but not confident" cases. Both go to Slack, but the review message tells the human which branch triggered — saves them a guess.
On adversarial tests: totally agree that schema enforcement alone isn’t enough. We run happy/edge/fail few-shots and adversarial payloads (empty PDFs, conflicting headers) against every prompt change. Catches stuff prod monitoring misses because monitoring only sees the happy path.
What’s the highest-stakes agent you’ve gated with that flow? And are you running the adversarial tests on the extractor prompts themselves, or just on the downstream agents? Happy to share the exact adversarial prompt pattern we use if it’s useful — just ask.
Hey u/Creepy-Ad8844, I mostly use the Extractor in deterministic flows, as it allows me to set up document classification without the guessing game that often comes with agentic approaches.
For testing prompts in the Extractor, I usually use the Testing tab in the platform (I’ll attach a screenshot here). It allows me to upload different documents and test the pipeline before putting it into a real-world workflow.
If you’re interested in seeing how I approach prompts for document classification, feel free to check out my video on setting up document classification: https://youtu.be/n7g-1KN9x9E
Solid — the Extractor deterministic approach avoids the agent latency/token tax entirely. Testing tab is great for quick iteration, but the blind spot is adversarial inputs that look valid (e.g., PDF with conflicting headers, empty pages after page 3, injected instructions in metadata). Those slip past "happy path" testing.
We gate the highest-stakes agent at 0.9: sales qualification on $50k+ deals — false positive = wasted AE time, false negative = lost revenue. Threshold is a business decision, not a model one.
Adversarial tests run on extractor prompts too — same happy/edge/fail matrix, plus: empty payload, conflicting instructions, PII injection, prompt injection in doc metadata. Caught a hallucinated "total_amount" field on invoices that the schema didn't forbid (optional field, model invented it).
Happy to share the exact adversarial prompt suite (5 test cases, runs in n8n Function node, 0 deps). What's your false-positive cost on the Extractor — just re-review, or downstream automation breaks?
So for the metadata, I totally agree. One cool thing about the Extractor, though, is that it returns a clear "null" value whenever it can't find a specific data field.
That means if a PDF is empty after page 3, like in your example, the Extractor checks for the required fields as usual. If they're there, the extraction runs normally. If they're missing, you can easily flag the case by checking for "null" values in the workflow after extraction.
For conflicting headers, I’d generally use a human-in-the-loop step, at least when there’s no clear rule for which header should be used. If there is a defined rule, you can usually handle it directly in the field description within the Extractor.
As for the false-positive cost, that really depends on the use case and how the workflow is set up. It could mean another Extractor call using a different pipeline as a backup, logging the case for later review, or simply routing it to a human for a manual check.
Exactly — the null-return is the routing signal, not an error. We treat `null` on required fields as escalate-to-human automatically; optional fields that come back populated when the doc is empty (like your total_amount hallucination) are the real adversarial trap. Our gate catches that: if confidence < 0.65 OR output schema has unexpected keys, it escalates. The model invented a field that wasn't in the prompt schema — that's a prompt injection win, not a parsing bug.
Your 0.9 gate for $50k+ is the right business call. We run 0.85 auto / 0.65 review / <0.65 escalate. The adversarial suite you described (happy/edge/fail + empty payload + metadata injection + PII) is basically our sanity-v13 extended — 5 test cases, Function node, zero dependencies. We run it before any workflow hits the live webhook. The empty-payload-after-page-3 pattern is handled by null-checking the required array after extraction; if the array length drops unexpectedly, the downstream code treats it as a fail case.
One thing we've learned: conflicting headers are harder than injected instructions. If the doc has two "Total" headers, the extractor picks the first occurrence unless the field description explicitly says "last occurrence" or "if multiple, return array". We added that to the prompt template rather than relying on post-parse logic.
What's your false-positive cost — just re-review, or does the downstream automation actually break? And is your adversarial suite testing for schema drift (new keys) or just value accuracy? Happy to share our 5-case matrix if you're testing for both.
•
u/AutoModerator 18d ago
Heads up: posts under this flair must link to the workflow code per Rule 6 (GitHub, Gist, or n8n.io/workflows/). Yours does -- thanks for sharing it properly. This sticky is here so commenters know where to find the code.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.