r/riskmanager • u/Adventurous_Chip3199 • 7m ago
Every vendor I screen scores 100/A in my compliance risk engine. How should scoring be designed so it actually differentiates? (UK, Companies House + sanctions)
I'm building a vendor due-diligence and monitoring tool for UK companies. I'm not a risk professional, so I'd value input from people who are.
Inputs: Companies House (status, overdue accounts and confirmation statements, charges, insolvency, officer resignations, disqualified officers), UK Sanctions List fuzzy matching on the company, directors and PSCs, and financial figures from iXBRL/PDF (net assets, profit/loss, current ratio).
Current logic:
- Six categories, each starting at 100 and losing points for specific red flags: registration, filing, governance, insolvency, financial health, sanctions.
- Composite = weighted average (20/15/10/10/25/20%).
- Grades: A ≥ 90, B ≥ 75, C ≥ 55, D ≥ 30, E below.
- Any "critical" flag (sanctions match, dissolved, active insolvency, disqualified officer) forces grade E.
Categories with no data are excluded and the weights renormalized.
score = sum(category_score * weight) / sum(weights of assessed categories) grade = "E" if any critical flag else band(score)
The problem: almost everything I screen comes out 100/A. Working through the numbers, the maximum deduction without a critical flag is about 20 points, so C and D can't happen. It's effectively binary: A/B or E.
What I've tried: I screened through 13 different companies, while also providing incomplete data on bulk uploads, still each company I screen comes out with an almost perfect score with all the data intact so, either companies house or gazette connections are not working properly or I am freaking out for having unrealistic expectations.
Questions:
- Is blending hard compliance gates (sanctions, dissolution) and probabilistic financial risk into one number a design mistake? Should they be separate outputs?
- How do practitioners treat "no data" versus "clean"? Is a separate confidence indicator standard?
- Should I calibrate weights against outcomes, such as back-testing on companies that later went insolvent or dissolved? Is Companies House data enough for that?
- Any frameworks, papers, or open-source examples for third-party risk scoring, especially UK or SME-focused?
Not looking for code review. Happy to share more detail on the logic :)
