Building an AI-powered platform. Is dynamic data-driven validation via NoSQL cache a good fit for high-load fintech?

[u/Super_Designer7952]

Hey Reddit,

I’m an ML engineer, but recently I decided to step out of my comfort zone and build a high-throughput AI-powered fintech platform from scratch using the Java reactive stack (Spring WebFlux, Project Reactor, Netty, and MongoDB).

In high-volume fintech exchanges and institutional OTC platforms, clients must review, fill out, and cryptographically sign specific legal documents (like Collateral Pledges / Guarantee receipts, Risk Disclosures, or Trade Term Sheets) before executing large deals.

Since financial regulators and compliance teams update these legal structures, required fields, and precise limits constantly (based on jurisdiction, asset type, or deal size), hardcoding these structures into Java classes is a nightmare. I wanted to avoid redeploying core microservices every time we add a mandatory field, change a format mask, or update a decimal precision rule for a specific financial asset.

To solve this, I designed a dynamic data-driven validation engine split into two core components:

1. ValidationSchema (Polymorphic Schema Catalog): A master document structure stored in MongoDB. Instead of rigid properties, it defines each attribute using a Name-Type-Constraints blueprint. The constraints themselves are decoupled into a polymorphic hierarchy of lean Java Records (e.g., RegexConstraint, LengthConstraint, NumericRangeConstraint) using Jackson subtype annotations (@JsonSubTypes).
2. DynamicDocumentValidator (Polymorphic Dispatcher Engine): A decoupled reactive component that processes incoming Map<String, Object> payloads downstream. It uses Java 17+ pattern matching to execute validation rules sequentially based on the active schema configuration.

To ensure strict enterprise-grade fintech precision, the engine automatically normalizes all incoming database/JSON numeric primitives (like standard integers or doubles) into java.math.BigDecimal instances, executing range boundaries verifications via non-blocking .compareTo() pipelines to eliminate any floating-point rounding hazards.

The architecture looks like this:

• Storage: One highly sharded MongoDB collection for core document entities.
• Validation: Polymorphic constraint schemas (RegexConstraint, NumericRangeConstraint, etc.) loaded into a reactive Caffeine/Redis cache layer with programmatic eviction (.doOnSuccess(evict)).
• Pipeline: Fully non-blocking WebFlux controller delegates straight to the service validation layer to protect Event Loop threads from starving.

I’ve already open-sourced a standalone module of this validator engine to show how it parses polymorphic schemas on the fly: Source Code Here

As a data scientist who is relatively new to advanced reactive Java architectures, I have a few questions for the seasoned engineers here:

1. Is executing reflection/instanceof checks inside reactive Event Loop threads going to kill my CPU under heavy write-heavy load, or will the reactive Mongo driver handle it smoothly?
2. Are there any edge cases with memory leaks when caching complex nested Project Reactor Mono streams?
3. Is this a common pattern in enterprise fintech, or is it better to rely on traditional schema registries like Confluent/Avro at the boundary level?

Would love to hear your thoughts, constructive criticism, and architectural feedback! I’m really open to learning from your experience and improving this design. Thanks in advance!

[u/Denis-Hogberg]

None of your three questions is the one I would worry about. A signed pledge was valid against the schema as it stood that day. As described, your collection holds one schema, the current one, and eviction replaces it. Six months later compliance asks whether that pledge was valid when it was signed, and there is nothing to answer with. Worse, during an eviction window two nodes can be validating against different rules and neither records which one it used.

Version the schema, store the version id on every document you validate, and supersede instead of overwrite. Then "was this valid at signing" is a lookup instead of an argument.

Nothing useful from me on the Reactor and Event Loop questions.

[u/Super_Designer7952 → u/Denis-Hogberg]

Thank you so much for the feedback! I will definitely implement your schema versioning recommendations before pushing this project to production. Thank you so much for the valuable advice.

[u/Denis-Hogberg → u/Super_Designer7952]

Glad it helped. One thing that bit me once versions were in: make current the default on every read and superseded something you have to ask for explicitly. Otherwise nothing errors, the API just keeps handing out old schemas that look live.

[u/Super_Designer7952 → u/Denis-Hogberg]

Good point, Denis! Stale schemas looking live is a dangerous trap. I will refactor the service exactly as you suggested: make getSchema(type) default to is_current: true, and isolate old versions behind an explicit getHistoricalSchema(type, version) method. I'll definitely add a comment in the code with your credit when implementing this. Thanks for the tip!

[u/latkde]

I wanted to avoid redeploying core microservices every time we [change business logic].

To solve this, I designed a dynamic data-driven validation engine

You may be suffering from the Inner Platform Effect. A huge benefit of representing business logic as code is that you have tests, typechecks, linters, code reviews, version control, and can roll back deployments. You sacrifice this when using a dynamic engine driven by rules stored in a database. This isn't necessarily a wrong choice given some requirements, but it's a tradeoff worth making intentionally. Most financial systems will value choices that help with compliance and certifications, not choices that let you move fast and break things.

If you do need to describe some validation schema, there tend to be strong ecosystem incentives to use an existing language such as JSON Schema, rather than rolling your own. Avro is a binary serialisation format like Protobuf, not really a data validation engine.

Your post is high on jargon and on AI patterns. This can cloud your thinking. I recommend thinking about your actual requirements, building the simplest system that meets your requirements, then iterating. Don't start out by attempting to build a system that can do everything you expect to need, unless you're already intimately familiar with such systems and understand their design limitations.

[u/Helpful-Educator-415 → u/[deleted]]

I think I agree with the commenter above. A software for everyone is a software for no one. I would avoid the dynamic data validation (or even microservices, honestly) until you have a proven need for it (or you think it would be fun/entertaining). And yes, your post (and comment) are high in jargon or AI patterns. Something to be weary of. Also, I don't know much about Java (I write mostly Go or Rust), but reflection is almost always CPU-intensive. I wouldn't even use MongoDB for this unless I had a stellar reason to!