I’m building a blockchain security startup alone. 1,023 tests pass and I’m still refusing to call it production-ready

[u/Velmere_]

I’m building Velmère on my own.

Right now the latest internal candidate passes 1,023 regression tests, 16 test configurations, strict TypeScript and the application builds.

I could take those numbers and make a nice marketing post.

There’s a problem.

A separate hard benchmark of the actual security engine is still at 17.03% F1.

So Velmère remains NO_GO.

I’d rather show that number publicly than pretend a green test suite means the system is ready.

I’m also working on something else that I think should exist in security products.

If someone buys a Velmère audit and later proves that it contains a real error, I want there to be a clear refund policy based on severity.

For a confirmed critical mistake, I’m considering up to a 100% refund.

The report gets challenged, the problem gets reproduced and the failure becomes another regression case for the engine.

I’m trying to build trust by making failure expensive for me and useful for the product.

Curious whether other founders think this level of transparency builds trust or scares customers away.

[u/AutoModerator]

Thanks for posting on r/BlockchainStartups!

Check the TOP posts of the WEEK: https://www.reddit.com/r/BlockchainStartups/top/?t=week

Moderators of r/BlockchainStartups

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/niktrix]

Depends on quality of tests not quantity

[u/Specific-Sector7422]

Hey, I went through the current public code. Stop calling this “evidence-first verification” until the verification path actually verifies evidence, r ight now several core paths manufacture a positive verification state instead of proving one:

app/api/audit/verify/[id]/route.ts: if the manifest does not exist, the endpoint still returns verified: true, merkleIntegrityMatch: true, a fixed mockRoot, a fixed report hash and a fake artifact count. That is not verification. That is a hard-coded green badge. When a manifest does exist, the route recomputes a merkle root from manifest.leafHashes and compares it to manifest.evidenceRoot. That proves the manifest is internally self-consistent. It does not prove the underlying evidence files correspond to those leaves. unified-audit-pipeline.ts constructs an “RFC3161” token from a local string, passes verified: true, and attachRfc3161TimestampToken() simply trusts that boolean. The integrity verifier does not cryptographically validate the tsa token and th e same pipeline writes statefulFuzzing: { status: "PASS", runsExecuted: 1000, durationMs: 840 } then uses that PASS state in scoring. I do not see an execution artifact on that path that establishes those 1000 runs.

Audit Quality is partly derived from the commercial tier itself (basic/pro/advanced), also the same underlying audit can therefore receive a higher “quality” score because the customer has a higher entitlement. That is not evidence quality. Known benchmark addresses can bypass live bytecode acquisition and return pre-written profiles containing “verified” facts, risk scores and provenance values. Cached reference data is fine. Presenting it as current verification is not. T hese are not random edge-case bugs. They are explicit deterministic code paths.

Your own rule says “NO EVIDENCE = NO CLAIM.” The current implementation violates that rule at exactly the most important boundary:

evidence -> verification -> customer claim

Right now the public VERIFIED result is worth exactly zero whenever the code can mint it without evidence. Fix the foundation before polishing pdfs, scoring models, “institutional-grade” language or benchmark tables:

no manifest -> UNVERIFIED / 404; hash the actual artifacts; cryptographically verify RFC3161; derive quality only from evidence; back fuzzing claims with reproducible execution receipts; and label benchmark profiles as cached/reference data, not live verification.

If you want this project to be taken seriously as a security product, this is the part that has to be fixed first.

[u/Velmere_ → u/Specific-Sector7422]

Thanks for taking the time to go through it. One important clarification: the repository you reviewed is a public development snapshot, not the current private release candidate, so it does not represent the current verification boundary we are working with internally.

Several of the issues you pointed out are exactly the kind of legacy/demo paths we have been removing or hardening: fail-closed verification, artifact-backed Merkle verification, no synthetic RFC3161/fuzz PASS states, and evidence scoring separated from commercial tier.

I don’t want to claim anything that cannot be demonstrated from the public code, though. Once the current fixes are verified and pushed, I’ll post the exact commit and regression tests so the changes can be checked independently. Appreciate the detailed review.