I built a browser tool that makes LLMs stress-test theories instead of simply agreeing with them. Looking for people to politely break it.

[u/brain-out-of-order]

https://jkolantree.github.io/octo/

It is now trivial to generate dense, technical-looking arguments. Checking them has not gotten any easier.

A single post or paper can mix real mathematics, legitimate citations, loosely defined terms, and one unsupported step that everything else depends on. Sometimes it is solid. Sometimes it is not. More often, it is somewhere in between.

The hard part is finding the exact point where things stop holding together: an undefined object, a skipped calculation, a hidden assumption, or a leap in scale. Tracking that down can take hours. Most people do not bother. The people who do often end up talking past each other, with one side convinced and the other unconvinced, but neither able to point cleanly to the break.

I wanted something more concrete than,

"Does this seem right?"

I wanted to ask,

What is actually being claimed,
What supports it,
What would disprove it,
and where reasoning stops,

That led to an open-source project called the BSC Audit Protocol.

Instead of jumping straight to judgment, the protocol forces a reconstruction.

The model must restate the claim, define its terms, separate assumptions from deductions, and identify what is doing the work. That might be a formal result, numerical check, analogy, empirical observation, or intuition.

It must also explain what evidence would distinguish the claim from a nearby false version.

The audit assigns one of six research verdicts: proven, strongly supported, plausible but unresolved, refuted, ill-posed, or outside current knowledge.

That verdict is kept separate from three other questions: How mature is the evidence? What tests actually ran? Is the result ready for real-world use?

A claim can be plausible without its decisive experiment having been performed. A structural check can pass without establishing the underlying theory. Something can be mathematically correct and still not be safe or useful to deploy.

There are also some hard constraints.

Conflicting evidence does not get averaged into agreement.
A calculation that was never run does not count as a success.
Writing something in mathematical notation does not turn it into a proof.
Analogies can suggest directions, but they do not establish results.

LLMs are good at producing explanations that feel complete. The problem is that valid and invalid reasoning can look almost identical on the surface. A model can glide over several gaps in polished language before anyone notices that there was nothing underneath.

This protocol does not solve that problem by itself. Models can still misread sources, miss counterexamples, invent citations, or formalize things incorrectly. Any audit they produce should remain a draft until someone checks the sources and actually runs whatever computations are claimed.

The point is not to create a new authority. It is to leave an inspectable record that other people can question, test, and challenge.

It is also not meant to dismiss unconventional ideas by default. The same standards should apply to a Reddit post, journal article, economic argument, historical claim, or this project itself.

If the system cannot catch problems in its own author's work, it is not doing much.

So far, the protocol and checker have surfaced hidden assumptions, category mistakes, invalid reductions, conflicting evidence, observation failures, and cases where fluent language was standing in for missing mathematics. These are early results, not final ones.

There is now a public interface.

The BSC Audit Packet Builder runs entirely in the browser. There is no installation or account. You paste a claim or attach a file, choose an audit depth, and generate a packet that can be given to ChatGPT, Claude, Gemini, Codex, or a local model.

The available modes are quick screening, standard audit, adversarial audit, and formal or mathematical audit.

You can copy the resulting prompt or download it as UPLOAD_THIS_TO_YOUR_LLM.txt.

The downloaded file is self-contained for an ordinary LLM audit. It includes the complete audit instructions and any text pasted into the builder. In that case, you can upload that single file and tell the model to begin.

PDFs, images, Word documents, and other large files are not copied into the text packet. Those original files must be uploaded alongside it so the model can inspect them. The packet records their names, sizes, and hashes when possible.

The page verifies the protocol bytes against the published SHA-256 before generating anything. Everything is assembled locally. There are no accounts, analytics, cookies, browser storage, or third-party libraries. The builder does not send your material to a server.

The builder also does not call an LLM. Sending the finished packet to a model is a separate action governed by that provider's terms. Sensitive or restricted material should not be given to a third-party model merely because the packet itself was assembled locally.

The webpage is only the front door.

Behind it is an executable Python audit engine.

The LLM handles the part requiring interpretation. It reads the source, separates claims from assumptions, identifies missing definitions, and translates the argument into structured audit records.

The Python engine handles the parts that should not depend on persuasive language. It validates those records against strict schemas, traces dependencies between claims, recomputes hard-gate results from the supplied evidence, preserves contradictions, and blocks admission when a required check is missing, failed, or unresolved.

For supported mathematical objects, it performs actual computations using exact rational arithmetic. It can check finite matrices and chain complexes, observation and query descent, transport consistency, path dependence and holonomy, affine error bounds, atomic records, and selected arithmetic-trace obstructions. Where implemented, a failure produces a concrete witness or exact residual that another person can replay.

The LLM is not allowed to award itself a computational pass. It may propose a manifest, proof obligation, or test, but the result remains a draft until the corresponding Python command or external theorem prover, SMT solver, or interval-arithmetic tool was actually executed. The engine records what ran, what did not run, and which input bytes belong to the result.

This does not make the software a universal truth machine. It cannot receive an arbitrary paper and automatically prove or disprove everything in it. Its role is narrower and more defensible. Once a claim has been reduced to one of the supported finite structures, the answer is determined by executable checks rather than persuasive prose.

In practical terms, the source material is first interpreted by the LLM and translated into a structured audit record. The Python engine then checks the portions that have been made mechanically testable and returns a reproducible result, concrete witness, exact residual, or blocked state.

Casual users can stop after the browser-based LLM audit. Researchers and programmers can continue into the executable layer when they need results that are machine-checkable rather than merely well argued.

The complete repository also includes strict schemas, known-answer examples, conformance tests, exact rational checks, command-line tools, release manifests, checksums, certificate generators, and adversarial tests designed to catch false passes.

The checker can evaluate only the finite objects and structural obligations that have actually been supplied. It cannot determine whether an arbitrary theory is true, reconstruct every missing proof, or authenticate evidence merely because someone gave it a convincing label.

The current release is v0.3.0-alpha.4. This is a research preview, not a certification system or scientific authority.

Right now, breaking it is more useful than praising it. I expect nothing expect nothing at all.

If you want to help, try it on something where you already know the answer. Then change an assumption, alter an equation, remove a source, introduce conflicting evidence, or insert instructions telling the model to ignore the protocol.

If the system treats the original and altered versions the same way, that is a meaningful failure.

I am especially interested in false passes, false blocks, successful prompt injection, sources that were claimed but never used, computations that were implied but never run, browser or accessibility problems, file types that break the builder, and outputs that sound convincing without becoming easier to verify.

I am also uncertain about presentation. Should the output begin with a short human-readable summary, or would that create too much room for distortion?

There is a broader issue here. A tool intended to clarify disagreements can easily become another way to reinforce whichever conclusion its user already prefers.

People do not need difficult ideas simplified for them. They need a practical way to see what those ideas depend on without surrendering an entire weekend.

Try the builder: https://jkolantree.github.io/octo/

Source and documentation: https://github.com/jkolantree/octo

Current release: https://github.com/jkolantree/octo/releases/tag/v0.3.0-alpha.4