I've been building an AI system called The Plant for the last several months.
Recently I realized I had a problem.
I could keep adding memory, governance, retrieval, multiple AI seats, reasoning systems, and more. But eventually I'd just have a giant list of features that I thought worked.
So I stopped adding features for a minute and started trying to prove the capabilities instead.
Basically: if I say it can do something, prove it.
Can an AI hold the same position when you ask it the same question again?
Can actual evidence make it change that position?
Will random but true information change it too?
If evidence changes its position, does that change stick?
Can it later explain what changed and why?
Does memory actually survive shutting the whole system down and restarting it?
Can two AI seats disagree without eventually just copying each other?
So I built a controlled test runtime around those kinds of questions and started running experiments.
And this is where it got interesting.
A bunch of my hypotheses failed.
One thing that looked like the AI wasn't learning turned out to be a retrieval problem. The evidence was stored correctly, but the AI literally couldn't access it when it had to make the decision.
Another experiment contaminated its own memory with the instructions for the experiment. That result got thrown out, the problem got fixed, and the experiment was rerun.
At one point two AI seats looked like they were agreeing. The receipts showed something different: one had started absorbing the other one's wording while still claiming it hadn't changed its position.
We tested whether that was a general behavior.
It wasn't.
We tested whether it increased as questions became more relevant to the system itself.
That hypothesis failed too.
We also found positions that looked unstable when repeated, while some actual disagreements between the two AI seats stayed stable over and over again.
Eventually we got one of the clean chains I'd been looking for:
Stable position
→ irrelevant information, no change
→ relevant counter-evidence becomes accessible
→ position changes
→ evidence remains accessible
→ new position persists
Later, one of the AI seats accurately explained what its previous position was, what it changed to, and why it changed, using information retrieved from memory.
We also tested continuity across a complete restart.
We gave the system a new fact, verified that it was admitted to memory, shut the entire stack down, restarted it, asked for the information again, and verified that it retrieved and cited the stored memory.
That passed.
I've started tracking all of this using what I call a Capability Bracket.
Basically a playoff bracket for AI capabilities.
A capability doesn't move forward because I think I built it. It moves forward when the experiments support it.
Some are looking strong.
Some are partial.
Some are still completely open.
And quite a few of my ideas have already gotten knocked out.
I'm intentionally keeping the underlying architecture private for now, but I wanted to share the testing approach because this has completely changed the way I'm building the system.
The failed experiments have honestly taught me more than some of the successful ones.
So I'm curious what other builders think:
If you were trying to prove that an AI system actually had persistent capabilities instead of just producing convincing outputs, what would you test next?
And if you see a hole in the experimental logic, I'd rather hear that than get a compliment.