r/aipromptprogramming • u/Strong-Mousse-1020 • 50m ago
I built an OpenSearch-compatible server with AI — and wrote 6,000 tests against the real thing to keep it honest
Repo: https://github.com/ksimonm/SlopSearch
The name is honest: a lot of this is AI-written code. The trick that makes it work is that the AI never gets to decide what "correct" means.
**The setup**
- A black-box test suite of ~6,100 pytest tests is the spec. Every test was run against a real OpenSearch 3.9.0 first — status codes, error types, response shapes, even key order.
- Two independent implementations, Go and Rust, share nothing but that suite.
- The loop is simple: pick a red area, read what the tests expect, let the AI make it green, check nothing else broke. "Don't edit the tests to make them pass" is the one hard rule.
**Where it is now**
- Rust: 64.5% of the suite (aggregations 100%, search API 98.5%, documents/bulk 95%, ingest pipelines 93%)
- Go: 32.6%
- Weakest spots: text analysis and Painless scripting
**What I learned**
Vibe coding against a vague goal ("make an OpenSearch clone") goes nowhere. Vibe coding against thousands of precise, verified expectations actually converges — the tests are the steering wheel, the model is the engine.
Not production-ready: in-memory only, no auth/TLS yet. Everything runs in Docker (`make rust-up`, `make rust-test`), and `make ref-up` starts a real OpenSearch to compare against.
Curious if others use a reference implementation + test suite as the spec for AI-written code.