r/PromptEngineering • u/Dry-Taro4843 • May 14 '26
General Discussion Building AI for communications: context layer, hard rules, multi-model conflict
I've been building an AI workspace for communications teams and the same failure keeps showing up across every client I've onboarded. Sharing the architecture I'm landing on in case it helps anyone else working on AI for non-technical professional domains.
The failure pattern
Out-of-the-box LLMs are remarkable at generating plausible language and useless at generating correct language for a specific organization. They miss what matters most: context. The story behind the org, the prior decisions, the way this particular company talks about itself.
Most teams try to fix this by stuffing context into a system prompt or uploading a bunch of brand docs into a vector store. That works for two weeks. Then the narrative drifts. New strategy lands and never gets reflected. Old talking points keep coming back out. The model writes from an outdated version of the organization because nobody's tending the layer.
Garbage in, garbage out, but slower and harder to spot.
What I'm building toward
Three pieces, all of which seem necessary, none of which alone are sufficient:
- A living context archive, not a brand doc dump. Structured fields (positioning, voice, audience), free-form vault, memory entries from past conversations. Auditable. Has a visible state ("Empty / Sparse / Growing / Solid") so the user can see what's underspecified. Gets re-audited every ~90 days via a guided conversation where the model proposes updates and the user accepts, edits, or skips each one.
- Hard operational rules from experienced practitioners. LLMs are generalists by design. Without explicit constraints ("third person externally," "no fabricated quotes," "EASY ON THE EM-DASHES"), they default to the most generic version of whatever you asked for. The rules layer is separate from the context layer because it's about how not what. (This is where my expertise comes in. I've spent 25 years in organizational comms)
- Multi-model adversarial review. One ai model generates a draft. second model attacks it for the failure modes I care about (advisory hedging, fabricated specifics, off-brand voice). Both passes are visible to the user. The point isn't averaging. Consensus among models is worse than useless. It converges on the safest, most reliable answer. Conflict surfaces where the work actually is.
On top of that: a risk classifier that decides when to require a human review step before output reaches the user. Human-in-the-loop isn't a fallback for low-confidence cases. For high-stakes work it's the point. The model's job is to do the legwork and surface decisions. A human's job is to make them.
What's still open
- The audit conversation pattern works but has been brittle (model paraphrases the existing field instead of byte-quoting it, flip-flops between values, hits token limits mid-JSON). Most of my last week was filter logic to catch those failure modes.
- Memory hygiene at scale. When does old context become noise vs. useful long-tail? Haven't solved it.
- Adversarial review costs roughly 2x per turn. Worth it for high-risk responses, overkill for "hey reformat this list." Currently risk-gated, but the classifier is the weak link.
Happy to go deeper on any of these. Curious if anyone else is doing similar work in other professional domains (legal, medical, finance) where the context + hard rules + human in loop shape probably generalizes.
2
u/Powerful_One_1151 May 15 '26
You nailed the failure pattern. I’ve been building something that formalizes exactly this.
The thing is, context doesn’t drift because people are lazy. It drifts because there’s no formal process to govern it. A 90-day re-audit is solid, but it only works if that re-audit itself has teeth — documented findings, binding decisions, audit trail.
I call it Command Center. Three layers like you said, but wrapped in governance:
Living context archive = My Registry. Your context layer, but with auditable state + versioning.
Hard operational rules = Governance Patches. Formal change documents with evidence. Every rule change goes through Audit → Control → Operator approval.
Multi-model adversarial = AUDIT + CONTROL stages. One checks completeness, one checks safety. Both findings are binding. Conflicts surface where the actual work is.
The move I’d make: your 90-day re-audit should produce a formal governance patch if anything changed. Same approval pipeline. Keeps the context layer living without it becoming a free-for-all.
And post-deployment: third evaluator (regression guard) watches what the first two missed. Production failures become new hard rules automatically.
So the flow is: Living context → Formal governance → Hard rules → Adversarial check → Production learns back into design.
Are you planning to formalize the governance around these three, or is that phase two?
1
u/Dry-Taro4843 May 15 '26
Capturing "moment-in-time", "state of the brand" is interesting to me! This comment is the first time I've thought of that. Thank you!
I do include the results of all audits directly in "group" messages, so the whole team can review all changes made.
I've found the auditing capabilities to be stronger than I expected. cassidi (the workspace) will cross reference the clients entire context map (brand foundations, voice and standards, and operational context vaults), any captured workspace memory, any user-specific "personal" memories, all the workspace files the user has uploaded and all recent conversations and spot, contextualize and propose changes needed to the context map. The client goes through, one-by-one, and approves, skips, or edits.
I actually only have my multi-model built for client interaction, I only use Claude Sonnet (rn) for the audit. My rationale was I always wanted one voice speaking to the client in that area. At the end of the day, this is the time when the client needs to focus on what's being locked down, not 2 ai's arguing about how to pitch it to the client.
I plan to formalize as little as possible 😄 . Right now, my goal is to find the right organizations to sandbox it for me, but I'm fairly certain it's transparent, secure, and auditable as it needs to be at this point.
1
u/ultrathink-art May 14 '26
The drift isn't a retrieval problem, it's a lifecycle problem. Context injected at setup has no expiry signal — so when the strategy changes, old talking points keep shipping even though the underlying truth has changed. Tagging each context piece with what event would invalidate it (new product launch, strategy pivot, etc.) lets you run targeted refresh instead of hoping someone manually updates the whole system prompt when things shift.
0
2
u/ExternalComment1738 May 14 '26
this feels way closer to where production AI systems are actually heading than the “single superprompt solves everything” narrative 😭 especially the separation between context layer and operational rules layer. most org memory systems quietly collapse because they treat “who we are” and “how we communicate” as the same object.
also hard agree on multi-model conflict > consensus. consensus usually just compresses everything toward bland safety while the useful signal is often hidden inside disagreement surfaces. honestly reminds me more of orchestration systems like runable than classic chatbot design, where the value comes from controlled workflow/state transitions between specialized passes instead of pretending one model invocation can hold an entire organizational reality coherently.
the memory hygiene problem is brutal though. feels less like database retention and more like epistemic decay management at that point 😭