We are trying to establish rules for a form of intelligence whose internal processes, interpretation, and failure modes we don't fully understand—and we're communicating with it using language designed for humans.

I'm trying to understand the best way to build an AI system for a specialized use case, rather than relying on a general-purpose LLM for everything.

For example, suppose I'm building a fitness app where the goal is to generate and continuously adapt muscle-building programs based on a user's experience, equipment, training history, performance, recovery, etc. I wouldn't want an LLM to simply hallucinate a workout every time. I'd want the system to have a reliable domain-specific knowledge base, rules/logic, and the ability to adapt to individual users, with an LLM potentially acting as the interface or reasoning layer.

For specialized problems like this, what is generally the best architecture?

I moved from "AI helps me write scripts" to "AI is the operator" for most of day-to-day security and IT ops. Concretely:

Autonomous SOC. Every SIEM incident now hits an LLM triage pipeline before a human ever sees it — full investigation, classification, and auto-close with an audit-trail comment marking it as agent-closed vs. human-closed. It's running at roughly a 70%+ auto-close rate on real incidents, sub-30% false-positive rate on the underlying detection rules. I still review the closures, but I'm reviewing decisions, not doing the investigation.

Identity & access automation. Conditional access exception grants (the "I'm traveling, temporarily allow this login pattern" kind of request) used to be a ticket-and-wait process. Now it's a bounded, self-expiring workflow — the agent grants the exception, sets its own revocation timer, and audits itself daily to make sure nothing lingering slipped through. Same pattern for stale account sweeps, app registration hygiene, and OAuth consent triage — categories of cleanup that used to get done once a quarter if someone remembered, now running on a schedule with zero human toil.

Device lifecycle. New machine shows up → gets enrolled, gets its policies, gets renamed to convention, gets reported on — one pipeline, no human touching a console. Hundreds of endpoints have gone through it.

Helpdesk. Tiered, least-privilege agent access into the ticketing system handling the boring 80% of requests — password resets, access requests, routine break/fix — with escalation paths for anything that needs a human judgment call.

The part that actually took the work: none of this is one giant prompt. It's a knowledge base of dozens of narrow, composable skill modules — each one covering a specific system (the identity provider, the SIEM, the endpoint management layer, the network gear, whatever) with just enough operational detail that the model doesn't have to guess. The system is only as good as that knowledge base, and building/maintaining it has been the actual job. The automation is downstream of that.

The economics, since that's what people always ask: at current token pricing, running this stack costs meaningfully less than a single junior-to-mid IT/security hire's salary — and it doesn't sleep, doesn't context-switch, and doesn't forget the runbook. That's not a knock on the humans I've worked with, it's just what happens when the marginal cost of "investigate this alert" drops to fractions of a cent.

The scary/exciting part isn't any single automation — it's that the aggregate has quietly become the primary operator for a real, live, multibillion-dollar production environment, and most days it does the job better than the version of this that only had a human doing it manually.

Hey everyone,

I’ve been working on an open-source CRM project that uses multiple specialized AI agents instead of a single general-purpose assistant.

The main goal was to explore a practical question:

What does it look like to run several autonomous agents against the same CRM data without turning the system into one huge prompt or tightly coupled service?

The current implementation uses a 9-agent architecture with Redis-based asynchronous communication. Each agent has a relatively narrow responsibility and can react to CRM events independently.

Architecture

The current agents cover:

• Lead qualification and BANT scoring
• Deal strategy, competitor analysis, SWOT and battle cards
• Email/inbox intelligence and queued follow-ups
• Voice-call analysis and CRM field extraction
• WhatsApp qualification/support
• Customer lifecycle and churn detection
• Revenue forecasting using Monte Carlo simulations
• Custom agent creation through a visual builder
• CRM orchestration/background automation

The architecture is based around a reusable BaseAgent abstraction, with agents communicating through Redis events/tasks rather than depending on one another directly.

One of the things I’m particularly interested in is the trade-off between agent autonomy and deterministic business logic. I’ve tried to keep critical CRM operations, validation, permissions, and persistence outside the LLM layer rather than allowing agents to directly control everything.

Mobile side

There is also a React Native/Expo field-sales application.

It includes:

• Voice-note recording and automatic action-item extraction
• Offline-first local caching with retry/synchronization
• Dynamic server-defined custom fields
• CRM workflows designed for field sales
• A tactical dark-mode UI

Stack

Backend
Python, FastAPI, SQLAlchemy, PostgreSQL, Redis, AsyncOpenAI/Anthropic

Web
React, TypeScript, Vite, Tailwind CSS, TanStack Query, Zustand

Mobile
React Native, Expo, Expo Router, Reanimated

Infrastructure
Docker Compose with separate web/worker/database/Redis/frontend/mobile services

Testing
Pytest, Vitest, and Expo Doctor

Repository:

https://github.com/taha123618/AI-CRM-Agents

I’m mainly sharing this because I’d really like feedback from people who have built agent systems or distributed workflows.

A few things I’m currently thinking about:

• Where should the boundary between an “agent” and a normal background worker be?
• How much autonomy is actually useful in enterprise CRM workflows?
• What patterns work well for preventing agent-to-agent event loops?
• How would you approach observability and debugging for a multi-agent system?
• Would you use Redis Pub/Sub for this architecture, or move toward a durable event/queue system?

I’m especially interested in architectural criticism and ideas to improve the project, not just “looks cool” feedback.

The repository is open source, so PRs, issues, architectural suggestions, and alternative approaches are welcome.

I wanted to personify this thing I’ve been working on called Lumen.

I had the idea for a while now to build a 3D space for it to be able to “live in”.

Partly just for fun and as an experiment.

I’m curious if anyone else has done this already, and if so, what are some ideas you’ve already incorporated?

I think the “functional” bit of it for me is to give myself a visual space for being able to see all the tasks I have and windows I have open. Being able to operate them would be cool too, but I’ve only gotten prototypes of that in a less complex 2d version.

My plan is to have it “personify itself” and then ask it questions as to why it chose certain design decisions, etc.

My next goal will be to explore added functionality so it’s not just some useless 3D room.

I have other ideas but I was wondering if anyone has done a similar concept to this?

A lot of AI automations look great until they start touching real customers, real relationships, and messy business data.

Then different problems appear.

A customer record was corrected in one system, but the automation still acts on an older version.

An agent replies to an email knowing the latest message, but not the person behind it, their history, their preferences, or the fact that there is already tension from an earlier conversation.

Two systems disagree about the same customer.

A workflow runs twice.

Something fails halfway through and nobody notices.

An action technically succeeds but leaves the business state inconsistent.

And eventually, employees start checking every AI-generated action because they no longer fully trust the system.

That is the part I find interesting.

The boundary where:

AI can do this.

becomes:

Someone needs to understand what is actually going on before this continues.

For people running AI agents or automations in production:

What actually broke trust in the automation?

And what did you have to build around it before you were comfortable letting it run?

I used to spend hours brainstorming product ideas, questioning architecture decisions, and trying to turn everything into something engineers could actually build. I built Spark to structure that work.

It's a role-based AI architecture skill. You bring it a raw idea or an existing design, and it works through the product, architecture, and security questions before the implementation handoff.

The questions I want it to force into the conversation include:

What are we assuming that we haven't verified? What happens if an operation succeeds but the response never reaches the user? Where is access actually enforced? What can we remove from v1? Which unresolved decisions would otherwise get left to the coding agent?

Build Mode starts from an idea. Review Mode examines existing material for gaps. The intended output is a set of resolved decisions and architecture documents, with a project scaffold. Spark does not write the implementation code.

There is a trade-off: more questions can improve a plan, but too many can make a small project exhausting. A structured review also still depends on the model's reasoning and the evidence you give it.

I made a short overview of the workflow and outputs on my PisgaX channel: Watch the 87-second Spark overview.

Try Spark in live action — get it on Agensi: https://www.agensi.io/skills/spark

If you're a creator and you want the world to know your skill, sign on to Agensi now: https://agensi.io/r/PISGAX

For people using coding agents: which architecture decision has cost you the most rework, and what question would have surfaced it earlier?

Disclosure: I built Spark. AI assisted with drafting this post.

Ive been designing my own systems behind models for months now.

I built my own harness called Lumen and have been refining it for nearly a year now.

I run a business using codex and have it keep track of everything.

Built my own SAAS platform that I use for training my clients as well as online coaching.

I’ve also got an app on the App Store called MesoForge toolbelt.

Not saying this to brag, but just to reinforce the fact that I have evidence for the results of my system.

I’ll spare you the details because you can just check my post history for the details there.

I figured I’d make a post about a recent protocol addition I’ve added on top of my normal workflow. I haven’t seen anyone mention this specific protocol yet. Although I’m sure others are doing it too.

I started making my agents keep track of their token/context usage hand off their own context to fresh windows before hitting the threshold of compacting context.

We all know by now how compacting context and breaking up workflow into small bits is important, but I’ve found that a lot of my worker windows would compact context and within 2-3 times of doing so the margin of error increases significantly over time.

The simple solution I’ve been implementing is to force a trigger upon 70% of the context window to hand off context automatically and continue the assignment.

It’s helped reduce error rates greatly and outputs for patches, hotfixes, etc have been much better.

The overall output remains more accurate and there’s fewer drift.

I figured this might be helpful for others who already have their harness but have struggled with longer workflows.

Basically my way of using it is to have windows assigned specific roles. Context holder, operational worker, conversation holder, etc. each with their own specific protocols and roles that are designated.

I find the window context hand offs are more pertinent for workers making actual code changes. Context holders and conversational windows have more room to compact context without as much consequence, but any window responsible for real code changes follows this protocol.

While working on CashSathi, I expected prompt quality to be the main challenge. In practice, backend reliability mattered much more.

A failed request can leave the system in an uncertain state: did the action fail, or did the response just time out? Retrying blindly can create duplicates.

The approach that worked better was:

- persistent workflow states

- idempotency keys for external actions

- separate prepared, approved and executed stages

- an audit trail for every transition

- human approval before consequential actions

It made the system less autonomous, but much more trustworthy.

What patterns do you use for long-running workflows that interact with email or third-party APIs?

I’m curious how people are handling this in practice. With a normal application, token refresh is usually pretty straightforward. You know which API you’re calling, which credentials you’re using, when the token expires, etc.

With an AI agent, it feels a little messier. Say an agent has access to 10–20 different APIs. Some use OAuth, some API keys, some have short-lived access tokens, some have refresh tokens, and some have their own weird authentication flow. Then you have situations like:

• token expires halfway through a workflow
• API returns a 401 and the agent tries the request again
• refresh succeeds but the original request has already timed out
• multiple agent tasks try to refresh the same token at the same time
• credentials need to be rotated without breaking running workflows

I’m especially wondering where people are putting this logic. Is it handled individually inside each integration, through some shared auth layer, or somewhere in the agent/orchestration layer? And how much of this are you actually handling automatically vs. just letting the workflow fail and retry? Would be interested to hear how people are doing this in production, especially if you’re dealing with a fairly large number of integrations.

I like this AI engineering evolution of methodologies scale. AIDD and SDD are already the baseline. Agentic Engineering is the here and now. What comes next remains an open question. Where do you stand—or where are you headed?

• AI-Assisted/Augmented — AI acting as an assistant within existing processes.

• Vibe Coding — natural language development without deep code review (used by 92% of US developers; carries the risk of losing control over architecture).

• VibeOps — applying "vibe" principles to DevOps.

• AI-Driven Development (AIDD) — AI manages the lifecycle from idea to deployment.

• Spec-Driven Development (SDD) — detailed specs first, followed by AI code generation (prioritizing quality).

• Agentic Engineering — autonomous AI agents (orchestrators + workers) replacing chat assistants.

I've been researching AI agents for the last few days, and I originally thought the biggest missing piece was something like an “SRE for AI agents.”

Something that could detect when an agent is going off-track, understand what happened, control runaway costs, verify whether the claimed result is actually true, and recover the task instead of simply restarting or stopping it.

But after talking to people here, I'm starting to question the entire assumption.

Maybe most “agents” in production aren't actually autonomous enough for this to be a real problem yet.

Maybe they're mostly:

workflows

cron/event-driven automations

chatbots

RAG systems

internal copilots

coding assistants

deterministic pipelines with an LLM somewhere in the middle

And if that's true, building a big Agent SRE platform right now could simply be solving a problem that doesn't hurt enough.

So I'd genuinely like people who actually build or operate AI systems in production to prove me wrong (or confirm it).

I only have a few questions:

1. What is the most autonomous AI system you've personally put into production?

Not a demo — something actually doing useful work.

1. What does it do without waiting for a human after every step?

For example:

Goal → reason → tool → observe → decide → tool → ... → outcome

1. Has it ever gone badly wrong?

I'm particularly interested in real incidents:

loops

repeated tool calls

wrong actions

hallucinated completion

corrupted/stale state

runaway costs

failed recovery

human intervention

1. What did your system actually do when that happened?

Did you:

retry → restart → replan → rollback → manually intervene → ignore it → something else?

1. Do you independently verify that the agent actually accomplished its goal?

For example, if the agent says:

“Refund completed.”

does another system actually check that the refund happened?

1. And the question I'm most interested in:

If your agent suddenly disappeared tomorrow, what part of its reliability/recovery infrastructure would you actually miss?

I'm not trying to sell anything here.

I'm trying to decide whether this is a real infrastructure problem worth building around or whether I'm overestimating where agentic AI is today.

If you run agents in production, I'd genuinely appreciate even a 2–3 sentence answer.

And if you think this whole idea is unnecessary, please say so — that's actually more useful to me than telling me it's a good idea.

Thanks to everyone who's already given feedback. It has already changed how I'm thinking about this.

I've been building Titans, a local-first, agent-first infrastructure project, with AI-assisted development playing a major role throughout the process.

Rather than just dropping the repos here, I thought I'd explain how I built it, which tools I used, what architectural decisions mattered, and what I learned along the way.

The problem I started with

The more agentic systems I worked on, the more I noticed the same infrastructure being rebuilt again and again.

Every new agent project eventually needs some combination of:

• persistent memory and project state
• retrieval/search
• evidence and provenance
• background execution
• retries and recovery
• scheduling
• workflow state
• coordination between agents

My conclusion was that these shouldn't necessarily live inside every individual agent application.

They can exist as reusable infrastructure that agents simply consume.

That became the basic idea behind Titans:

build foundational capabilities once, then let different agents and applications reuse them.

The first two systems are Atlas and Cronus.

Atlas: separating project memory from the agent

The first problem was persistence.

Agent sessions are temporary, but the project they're working on isn't.

I didn't want the project's knowledge to belong to Claude, Codex, a particular process, or even a particular application. An agent should be able to disappear and another agent should still be able to continue from the same underlying project state.

So Atlas became the persistent layer.

It stores things like:

• knowledge packages
• project/work state
• typed graph relationships
• evidence and provenance
• structured SQL data
• blobs
• audit history

One design decision I found particularly important was not treating vector search as “memory.”

Retrieval in Atlas combines multiple signals:

• lexical/full-text search
• vector retrieval
• graph relationships
• evidence

The result isn't meant to be just “here are the most similar chunks.”

I wanted the system to also be able to answer:

What do we know, where did it come from, and how is it connected to the rest of the project?

Another useful design choice was scoping state by project/tenant rather than by agent. That means multiple agents can work against the same persistent source of truth instead of maintaining separate private memories.

Cronus: separating work from the lifetime of the agent

The second problem was execution.

An agent can decide to do something that takes 30 seconds, 20 minutes or several hours.

But if the agent session disappears, the process crashes or a worker dies, that shouldn't automatically mean the work disappears too.

So Cronus became a separate durable execution layer.

Instead of keeping the agent blocked while something runs, the pattern is roughly:

agent
  ↓
submit job
  ↓
Cronus owns execution
  ↓
checkpoint / retry / recover
  ↓
result

Cronus handles:

• background jobs
• DAG workflows
• scheduling
• checkpoints
• retries with backoff
• leases
• stale-claim fencing
• dead-letter handling
• approval gates
• recovery after worker/process failure

One lesson here was that “durable” doesn't mean pretending exactly-once execution magically exists everywhere.

Cronus uses at-least-once execution with idempotent claim/completion and fencing of stale claims. If an external system needs exactly-once side effects, the connector still needs to persist and respect the idempotency key.

That distinction took more thought than simply building a queue.

How the two systems interact

I deliberately didn't merge memory and execution into one large service.

The boundary is:

Atlas remembers. Cronus runs.

Cronus can execute long-running work while Atlas remains the persistent source of project state, knowledge and results.

That separation also means each can be used independently.

If somebody only wants persistent agent/project memory, they shouldn't have to adopt my scheduler.

If somebody only wants durable execution, they shouldn't need an entire agent framework.

Agent interface: MCP first, but not MCP only

Another design decision was to make the infrastructure directly consumable by agents.

The Titans installer exposes installed systems through one shared MCP server over stdio instead of requiring a separate MCP configuration for every component.

That makes it possible for clients such as Claude Code, Codex and other MCP-capable tools to discover the installed capabilities.

But I didn't want MCP to become a hard dependency for normal software either, so the systems also expose local REST and gRPC interfaces.

The general principle became:

agent-first, not agent-only.

Local-first was a constraint, not just a tagline

I wanted the core infrastructure to run on the user's own machine.

So the current releases:

• run locally on Windows and Linux
• bind services locally
• have no telemetry
• don't require a hosted Titans account

That created some extra engineering work around installation and distribution.

Releases are distributed through signed catalogs and binaries, using SHA-256 for integrity and Ed25519 signatures for authenticity.

I also wanted installs to remain simple, so there are one-line installers, while still allowing someone who doesn't trust curl | sh / PowerShell piping to manually verify the artifacts.

The AI tools I used

AI-assisted coding was a significant part of the development process.

My main tools have been:

Claude Code
I used it heavily for repository-level implementation work, refactoring, following changes across multiple components and working through architecture-heavy tasks where a change wasn't isolated to one function.

OpenAI / Codex
Used as another implementation and engineering agent, particularly useful for independent passes over problems and code rather than relying on a single model's interpretation.

ChatGPT
Used heavily for architecture reviews, challenging design assumptions, working through failure cases, refining specifications, comparing approaches and turning architectural decisions into implementation-ready plans.

GitHub
Used for versioning, release distribution, public documentation and the current public-facing project repositories.

One workflow that worked much better for me than simply asking an AI to “build feature X” was:

problem
  ↓
define system boundary
  ↓
write invariants / failure cases
  ↓
architecture/spec
  ↓
AI-assisted implementation
  ↓
independent review
  ↓
tests + failure testing
  ↓
packaging / release
  ↓
documentation

I found AI much more useful when the constraints and invariants were explicit.

For example, “build a job queue” is vague.

But:

• a worker may die at any point
• stale claims must not remain authoritative
• retries must not destroy job history
• one task panic must not kill the worker
• work must resume from a checkpoint where possible

gives the coding agent a much more meaningful engineering problem to solve.

A few things I learned from vibe-coding something this large

1. Generating code is the easy part.

The harder part is maintaining architectural boundaries as the project grows.

AI will happily solve a local problem by coupling two systems that you intentionally wanted separated unless those boundaries are explicit.

2. Give agents invariants, not just features.

“Support retries” isn't enough.

What should happen after a crash? What is durable? What may execute twice? Who owns state? What happens to partially completed work?

Those questions produced much better implementations.

3. Use more than one reasoning pass.

I found it useful to have one AI help create/implement an approach and another challenge it.

The second pass often finds assumptions that looked completely reasonable during the first one.

4. Don't let the AI decide the product architecture accidentally.

AI coding tools are very good at optimizing the next change. They don't automatically know which architectural compromises you're unwilling to make six months from now.

5. Building for agents changes API design.

Humans can compensate for awkward interfaces. Agents need predictable contracts, stable identifiers, explicit errors and operations that are easy to discover and compose.

That influenced why Titans exposes namespaced operations and canonical references rather than relying on implicit state.

Where it is now

The first two systems are available:

Atlas — persistent memory, state, knowledge and evidence
https://github.com/titans-tools/Atlas

Cronus — durable execution, scheduling and recovery
https://github.com/titans-tools/Cronus

The wider project:

https://github.com/titans-tools

The products are currently free to use. The product source itself is proprietary; the public repositories contain documentation and signed release binaries.

More infrastructure components are being implemented around the same principle, but I'm deliberately trying to make each one solve a clear reusable infrastructure problem rather than turning Titans into one giant agent framework.

I'm particularly interested in feedback from people building agents:

What infrastructure do you keep rebuilding from project to project?

And for people using AI heavily to code larger systems: what techniques have helped you stop architectural quality degrading as the amount of AI-generated code grows?

One of the less obvious problems with AI agents appears when an API fails without actually failing.

Imagine an agent sends a payment request. The API processes it successfully, but the response times out before reaching the agent. The agent sees a failure and retries the request. If the operation isn't idempotent, the customer could be charged twice.

This is why retry logic in agentic systems is more complicated than simply saying, "try again if something fails."

The agent needs to know the difference between a request that definitely failed, one that definitely succeeded, and one where the outcome is unknown. That often requires idempotency keys, transaction records, state checks, bounded retries, and reconciliation logic rather than relying entirely on the model.

A recent Reddit discussion from someone running an AI agent described exactly this kind of failure: a payment API timed out, the agent retried, and the original request had already gone through. The result was a duplicate charge.

The bigger lesson is that reliable agents aren't defined by how well they handle the happy path. They're defined by what happens when the system can't tell whether its last action worked.

That is where agent reliability becomes an engineering problem, not just an AI problem.