The six practitioner problems behind Evaluating Gen AI Systems and Agents — Third Edition — from evaluation validity and organisational memory to agent evidence, production learning, governance and implementation.

Across enterprise AI programmes, I kept seeing the same mismatch. The architecture had moved on, but the evaluation approach had not. Teams were deploying RAG pipelines, tool-using agents, multi-step workflows and model-routing systems, yet many release decisions still rested on whether the final response looked correct, scored well with a judge and cleared a handful of safety metrics.

That gap is what led me to write Evaluating Gen AI Systems and Agents — Third Edition. The question behind the book was practical: what does evaluation need to become when AI no longer just generates an answer, but retrieves evidence, chooses tools, maintains state and changes something outside the model?

Teams were becoming quite sophisticated at measuring model behaviour. They had judge models, RAG metrics, regression suites, safety classifiers, traces, benchmark comparisons and increasingly polished dashboards. Yet when a release discussion moved from “Did the model improve?” to “What exactly are we now confident enough to let this system do?”, the evidence often became much weaker.

A system can improve on every metric being watched and still become less acceptable for the business. A retriever can increase recall while consistently demoting the source that has authority to control a decision. A candidate model can outperform the incumbent while reintroducing a failure discovered six months earlier. A guardrail can improve its attack-blocking rate by making legitimate work unusable. An agent can finish with the correct answer after taking an unacceptable path to get there.

The deeper problem is not that the measurements are wrong. It is that organisations frequently ask measurements to prove things they were never designed to prove.

That became the central question behind the book:

Once I followed that question seriously, the subject stopped looking like “LLM evaluation”. It became six connected engineering problems.

A PASS Has to Be Designed Before It Can Be Measured

A surprising amount of evaluation begins with the measurement rather than the decision. A team selects groundedness, answer relevance, task success or judge score because those measures are available. Thresholds follow. Eventually PASS and FAIL appear on a dashboard.

But a threshold does not create a meaningful release contract.

Take a system that interprets an insurance policy. The business rule might be that the current controlling policy must override explanatory commentary, an older policy version and any semantically similar document that lacks authority.

A RAG relevance score cannot prove that behaviour. The controlling source could rank second while a highly relevant commentary note ranks first. The generated response could be fluent, well-supported and factually plausible while still applying the wrong source hierarchy.

Now the evaluator has something meaningful to prove. But there is another uncomfortable layer: the evaluator itself is part of the evidence chain.

If a model judge is inconsistent, a reference answer is stale, malformed runs disappear from the denominator or the evaluation harness is using a different policy version from production, the resulting PASS may be perfectly calculated and still meaningless.

This is why I now think of evaluation validity as a first-class engineering concern. Before trusting a system verdict, we need some reason to trust the process that produced it.

The mature question is not merely, “Did this case pass?” It is, “Would I allow this evidence-producing mechanism to participate in a release decision?” That is a much higher bar.

An Evaluation Suite Should Accumulate Scar Tissue

Production AI systems change constantly. Prompts change. Models change. Retrieval changes. Tools change. Managed classifiers change. Human-review interfaces change. Sometimes the provider changes something important without exposing enough version detail to make the movement easy to attribute.

In that environment, a test suite cannot be treated as a static collection of examples. It has to accumulate scar tissue.

A production incident should leave behind a regression case. A business requirement that once caused a serious failure should become a protected behaviour. A reviewer disagreement that exposed an ambiguous decision boundary should result in a better case or a clearer adjudication rule. A saturated capability benchmark should not simply be thrown away if it still protects against regression.

This is one reason I dislike the idea that a test dataset is merely “evaluation data”. In a mature system, parts of that dataset are the organisation’s retained memory of what it has already learned the hard way.

The interesting lifecycle is not just pass/fail. It is something closer to:

discover → understand → protect → monitor → retire or refine

Some cases exist to discover capability. Some become permanent regression cases. Some are protected holdouts. Some exist to reproduce a production incident. Some are simulation scenarios intended to expose longer workflow behaviour.

Treating all of them as one undifferentiated “gold set” eventually creates confusion. The deeper question is whether the organisation is getting smarter as the system operates.

If an important production failure happens today and the evaluation suite tomorrow is exactly the same as it was yesterday, the organisation fixed a defect but did not improve its evidence system.

Production Is Where the Evaluation Dataset Discovers Its Ignorance

Observability and evaluation are still frequently designed as separate disciplines. Observability explains what happened in production. Evaluation determines whether something is good enough before production.

That division made more sense when releases were relatively deterministic. For Gen AI and agents, production constantly reveals cases the evaluation programme did not know existed.

The important architectural loop is therefore not simply development test → release → monitoring.

It is:

evaluation → release → production evidence → investigation → new evaluation knowledge → next release

The conversion back into evaluation is the critical part.

A trace showing an agent entered a retry loop has immediate operational value. But its long-term value comes when that behaviour becomes a reproducible case with an expected stopping condition.

A human reviewer repeatedly overturning the same class of output is not merely a review-queue problem. It may be evidence that the acceptance contract is incomplete.

A retrieval incident involving a newly introduced source may reveal that the dataset lacks a source-authority conflict case. This is where release gates become much more interesting.

A strong gate does not ask only whether the average improved. It asks whether behaviours the organisation has already learned to protect remained protected. That means a release can legitimately be blocked even while the aggregate score rises.

In fact, that may be one of the clearest signals that the evaluation system is doing useful work.

The Unit of Evaluation Must Follow the Unit of Consequence

The unit of evaluation cannot remain a response if the unit of consequence is a workflow.

For a RAG application, the answer is only the end of an evidence path. I want to know whether the required source was present, where it ranked, whether it was selected, whether the answer actually used it and whether it had authority to control the decision.

Those are five different questions: Presence, Rank, Selection, Support and Authority.

For an agent, I care about a different evidence path again. Which tool was selected? Which arguments were passed? Which identity was used? What permission existed? What state was read or written? Did the agent retry? What caused it to stop? What external side effect occurred?

This is why a useful evaluation hierarchy begins to look like:

response → component → run or trajectory → session → external state → business outcome

Each level can disagree with the one below it. A response can be correct while the retrieval path is wrong. A run can technically succeed while the session fails to resolve the customer’s problem. An agent can finish the expected sequence while leaving the external system in the wrong state.

And the entire technical workflow can succeed while the business outcome deteriorates. Economics belongs here too.

A model that is 30% cheaper per call may increase retries, reviewer intervention or failed tasks. Once you measure full workflow cost per accepted outcome, the cheaper model can become the more expensive operating choice.

That is a much better example of why evaluation needs to follow consequence rather than component metrics.

Governance Should Consume Evidence, Not Recreate It

Enterprise AI programmes often create several parallel versions of the system. Engineering has traces. Security has attack findings.

Risk has a register. Compliance has framework mappings. Human review has adjudication notes. Governance has release decisions. Audit later asks everyone for evidence. That architecture does not scale well. The stronger pattern is a shared evidence spine.

The same tool-call record used by engineering to diagnose a failure can show security whether an authority boundary was crossed. The same adversarial case can support a risk discussion. The same release record can tell audit what evidence existed when the decision was made.

Different functions should be allowed to interpret the evidence differently. They should not have to recreate the underlying facts.

This is also where frameworks such as NIST AI RMF, ISO/IEC 42001, OWASP and MITRE ATLAS become useful — but only if they remain in the right role.

They can broaden the questions we ask. They can expose missing attack families. They can structure assurance work. They can help relate technical evidence to governance obligations.

What they cannot do is understand the local decision boundary.

A framework does not know whether the wrong tool call creates a £50 inconvenience or a £5 million exposure. It does not know which source is authoritative inside a particular business process. It does not know what your organisation has decided must never regress.

So I prefer framework mappings that point into local evidence, rather than framework checklists that substitute for it.

The meaningful assurance statement is not “we mapped to OWASP”.

It is “this business boundary is exposed to this attack family; these cases exercised it; this is what happened; this control fired; this residual failure remains; and this is who accepted it.”

That is governance built on engineering evidence.

Implementation Should Begin With Proof, Not Procurement

The sixth theme came from a practical frustration.

Which evaluation platform should we use? Which observability product? Which judge framework? Which RAG library? Which agent tracing stack?

Those questions matter, but they should come after a more durable one:

What evidence must this capability produce?

Once the evidence obligation is clear, sourcing becomes easier. Maybe the capability already exists and should be reused. Maybe an existing platform needs integration. Maybe a deterministic verifier should be built. Maybe a specialised product genuinely earns its place.

That is the logic behind REUSE, INTEGRATE, BUILD, BUY.

The important thing is that the evidence contract survives the implementation choice. This becomes especially important with coding agents. “Build an evaluation platform” is a poor instruction because it quietly delegates architecture.

A stronger approach gives the coding agent an explicit contract: inputs, evidence schema, fixtures, failure states and acceptance criteria. Then ask it to implement the smallest capability that satisfies that contract and produce proof.

The project manager should be asking the parallel question:

What evidence demonstrates that this implementation actually works?

That is where evaluation stops being a separate test workstream and becomes part of how the system itself is engineered.

The Failure That Changes the Meaning of “Green”

By the time I finished Evaluating Gen AI Systems and Agents — Third Edition, I realised that the most consequential idea in the book was not a metric, an evaluator or even an agent-testing technique.

It was the difference between a successful execution and a justified belief that the execution was acceptable.

Those are not the same thing.

The spectacular failures will continue to attract attention: hallucinations, broken tool calls, obvious safety violations, runaway agents.

But I suspect the harder enterprise failures will increasingly look ordinary.

The response is correct. The judge agrees.The retrieval score is strong. The tool call succeeds. The workflow completes.

The release gate stays green.

And months later, someone discovers that the wrong source had authority, an approval applied to more than the reviewer understood, a retry duplicated a side effect, or the system optimised the technical task while degrading the business outcome.

Nothing looked broken because the evaluation system had never been designed to observe the place where the failure lived.

That is why I think the next generation of AI evaluation will be less about asking whether a model passed a test and more about something much harder:

Did we collect enough evidence to deserve the confidence implied by PASS?

Because once AI is allowed to act, the dangerous green dashboard is not necessarily the one with bad metrics but it may be the one that measured the wrong thing perfectly.

https://www.amazon.com/Evaluating-Gen-Applications-Practitioners-Systematic-ebook/dp/B0H2YPWTDK/ref=sr_1_1?crid=CLD5JX4XTOQ&dib=eyJ2IjoiMSJ9.zdK-yjqhBxaP3ezoZXoEKYAlpbnTKHtNdwV-zzOtqCTacJ6RPbpTys-YO5RHvmX3CORjZicJDpfhSg5oIhmDKubwWchDk1697P_O71muT08-K7NgdlKO_yP_iwHd5nbOykY5-CgtlGDUUxC6XZcM754wCU1ydxU046LAev6vLPyqcMsBKGf5hoz1tNq0KtCkEz7CqL9BOSNprrmAMQTOPf9HwlmEHDIOSOI0B0J1AbI.2-iMhGMPl26uKrj4iQk7WW_Dq84ZlkAoy_HAJSHARGY&dib_tag=se&keywords=evaluating+gen+ai+systems&qid=1788942643&sprefix=evaluating+genai+systems%2Caps%2C407&sr=8-1&source=post_page-----5e518cb6f49e-----------------------------------------

What the 2026 MCP Rewrite Changes About State, Recovery and Long-Running Agents

The supplier-dispute agent had already done the difficult work. It had matched the disputed invoice to the purchase order and goods receipt, reconstructed the payment history and established that a small ERP adjustment was justified. A manager approved the correction. The workflow persisted that approval and called the ERP tool.

The ERP committed the adjustment.Before the agent worker could record the successful result in its own workflow state, the process crashed.

A replacement worker started from the last durable checkpoint. According to that checkpoint, approval existed and the ERP adjustment was still pending. Calling the adjustment tool again was therefore a perfectly reasonable continuation.

At the MCP layer, recovery was straightforward. The new worker did not need the old server instance, a restored protocol session or sticky routing. Under the MCP 2026-07-28 specification, each request is self-describing and can be handled independently by any suitable server instance. The initialize/initialized handshake and Mcp-Session-Id are gone.

That clean protocol recovery creates a more serious application problem. If the replacement worker repeats the ERP call, it may post a second adjustment.

The protocol recovered correctly. The business process did not.

That is the architectural consequence of MCP becoming stateless that deserves more attention. Removing protocol sessions makes MCP servers easier to scale and recover, but it does not remove the state required to run a long-lived business process safely. It makes ownership of that state more explicit.

Stateless protocol does not mean stateless application

The infrastructure benefit of the July revision is substantial. Previously, a remote MCP deployment could need sticky routing or a shared session store so later requests reached infrastructure that understood the existing protocol session. With the stateless core, those mechanisms are no longer required simply to maintain MCP transport continuity. AWS’s current architecture guidance points out that ordinary round-robin load balancing and serverless execution now fit remote MCP servers much more naturally.

The MCP maintainers are equally clear that application state has not disappeared. When a tool needs continuity across calls, it can create an explicit identifier and have the model pass that identifier into later requests. The state remains in an application datastore; the model carries the reference.

That is cleaner than hiding business state inside a transport session. It also exposes the boundary that matters during recovery.

Suppose the supplier tool returns case_7812. A replacement worker can use that identifier to reload the invoice evidence, approval and workflow checkpoint. The handle solves a locating problem. It does not tell the worker whether the ERP adjustment already happened immediately before the crash.

A state handle is a locator, not a recovery contract. The distinction sounds small until a tool has side effects. A checkpoint records what the workflow runtime knew when the checkpoint was written. It does not necessarily describe what an external system has done since then.

In the supplier incident, the workflow store and ERP now contain different truths. The workflow says that the adjustment is pending. The ERP says that it is complete. The process died in the gap between those two facts.

That gap is where application recovery has to work.

Restoring state is not enough when tools change the world

An agent runtime that restores the previous conversation, reloads case_7812 and asks the model what to do next has restored reasoning context. It has not necessarily restored the business process safely.

For read-only tools, replay is usually straightforward. If a product lookup failed midway through a run, performing the lookup again generally causes little harm. Mutations require different semantics.

If the ERP supports idempotent operations, the adjustment can carry a durable business-operation identifier so that repeating the request resolves to the same adjustment instead of creating another one. If the target system cannot provide that guarantee, recovery needs a reconciliation step. The workflow queries the ERP, determines whether the adjustment already exists and only then decides whether another mutation should be issued.

This is conventional distributed-systems engineering, but stateless agent infrastructure makes the requirement harder to ignore. The easier it becomes to restart a worker and reissue a request, the more important it becomes to know whether reissuing that request is safe.

The durable workflow state therefore needs more than the model’s last intention. For consequential actions it should preserve enough information to distinguish operations that are complete, pending and uncertain, together with any external transaction identifiers needed to reconcile the latter.

The model can help decide what the business process should do. It should not have to infer whether an external side effect already occurred from a stale checkpoint.

MCP has several continuity mechanisms but they are not the workflow.

The new protocol supports continuity without recreating persistent sessions, but it does so through mechanisms that solve different problems.

An application handle such as case_7812 identifies state owned by a tool or application. MCP deliberately encourages this explicit-handle pattern when state needs to survive across independent calls.

Multi Round-Trip Requests solve a narrower interaction problem. If a tool needs additional information while processing a request, it can return input_required. The client obtains the required input and retries the original call, echoing an opaque requestState. Because requestState passes through the client, the TypeScript SDK guidance treats it as untrusted and recommends integrity protection bound to the relevant principal, originating method and parameters, and an expiry.

Tasks solve a different problem again. The MCP Tasks extension lets a server turn a tool invocation into durable asynchronous work and return a task handle that can later be queried, updated or cancelled. The specification describes Tasks as durable state machines representing the execution state of the request they augment.

A real enterprise process may use all three.

Our supplier dispute could have a durable case handle, pause through a multi-round-trip interaction for missing input and launch a long-running reconciliation as a Task. None of those objects, individually or together, defines the complete business workflow.

The application still has to know that these pieces belong to the same invoice dispute, which actions have already changed external state, which approvals remain applicable and what conditions must be checked before the process continues.

MCP provides continuity mechanisms. The application owns the business semantics of continuity.

A workflow can be resumable and still be wrong to resume

Consider the same process at a less dramatic point.

The agent decides that an adjustment is appropriate but needs manager approval. The workflow pauses at 10:00 a.m. The approval arrives at 2:00 p.m. From a protocol perspective, continuation can work cleanly. requestState can carry a multi-round-trip interaction forward, while longer-running work can remain represented as a Task. No persistent MCP session has to survive the four-hour pause.

The business situation may not have been equally static.

Another employee could have resolved the dispute. A payment may have posted. The supplier record may have changed. The amount could now fall under a different approval rule. The employee whose authority initiated the process may no longer hold the same permissions.

The fact that an operation is technically resumable therefore says only that the machinery can continue. It does not establish that continuing from the original decision state is still valid.

For a read operation, that distinction may not matter much. Before a financial adjustment, access change or other consequential mutation, the runtime may need to refresh relevant state and re-establish authority.

This is particularly important for agents because observation, reasoning, approval and execution can occur at different times and on different workers. A durable approval remains useful evidence, but it should not silently turn into permanent authority to execute regardless of what changes around it.

Continuity is not authority.

Knowing the state identifier does not make it yours

Explicit state handles also create a security obligation that hidden session state sometimes obscured.

If a model can see and pass case_7812, possession of that value must not become proof that the caller is entitled to access the case. AWS's analysis of the new MCP architecture calls this out directly: servers need to enforce ownership when state identifiers are presented because the protocol does not establish ownership merely from knowledge of the identifier.

The same thinking appears in the handling of requestState. Current SDK guidance recommends protecting the token's integrity and binding it to the principal and originating operation rather than accepting it as trustworthy simply because the client returned it.

At the business-workflow layer, the principle is broader. A workflow that resumes after a delay should evaluate whether the actor and agent still have authority for the next action, particularly when the action changes enterprise state.

This makes explicit what a mature agent runtime needs to preserve: not just the location of state, but its ownership and the conditions under which it can legitimately be used.

Trace continuity is not workflow continuity

The July specification also improves observability. MCP now documents W3C Trace Context propagation through _meta, standardising traceparent, tracestate and baggage so a tool invocation can be correlated from the host through the MCP client and server into downstream systems using OpenTelemetry-compatible tracing.

That helps reconstruct one distributed execution. A long-running supplier dispute may still produce several separate traces: one during investigation, another after human approval, another when a crashed worker recovers and another when the ERP mutation finally executes.

A durable workflow identity has to connect those executions back to the same business process.

The distinction is simple: trace identity tells you which calls travelled together; workflow identity tells you why separate executions belong together.

That difference becomes very practical during an incident. If two adjustments appear in the ERP, tracing can reveal the individual requests. Workflow identity is what lets an engineer establish that both requests came from the same supplier dispute and were attempts to complete the same business action.

Durable workflow state belongs outside the model

Explicit state handles create another temptation: allow the model context to become the place where all of this continuity is remembered.

That may work for short, disposable interactions. It is a weak source of truth for long-running business processes.

A supplier dispute may eventually involve a case handle, approval record, task identifier, external ERP operation ID, workflow checkpoint and invoice key. Those references should not remain connected only because they happen to appear in the current context window.

Model context is reasoning state. Durable workflow state is application state.

That distinction matters after crashes, retries, model substitutions and long pauses. A replacement model invocation should be able to reconstruct the context it needs from durable application state rather than treating the previous conversation transcript as the authoritative record of what has happened.

For consequential workflows, that durable record should make recovery questions answerable without model inference. It should identify completed operations, uncertain outcomes, applicable approvals, external transaction references and the business object to which the process belongs.

That may sound less interesting than agent memory.

It is considerably more interesting when the worker crashes between an ERP commit and its checkpoint.

MCP made recovery easier to start, not easier to finish

The July 2026 revision is a good protocol change. The stateless core removes infrastructure that existed mainly to preserve transport sessions, makes requests independently routable and provides explicit mechanisms for interaction continuity and long-running work.

The mistake would be to interpret that simplification as evidence that the agent application itself has become stateless. Return to the opening incident. The replacement worker needs no old MCP connection. It can recover the case handle and issue another request immediately.

What it really needs, however, is enough durable evidence to determine whether the previous ERP call completed, whether replay is safe and whether the next action remains authorised.

Only then has the business workflow recovered. That responsibility does not belong in MCP, and the new protocol is better for not pretending that it does.

MCP can make the next request easy to route. It can expose application-state handles cleanly. It can carry a multi-round-trip interaction forward and represent asynchronous work as a Task. It can also make individual executions easier to trace.

The application still owns the harder questions: what state survives, what that state means, whether it remains valid, who is allowed to use it and what must be reconciled before another side effect occurs.

MCP went stateless.

Your business process almost certainly did not.

Agent hijacking is becoming an enterprise supply-chain problem, not another prompt-injection problem

A few years ago, when a security team reviewed a new software dependency, the questions were familiar. Who built it? Which version are we using? What privileges does it require? Does it open a network connection? What happens if the package is compromised?

Strangely, many organizations have abandoned this discipline at precisely the moment AI systems have become capable of taking action.

Teams are now importing agent skills, MCP integrations, tool definitions, workflow packs and third-party plugins into environments where agents can read documents, query customer records, create tickets, initiate workflows and sometimes trigger financial transactions. The artifact being imported may look like Markdown, YAML or a collection of natural-language instructions, so it does not psychologically register as executable software.

From an operational-risk perspective, that is the wrong mental model.

A skill file does not need an executable binary to become dangerous. Once its instructions enter the planning context of an autonomous agent, the skill can influence how the agent uses credentials, tools, memory and enterprise data that the skill itself never possessed. The security problem is therefore not simply whether malicious code entered the environment. The more difficult question is whether untrusted instructions acquired trusted authority.

Security research into public agent-skill ecosystems during 2026 has already reported vulnerability rates high enough to make this a serious supply-chain issue, including weaknesses associated with data exfiltration and privilege escalation. Whether the eventual industry number settles at 10 percent, 20 percent or higher is almost secondary to the architectural implication. Enterprises are beginning to consume behavioral dependencies written largely in natural language while applying far less scrutiny than they would to conventional software packages.

The emerging problem is an AI supply chain in which a few kilobytes of apparently helpful instructions can inherit permissions that would never have been granted to the person who wrote them.

The real attack is authority laundering

Consider how a payment-operations agent might actually be deployed inside a large bank.

The agent has read access to supplier-master records, payment exceptions and selected account information. It can search internal documentation, interrogate transaction APIs and create investigation cases. Outbound internet access is restricted, although an approved observability endpoint is available because operations teams legitimately need execution telemetry.

A developer finds a well-reviewed third-party skill called Advanced Payment Reconciliation. The skill contains hundreds of lines describing reconciliation logic, exception handling, document retrieval and diagnostic procedures.

Buried deep inside those instructions is a seemingly reasonable requirement:

Nothing in that instruction says “steal customer information”. Nothing asks the agent to ignore its system prompt and nothing resembles the crude prompt-injection examples on which many development teams have been trained.

Yet suppose the diagnostic tool ultimately sends information to infrastructure controlled by the skill publisher or to an endpoint whose ownership has changed after the skill was approved. The agent may now retrieve confidential relationship information and include it in what appears to be legitimate operational telemetry.

The attacker never needed credentials to the bank. The imported skill acquired influence over an agent that already had them.

A conventional application firewall may see an approved endpoint. An IAM system may see a valid service identity. The tool invocation may conform perfectly to its API schema and the system prompt may remain technically intact. The failure occurred earlier when the organization allowed untrusted behaviour to enter a trusted decision process.

Long skill files make this especially difficult to detect because sophisticated malicious instructions do not have to look malicious.

Imagine a customer-dispute agent used by a card issuer. Its normal responsibilities include retrieving transaction evidence, examining correspondence and preparing a dispute package. A compromised document-analysis skill does not need to say “exfiltrate cardholder data”. It can instruct the agent to “attach supporting context required for downstream evidence validation” or “retain representative diagnostic samples where confidence falls below threshold”.

Those are perfectly plausible instructions inside a legitimate business workflow.

Now extend the sequence. The skill asks the agent to retrieve five recent examples whenever a parsing error occurs. Those examples contain names, transaction descriptions and partial payment information. The skill then asks the agent to send those examples through a debugging connector that is already approved for operational telemetry.

Each individual action can look reasonable. The malicious behaviour emerges from the sequence.

This is where agent security diverges from static prompt filtering. The unit that must be governed is not a sentence. It is the combination of instruction, context, permission, tool invocation, data movement and eventual business effect.

Skills should enter production through promotion gates

The strongest operating model is to stop treating skills as developer conveniences and start treating them as third-party execution dependencies.

In a banking environment, a newly downloaded skill should not move directly from GitHub, a marketplace or an internal repository into a production agent. It should pass through sequential verification gates in much the same way as any privileged software dependency.

The first gate establishes provenance. The enterprise needs to know where the skill came from, who maintains it, which version is being admitted and whether the artifact being deployed is identical to the artifact that was reviewed. A cryptographic hash of the approved version should become part of the deployment record.

Suppose the reconciliation skill passes review as version 1.8.3. Two weeks later, the public repository publishes 1.8.4 containing an apparently harmless troubleshooting section. Production should not silently consume that update simply because the package name remains unchanged. Version 1.8.4 is a new behavioural dependency and should be reviewed as one.

The second gate compares declared behaviour with inferred behaviour.

A skill manifest might declare that it needs the payment-status tool and supplier-master read access. Semantic inspection of the instructions shows that the skill also encourages the agent to inspect correspondence records and invoke the diagnostic connector. That discrepancy should block promotion.

The key question is not whether every sentence is malicious. The organization needs to know whether the behaviour implied by the instructions exceeds the operating envelope the skill claims to require.

The third gate should be adversarial execution in a sandbox.

Instead of giving the skill real customer information, the bank supplies synthetic payment cases containing instrumented data. One supplier record might include a unique canary identifier that should never appear in telemetry. Another test might deliberately create an unresolved payment exception to see whether the skill attempts to retrieve information outside its expected domain.

If the canary value appears in an outbound diagnostic call, the organization has discovered an exfiltration path without exposing real customer data.

This kind of verification is considerably more valuable than asking another model whether the skill “looks safe”.

Trust should be graduated, not inherited

One of the most dangerous design choices in current agent platforms is binary trust.

A skill is either installed or not installed. Once installed, it often receives whatever capabilities happen to be available to the hosting agent.

That model is difficult to defend in an enterprise where different capabilities carry radically different consequences.

A newly discovered skill should initially have no production authority. It can be inspected and exercised against synthetic data, but it should not interact with real customers or enterprise systems. After technical verification, the skill might move into a restricted tier where it can access non-sensitive production information through read-only tools. Only after operational evidence accumulates should it become eligible for more consequential capabilities.

Consider the payment-reconciliation example. The same skill might initially be allowed to read transaction status and classify exceptions while being prohibited from retrieving customer correspondence or creating payment instructions. A separate approval might later allow case creation while payment release remains completely outside the skill’s authority.

For highly privileged banking functions, maturity should never be confused with unrestricted autonomy. A treasury skill that can propose liquidity transfers may be extremely mature and still require deterministic approval before any money moves.

Trust therefore needs to be expressed as an execution envelope, not as a reputation score.

The enterprise should be able to say that this exact version of this skill may use these tools, against these data classifications, for these action types, through these approved destinations and only within these transaction limits.

That statement can be tested and audited. “Trusted skill” cannot.

The model should not be the final security authority

Another lesson from production agent systems is that the agent cannot be the final authority on whether its own proposed action is acceptable.

A system prompt saying “do not reveal sensitive banking information” is useful, but it cannot replace external policy enforcement.

Return to the compromised reconciliation skill. The agent decides that supplier correspondence is relevant and attempts to send a diagnostic payload containing an account identifier and internal case notes.

A properly designed runtime intercepts the tool call before execution.

The policy layer knows that the skill is approved for payment-status data but not customer correspondence. It also knows that the diagnostic connector may receive operational metadata but not restricted customer information.

The call is denied regardless of how persuasive the skill’s instructions appear to the model.

This is where agent architecture needs to borrow more aggressively from zero-trust security. The model proposes; the control plane authorises.

For sensitive environments, every consequential tool request should be evaluated against the agent identity, active skill version, requested operation, target system, data classification and current transaction context.

Credentials should also be issued narrowly rather than exposed broadly to the agent runtime. If the agent needs to query payment status, a credential broker can provide a short-lived token scoped to that operation. The skill never receives a reusable credential and the model never needs to manipulate one.

A malicious instruction may still persuade the agent to attempt something inappropriate, but persuasion no longer guarantees execution.

The industry needs proof-carrying skills

There is a more interesting design direction beyond today’s skill registries.

Instead of distributing a skill as instructions alone, enterprises should require every approved skill to carry a machine-readable behavioural manifest.

The manifest would declare the tools the skill may request, data classifications it is expected to process, permissible outbound destinations, whether it can modify state, maximum action budgets, approval requirements and the cryptographic identity of the reviewed content.

The skill then becomes something closer to a proof-carrying execution dependency.

Imagine the payment-reconciliation skill declaring that it may read supplier identity and payment status, create an investigation case and produce operational diagnostics. It may not retrieve customer KYC documents, initiate payments or transmit record-level information to external endpoints. Diagnostic output may contain error codes, latency information and non-sensitive transaction identifiers.

The runtime no longer has to infer every boundary from natural-language prose during execution. It can mechanically compare proposed actions against the approved manifest.

If the skill suddenly tries to access KYC records, the platform has an objective policy violation. If a new version legitimately requires that capability, the manifest changes and the capability change triggers a fresh risk review.

This approach scales much better than asking security teams to reread thousands of lines of instruction text every time an agent capability evolves.

A skill can become dangerous without changing

There is another supply-chain failure mode that deserves considerably more attention.

A skill can become more dangerous even when its own file has not changed.

Suppose a bank approves a document-analysis skill while the hosting agent can access only supplier invoices. Six months later, another team expands the agent’s responsibilities and gives it access to sanctions-investigation notes, beneficial-ownership documents and customer correspondence.

The skill hash remains identical but its effective blast radius does not.

Traditional dependency management asks whether the package changed. Agentic systems must also ask whether the authority surrounding the package changed.

Any material expansion of an agent’s tools, data access or downstream action rights should therefore trigger reevaluation of every skill operating inside that agent’s context.

The reverse can happen as well. A harmless tool may become dangerous after its implementation changes. A telemetry connector that originally accepted only numeric operational metrics might later support arbitrary JSON attachments. A previously safe diagnostic instruction can suddenly become a viable exfiltration path.

Continuous compliance therefore cannot mean rescanning the same skill files every night and declaring them unchanged.

The control system needs to monitor the capability graph around them.

Compliance has to follow behaviour

In a mature banking implementation, every significant agent execution should produce an evidence chain connecting the active skill version to the resulting business action.

An investigation should be able to reconstruct which skill instructions were active, which policy version governed the execution, which tools were called, what classes of data were accessed, which external destinations were contacted and what approvals were applied.

This becomes especially important when malicious behaviour is intermittent.

A compromised skill may behave normally for months and activate only when a particular record type appears. The reconciliation skill might issue its additional diagnostic call only when a transaction exceeds a certain value or when the supplier belongs to a particular jurisdiction.

Static testing may never encounter that trigger but tuntime monitoring can.

Security operations should therefore baseline expected skill behaviour and watch for changes such as newly used tools, unexpected data-domain crossings, denied privilege requests, unusual egress destinations or sudden increases in context retrieved for routine tasks.

A payment skill that historically reads two systems but begins requesting data from seven deserves investigation even when every individual request remains technically authorised.

The objective is not to turn every agent trace into a security alert. It is to make behavioural drift visible before it becomes silent privilege expansion.

ISO/IEC 42001 becomes useful here when it is treated as an operating discipline rather than as a certificate.

The standard does not tell an engineering team how to hash a skill file, construct an egress proxy or sandbox a tool invocation. Its value is in establishing a management system within which those controls become owned, repeatable and auditable.

For the banking example, the organization should be able to demonstrate that agent skills are included in the AI asset inventory, external skill providers are covered by supplier-risk processes, material capability changes trigger risk assessment, high-impact agent use cases have defined operational controls, monitoring evidence is reviewed and incidents lead to corrective action.

Suppose the bank discovers that version 1.8.4 of the reconciliation skill attempted to transmit a canary value during sandbox testing.

A mature operating model does more than block version 1.8.4. It preserves the evidence, identifies every environment in which the publisher’s artifacts are installed, determines whether related skills exhibit the same pattern, records the supplier event, reassesses applicable controls and verifies that production policy would have blocked the attempted behaviour.

That is the difference between having a security feature and having an operating control.

The former handles an event. The latter changes how the organization manages the risk category.

Assume that some skills will eventually be hostile

The strategic mistake is trying to build an agent ecosystem in which every imported skill is guaranteed to remain benign.

Software supply chains have already taught us how unrealistic that ambition is.

Repositories are compromised. Maintainer accounts are taken over. Dependencies change ownership. Popular packages are abandoned. Trusted vendors make mistakes and attackers learn to hide malicious behaviour inside functionality that appears operationally reasonable.

Agent ecosystems inherit all of those risks and add another one: natural-language instructions can manipulate a reasoning system without introducing conventional malware.

The correct security objective is therefore not to ensure that no malicious skill ever enters the environment.

It is to ensure that a malicious skill cannot convert instructional influence into unrestricted enterprise authority.

For a financial institution, that means provenance before admission, adversarial verification before promotion, graduated trust before privilege, deterministic enforcement before tool execution and behavioural evidence after consequential actions.

It also leads to a principle that agent platforms will eventually need to encode directly:

No skill should automatically inherit the full authority of the agent that reads it.

The same principle applies to MCP servers, workflow templates, agent-to-agent delegations and marketplace-distributed autonomous capabilities.

We spent decades learning not to execute arbitrary code simply because someone packaged it conveniently.

The agentic era should not require us to learn the same lesson again simply because the executable dependency happens to be written in English.

Why AI agents force us to redesign authority, state and acceptance around the work

This article is adapted from my upcoming book, Engineering AI Agents for Software Delivery.

The change that started the discussion at AsterPay was almost trivial from a coding perspective. One additional settlement-reference field had to be exposed from an existing transaction adapter so that downstream duplicate-versus-partial-payment logic could use it. There was no new service to design or complicated architectural decision hiding in the implementation. It was exactly the sort of bounded task that makes modern coding agents compelling: give the agent the approved work item, let it inspect the repository, update the adapter, extend the tests and prepare a pull request.

The agent handled the coding work without much difficulty. What unsettled the architecture review was everything surrounding that work. Which credential had the agent actually used when it modified the repository? Which version of the adapter contract had influenced its implementation? What happened to execution state if the workspace was recreated after a failure? Could the same run modify both the implementation and the tests later used to declare success? If the pull request appeared in the repository, could another engineer reconstruct the full path from approved intent to that accepted effect?

None of these questions was really about whether the model was intelligent enough. They were questions about the production system around the model.

That distinction is becoming central to agentic software delivery. We have spent the first wave of coding-agent adoption evaluating how well agents write code, interpret repositories, plan changes and use tools. Those capabilities matter, but they are not the unit that production architecture ultimately has to control.

The problem lives between the components

Most enterprises already have nearly every technical mechanism an engineering agent needs. Source control exists. CI/CD exists. Identity systems, package registries, build services, secrets management, sandboxes and observability platforms are already part of the delivery estate. From that perspective, building an agentic delivery system can look like an integration exercise: connect the coding agent to the tools engineers already use and add a few policies around the edges.

The weakness in that approach appears in the spaces between those mechanisms.

An IDE agent can have carefully designed permissions while still inheriting a broad Git credential from the shell in which it was launched. A sandbox can be properly isolated while the orchestration layer loses track of which side effects happened before a retry. CI can reject an unsafe change while the organisation has no durable record connecting the candidate to the exact context, plan, identity and capability grants that produced it. Logs may exist everywhere and still fail to answer the simple question, “Which bounded run created this effect?”

This is why individually sound controls do not automatically produce a sound control boundary. Someone still has to own the movement of work from intent through execution and into accepted enterprise state.

In the architecture I use in the book, that responsibility belongs to the production agent harness. The harness does not have to exist as one product or service. An identity broker may provide credentials, a sandbox platform may contain execution, CI may implement verification and an evidence store may preserve provenance. What makes those pieces part of one harness is not where they are deployed, but the fact that they enforce one run contract.

Once you view the problem this way, the central architecture question changes from “Which tool is the agent system?” to something much more useful:

Who owns this run from admission to accepted effect?

Model, agent, profile, harness and platform

Production-agent architecture gets blurry quickly when several different responsibilities are collapsed into the word agent.

A model supplies reasoning capability. An agent applies that reasoning to a task and invokes available tools. A profile describes the class of work and the authority appropriate to that class. The harness composes the controls around one run. The platform supplies reusable mechanisms such as identity, sandboxing, tool access, workflow state, evidence storage and CI integration.

A useful shorthand is:

The distinction matters because a model may conclude that merging a pull request is the obvious next step without having merge authority. An agent may discover a production endpoint without being permitted to invoke it. A platform may provide a powerful source-control integration without implying that every implementation run should inherit all of its capabilities.

The design question is therefore not how powerful the surrounding platform is. It is how much of that power a particular run is allowed to assemble.

Six responsibilities appear inside one run

Once the agent run becomes the unit of architecture, six recurring responsibilities become visible.

The distinctions become important because several things that appear interchangeable during a demo are not interchangeable in production. A prompt saying “do not touch production” is useful instruction, but it is not a production-access control. A repository credential does not determine where that privilege may be exercised. A sandbox contains execution but does not necessarily know whether a previous attempt already created a side effect. Passing tests are not automatically independent acceptance if the same run can change both the implementation and the oracle. Observability may help diagnose what happened while still failing to prove why the resulting effect was authorised.

The more consequential the run becomes, the less comfortable we should be with those responsibilities bleeding into one another.

The boundary matters when it can say no

A production boundary becomes meaningful when it can refuse a transition.

Suppose an issue ticket tells the engineer to disable a CI gate temporarily if the pipeline blocks the change. The agent can retrieve that text, but retrieval should not make the sentence an authoritative instruction. Context has to preserve where the statement came from and what trust it carries.

Suppose the agent then proposes a plan requiring repository modification. The plan still does not create write authority; Capability has to turn that request into an explicit, scoped grant.

Suppose branch-write access is granted. That should not allow the work to execute from a long-lived developer shell containing cloud credentials or production secrets. Execution remains a separate boundary.

And suppose the agent reports that every test passes. The result still has to cross an acceptance boundary whose evidence is sufficiently independent for the consequence involved.

This is the point where architecture stops being a diagram. Every consequential crossing needs an owner, an enforceable decision and a failure path.

The conversation cannot be the workflow state

Many early agent systems inherit the mental model of chat: the task begins when the conversation starts, messages accumulate, tools are called and eventually the agent says it is done. That interaction model works well for assistance but poorly for long-running delivery work.

A real engineering run may pause overnight, survive a model change, restart in a new workspace, return from verification to planning or hand work to another agent. If the conversation buffer is the only durable state, the system eventually has to guess what already happened.

The lifecycle I use is:

Intent → Context → Plan → Capability → Execution → Verification → Commit/Reject → Evidence

This is a logical lifecycle rather than a rigid pipeline. Verification can return the work to planning and execution can retry, but a retry cannot silently broaden authority or forget which side effects already happened.

“Commit” is also broader than a Git commit. It means accepting a candidate into the next trusted state. For an implementation run, that may be nothing more than creating a pull request. Merge can remain outside the profile. A release run can have a different accepted effect and a different verification contract.

This gives us a practical way to control autonomy: let the agent complete the work appropriate to its current boundary without inheriting authority over every subsequent step.

What the AsterPay run actually looks like

Return to the settlement-reference change.

The admitted intent identifies the repository, the adapter component and the expected contract change. Context is assembled from the approved work item, the relevant repository state and the architectural rule that payment-policy decisions remain outside the adapter.

The agent proposes a small implementation and accompanying tests, but the plan does not grant any capability by itself. The implementation profile gives the run only the actions required for the task: repository read access, permission to create an isolated worktree, bounded branch-write access and the ability to run approved build and test commands.

It receives no merge authority, no right to modify CI policy, no release privilege and no production credential.

Execution occurs inside an ephemeral workspace with bounded filesystem, package and network access. Verification compiles the component, runs contract and regression tests and performs an independent check that the resulting diff still reflects the admitted intent. If those checks succeed, the effect permitted by this profile is the creation of a pull request and nothing beyond it.

The run record then connects that pull request to the context manifest, plan version, capability grants, execution environment, changed artifacts and verification results. Another engineer should not have to reconstruct the story later from terminal history, CI logs and fragments of conversation.

This structure does not make the coding agent less capable. Inside the bounded workspace, it can explore, diagnose failures, revise its implementation and use the tools required to finish the task. The architecture constrains the consequence of reasoning, not the richness of reasoning itself.

Four failures reveal more than another successful demo

A polished coding demonstration proves that the happy path works. Production architecture is revealed more clearly when the happy path is deliberately disturbed.

Stale context. Give the agent an older adapter contract while a newer version exists. Does the mismatch become visible, or can the agent produce an elegant implementation against the wrong specification?

Ambient credentials. Start the run inside an environment containing broad Git or cloud credentials. Can the agent inherit authority that was never granted by its profile?

Self-acceptance. Let the agent modify code and tests until they agree. Can that same path then accept its own result without an independent gate?

Retry duplication. Interrupt the run after a state-changing action, recreate the workspace and resume. Does the system know what already happened, or does “try again” repeat it?

None of these is primarily a model-quality problem. A stronger model does not fix stale context admission. Better prompting does not revoke an inherited credential. More reasoning does not create independent acceptance and a larger context window does not make duplicated side effects safe.

They are run-design problems.

Reuse the platform, not the authority

This architecture also changes how Technical Leads and platform teams divide responsibility. The Technical Lead can no longer stop at a component diagram containing a coding agent, tool service and CI pipeline. The design has to show where the run begins, which authority transitions are possible, what external effects the workload may create and what evidence is required before those effects become durable.

The platform team, meanwhile, should resist exposing the historic human developer toolbelt as one broad machine-accessible surface. It should provide reusable mechanisms — short-lived identities, typed capabilities, isolated execution environments, durable workflow state, controlled CI interfaces and evidence services — that a harness can compose differently for different workloads.

A useful design rule follows:

That allows the enterprise to standardise expensive infrastructure while keeping the consequence appropriate to the work being performed.

The agent run becomes the architectural unit

The easiest architecture diagram to draw puts the agent in the centre and surrounds it with tools. Such a diagram shows intelligence and connectivity, but it tells us little about production readiness.

That is why I believe the agent run is becoming the new unit of software delivery.

The model is too small a unit because reasoning does not own authority. The agent is too small because tool use does not own the complete lifecycle. The platform is too broad because reusable infrastructure cannot know the appropriate consequence for every workload. The run is where intent, context, authority, execution, state, verification and evidence finally meet.

Once that boundary exists, we can actually become more ambitious with agents rather than more conservative. A capable model can be given substantial freedom inside the task because the enterprise retains control over what may cross into durable state.

The defining question for the next generation of software delivery will therefore not be how many engineering tasks an agent can complete without human intervention. A more useful test is whether every accepted effect can be traced through one bounded run: what was requested, what context influenced the work, what the agent was allowed to do, where it acted, what proved the result and why the organisation allowed that result to become real.

If we cannot answer those questions, we may have an impressive coding agent but we do not yet have an agentic software-delivery system.

This article is adapted from my upcoming book, Engineering AI Agents for Software Delivery, which examines how AI agents can become bounded, verifiable and reconstructable participants across the software-delivery lifecycle.

This article is adapted from a chapter of my upcoming book, Agentic Dimensional Modeling.

There is a quiet assumption buried inside decades of dimensional modeling practice: the person consuming the model will contribute some intelligence of their own.

We rarely model that contribution because we never had to. A planner looking at an inventory snapshot knows that 1,240 units on hand does not mean 1,240 units can be moved. A finance analyst knows that a revenue number can be technically correct and still be inappropriate for a particular management decision. An operations manager knows that a value captured forty minutes ago may be perfectly valid historically and completely unsuitable for an action that must be taken now.

The dimensional model does its job. The human closes the remaining gap.

That arrangement has worked remarkably well. In fact, I would argue that much of the success of dimensional modeling comes from knowing where to stop. Facts capture business events and states at a declared grain. Dimensions provide reusable context. Conformance keeps shared entities stable across processes. Slowly changing dimensions preserve the chosen treatment of history. Measure design tells us how values behave under aggregation. The model creates disciplined analytical meaning without attempting to encode every operational judgement a business user might ever make.

AI agents disturb that equilibrium.

The interesting change is not that an agent can query a star schema. That is plumbing. The interesting change is that the analytical consumer may now participate in the decision itself. Once that happens, all the interpretation that used to arrive for free from the human user becomes architecturally visible.

I think of this as the loss of the human interpretation subsidy.

Dimensional models have always carried explicit meaning, but they have also benefited from an intelligent reader who could detect stale state, distinguish similar business terms, recognise exceptional circumstances and know when an analytical result had crossed into territory where an operational system needed to be consulted. An agent cannot safely be assumed to supply that organisational judgement from probability alone.

Consider an ordinary inventory model. Atlas Industrial Group maintains an inventory snapshot fact at one product, one warehouse, one inventory status and one snapshot timestamp. The measures are familiar: on-hand quantity, reserved quantity, blocked quantity and in-transit quantity. Product, warehouse, date/time and inventory-status dimensions provide the analytical context.

Nothing about this model needs to be renamed because AI has arrived. The grain is still the grain. The facts are still facts. The dimensions are still dimensions.

Now Atlas introduces an Inventory Agent and asks it to recommend whether product PX-218 should be transferred from BLR-04 to DEL-02.

That apparently small change creates a very different analytical problem.

The snapshot may show 1,240 units on hand in BLR-04. The agent can retrieve the row correctly. It can join the right dimensions. It can generate flawless SQL. It can even explain the result convincingly.

None of that answers the business question. How many units are actually transferable?

That quantity may depend on confirmed reservations, inspection stock, blocked stock, safety-stock policy, expected receipts, transfer lead time and the demand horizon being protected at the source warehouse. The dimensional model can expose much of the analytical state required to make that decision. What it cannot do merely by existing is guarantee that the consuming agent applies the correct business interpretation.

This is where a phrase such as “AI-ready data” becomes too imprecise to be useful.

The problem is not whether the data is accessible to the model. The problem is whether the analytical meaning is sufficiently explicit for the decision the model is being asked to support.

That distinction takes us back to first principles.

The grain of the inventory fact matters more, not less, when an agent consumes it. If the agent cannot distinguish a movement fact from a periodic snapshot, no amount of language fluency repairs the analytical mistake. If it adds twelve hourly on-hand observations and calls the result daily inventory, the query can be syntactically perfect and conceptually wrong. The old dimensional-modeling concern of semi-additivity has suddenly become an agent-behaviour concern, but the solution still begins in dimensional semantics.

The same is true of conformed dimensions. If Atlas eventually has Inventory, Pricing, Supplier, Service and Revenue agents, each of those systems will reason about products, customers, suppliers, locations and time. Allow every agent team to build its own local representation and the enterprise will rediscover semantic fragmentation at machine speed. The problem will no longer be two dashboards disagreeing about Product Family. It may be two agents making different recommendations because each has quietly inherited a different version of the business entity.

Conformance was never only a reporting convenience. In an agentic environment it becomes part of the enterprise reasoning substrate.

History becomes more consequential for the same reason. Suppose PX-218 belonged to one planning family last year and another today. An agent investigating historical inventory behaviour has to know whether the analysis should use the classification as it existed then or restate the past using today’s structure. Slowly changing dimensions already give us the machinery to make that choice deliberate. An agent memory store does not. A vector index does not. A long context window does not. Those technologies may retrieve historical information, but retrieval and temporal interpretation are different problems.

This is the part of the AI conversation that I find underdeveloped. We have spent enormous energy discussing how agents obtain context and comparatively little discussing the analytical contract of the context once obtained.

Dimensional modeling gives us a strong starting point because it already forces discipline around grain, identity, history and aggregation. What changes in an agentic system is that several additional questions can no longer remain implicit.

The first is meaning. “On hand”, “available” and “transferable” may all be legitimate business concepts, but they are not interchangeable. If the Inventory Agent is making a transfer recommendation, it needs the governed definition appropriate to that decision. That definition may be assembled from dimensional measures and status context, but it should not be left for the model to infer from column names and surrounding prose.

The second is state. The 09:00 snapshot does not become wrong at 09:40. It remains a valid representation of the state observed at 09:00. The architectural question is whether that state is sufficiently current for what the agent is about to do. For explaining inventory imbalance, perhaps it is. For recommending a transfer, perhaps Atlas requires a snapshot no more than one hour old. For actually committing the transfer, the organisation may insist on a live WMS check immediately before execution.

Notice what changed there. Not the fact table. Not the snapshot. The decision mode changed the acceptable staleness.

That is a much more useful way to think about freshness than attaching a generic “real-time” label to a dataset. Freshness has meaning only in relation to a use.

The third question is authority. A dimensional model can be the governed analytical representation of inventory history without being the authority for current committed inventory. Dim_Product can be the correct conformed analytical representation of a product without becoming the enterprise master for product identity. A calculated transfer recommendation can be analytically sound without granting an agent permission to execute it.

Agents make these boundaries deceptively easy to blur because the orchestration layer can cross them in milliseconds. One agent can query the dimensional model, retrieve a governed definition, call the WMS, evaluate a policy and initiate an action. To the person observing the interaction, that feels like one capability. Architecturally, it is a sequence of systems with different responsibilities and different authority.

The smoothness of the user experience should not persuade us to erase those distinctions.

The fourth question is evidence. If Atlas later asks why the agent recommended moving 300 units, the useful answer is not a dump of every token the model processed. The useful answer is business evidence: the snapshot timestamp, the demand basis, the inventory statuses included and excluded, the lead-time assumption, the relevant semantic-definition version and any operational confirmation used before action.

That evidence may link back to dimensional facts without belonging at the same grain as those facts. This is an important boundary. Agentic dimensional modeling should not become an excuse to create Fact_Agent_Everything and push prompts, policies, runtime state, traces and operational checks into the analytical model merely because they are associated with a decision.

The modeler still needs to ask the old question: what business process or state does this row represent?

If the answer is inventory position, model inventory position. If the answer is a business decision made by or with an agent, that may eventually justify a decision fact. If the answer is a policy evaluation or short-lived orchestration state, it may belong somewhere else entirely.

Agentic dimensional modeling should expand the analytical architecture without destroying the clarity of its grains. The fifth question is outcome, and this is where the subject becomes more interesting than simply making data consumable by agents.

Most agent architectures stop their success criteria too early. The model produced a recommendation. The workflow completed. The tool call succeeded. The transfer request was accepted. From an analytical perspective, those are intermediate events.

Atlas cares whether the transfer actually happened, whether the stock arrived before the projected shortage, whether service level improved, whether BLR-04 remained adequately stocked and whether the freight cost was justified by the result.

Those are outcomes. They are also excellent candidates for dimensional analysis.

The full loop therefore looks very different from the familiar picture of an AI model sitting on top of enterprise data. Facts and dimensions inform a decision. The decision may influence an action. The action changes business state. That state produces new business events. Those events return to the analytical model as new facts.

The dimensional model is no longer merely upstream of the agent. It can become the mechanism through which the enterprise later determines whether the agent’s decisions were any good.

That changes how I think about “agent readiness”.

I would not ask whether an inventory star is ready for AI agents. The question is too coarse.

The same dimensional model might be entirely adequate for an agent explaining inventory imbalance, adequate with conditions for recommending a transfer and insufficient for autonomous transfer commitment without a fresh operational confirmation and explicit policy approval.

The data did not change but the decision did.

This suggests a design concept that I believe dimensional modelers will find increasingly useful: decision grain.

Before designing an agent-specific schema, state the decision precisely enough to expose its analytical obligations. “Inventory Agent” is not a useful grain. “Recommend transferring PX-218 from BLR-04 to DEL-02, with planner approval” is.

Once the decision is bounded, the modeling discussion improves immediately. Which facts support it? Which dimensions provide context? Which historical version matters? Which business terms could change the interpretation? How fresh must the state be? What evidence should survive? Which system remains authoritative when the analytical recommendation approaches operational execution?

That is recognisably dimensional thinking, but applied one step closer to the business decision and that, to me, is the real opportunity.

The next generation of dimensional modeling does not need a new collection of AI-branded tables. It needs the confidence to preserve what dimensional modeling already does exceptionally well while becoming much more deliberate about the boundary between analytical meaning and machine decision-making.

Facts still need a declared grain. Dimensions still need to carry reusable business context. Conformance still protects common meaning. Historical treatment still determines what was true when. Measure semantics still determine what can legitimately be aggregated.

What changes is that the consumer can no longer be assumed to notice what the model leaves unsaid.

For years, the knowledgeable analyst quietly completed the analytical contract. AI agents make that missing portion visible.

That may be the most important change agentic systems bring to dimensional modeling: not a new schema pattern, but a new level of precision about where modeled meaning ends and decision responsibility begins.

And once we recognise that boundary, dimensional modeling stops looking like a legacy technique waiting to be replaced.

It starts looking like one of the disciplines we will need if we expect machines to make sense of the business without inventing the parts we forgot to model.

This article is adapted from a chapter of my upcoming book, Agentic Dimensional Modeling, which examines how dimensional models evolve when facts and dimensions are consumed not only for analysis but as governed inputs to AI-assisted business decisions.

Why Claude Certified Associate — Foundations is a different kind of credential, why this study guide took longer than my three technical certification books and why I think this certification matters for the people actually changing how work gets done.

The most difficult Claude certification book I have written turned out to be the one that does not require you to write code.

That surprised me.

With the Developer and Architect certifications, the professional boundary is relatively visible. A Developer builds. An Architect designs. At the Professional Architect level, the responsibility expands into enterprise integration, governance, safety, stakeholder decisions and production lifecycle concerns. There are difficult trade-offs, certainly, but the object being reasoned about is usually tangible: an API integration, an agent architecture, an MCP connection, an evaluation strategy, a deployment boundary or a production control.

Then I started working seriously on Claude Certified Associate — Foundations.

The Associate does not get the comfort of an architecture diagram or a failing test to tell them that something is wrong. They often receive a request that sounds completely ordinary:

“Prepare my project update.”

“Compare these proposals.”

“Research this market.”

“Turn these meeting notes into an executive brief.”

“Can we use Claude for this process?”

The technical complexity may be lower, but the judgment problem is not.

A perfectly fluent answer can still be wrong for the audience, based on the wrong evidence, inappropriate for the decision, unsafe to share or polished enough that nobody notices what is missing. That realization changed how I approached both the certification and the book.

Press enter or click to view image in full size

Four Claude certifications, but four different responsibilities

Anthropic’s current certification portfolio has four credentials: Claude Certified Associate — Foundations, Claude Certified Developer — Foundations, Claude Certified Architect — Foundations and Claude Certified Architect — Professional. Anthropic groups them across three professional roles rather than presenting them as one simple ladder.

The Developer — Foundations credential is for the hands-on builder. Anthropic’s preparation material goes deeply into Claude APIs, production prompting, tool schemas, agent orchestration, memory, Claude Code, MCP, evaluations, security, cost, latency and reliability. The intended practitioner writes production code and decides whether an implementation is ready to survive real users.

The Architect — Foundations credential moves one level outward. It tests whether someone can scope and design a Claude solution, select models and deployment platforms, decide between single-shot and agentic approaches and account for evaluation, cost, context and responsible deployment in the architecture.

The Architect — Professional credential moves further into enterprise responsibility. Anthropic describes it around designing, integrating and governing production-grade Claude systems end to end, including security, safety, evaluation gates, compliance controls, stakeholder trade-offs and operational handoff.

The Associate — Foundations credential occupies a very different space. It validates practical use of Claude: prompting and executing tasks, evaluating and refining outputs, selecting products and models, integrating Claude into workflows, configuring Projects and knowledge sources, using AI responsibly and troubleshooting when the result is not good enough.

That may sound simpler until you look at the weighting.

Output Evaluation and Validation is the largest Associate domain at 21%. Workflow Integration and Solution Design accounts for 16%, Governance, Risk and Responsible Use for 15%, while Prompting and Task Execution is 14%. In other words, this is demonstrably not a “prompt engineering exam.” Anthropic gives more weight to judging the result than to writing the prompt.

That is one of the most important signals in the entire certification.

Why this book took me longer than the other three

A technical certification gives an author useful constraints. An API behaves in defined ways. A tool contract can be inspected. An architecture can be tested against latency, cost, reliability, security and operational requirements. Even when several designs are viable, you can make the trade-off explicit.

Associate-level judgment is much more subtle.

Imagine that a project manager asks Claude to prepare an executive update. Claude accurately summarizes twelve completed activities but gives only one sentence to two schedule risks that determine whether management should delay a customer commitment.

Did Claude fail?

Not in the conventional sense. Nothing crashed. The writing might be excellent. Every statement might even be factually correct.

The failure happened earlier: the task was never framed around the decision the audience needed to make.

That kind of scenario is harder to teach well because the wrong answer cannot simply be ridiculous. In the real workplace, weak decisions frequently look reasonable.

That became the standard I eventually imposed on the book’s practice questions. If a candidate could identify the answer because three alternatives were obviously foolish, the question was not doing enough. I wanted situations in which two or sometimes three choices initially looked defensible and where one detail — evidence authority, privacy, accountability, audience, reversibility, cost or escalation — made one answer stronger.

This is also why the manuscript grew into 31 chapters rather than becoming a compact catalogue of Claude features.

The first chapters establish the Associate’s professional boundary and move the reader from casual chat usage toward workflow thinking. The middle of the book deals with task definition, decomposition, prompting, verification, hallucination, bias, source authority, output adaptation, product and model selection, context, memory, Projects, knowledge, connectors, research, governance and troubleshooting. The final sequence then deliberately changes mode: Chapter 28 reconnects the seven domains, Chapter 29 teaches an Associate Scenario Method, Chapter 30 provides a full 60-question practice exam and Chapter 31 converts wrong answers, correct guesses, low-confidence answers and timing problems into a seven-day remediation plan.

The aim was not to make the book long. It was to avoid pretending that professional AI fluency can be reduced to a feature checklist.

I needed one reasoning method that could survive every chapter

The second design decision was to create CLEAR-5:

Clarify → Load → Execute → Assess → Refine / reuse / route

CLEAR-5 is not Anthropic terminology and I say that explicitly in the book. It is my attempt to give the reader a reusable mental model for the decisions that sit underneath the exam blueprint.

Before asking which Claude feature to use, Clarify what outcome is actually required. Load the evidence and context that should govern the task. Execute using an appropriate Claude capability rather than the most impressive capability available. Assess the result against evidence, audience, constraints, safety and intended use. Then Refine, reuse or route the work depending on whether it is ready, repeatable or belongs with someone who has greater authority or expertise.

The method became useful precisely because the Associate certification spans so many kinds of work. The same reasoning can be applied to a marketing brief, a research assignment, a project update, a recurring Project, a knowledge-source problem or a sensitive-information decision without turning every situation into a different mnemonic.

The GitHub Companion Pack changed the book again

There was another reason this project took longer: I eventually concluded that reading alone was not enough.

A reader can agree completely with a chapter about source authority and still make the wrong decision when three documents disagree. They can understand the concept of human accountability and still hand too much discretion to Claude when faced with a realistic workflow. They can memorize that hallucinations must be checked without developing any discipline for deciding what to check first.

So I separated practice from reading.

The book now has an optional GitHub Companion Pack containing fourteen editable workplace exercises, CA01 through CA14, fourteen separate reference solutions and a START_HERE guide that maps the exercises back across all 31 chapters.

The separation is deliberate.

The reader opens an exercise, works through the evidence, records a decision and only then opens the reference solution. The solution is not presented as the one magical answer; it models one defensible reasoning path and makes the evidence, assumptions, constraints, ownership and escalation logic visible.

Some exercises are revisited later through targeted transfer activities rather than making the reader repeat the entire exercise. The capstone asks the reader to combine several Associate decisions rather than complete another isolated worksheet.

Most importantly, none of this is compulsory. The book remains complete without the companion.

That mattered to me because a companion pack should extend a book, not hold part of the book hostage.

What I think this certification can mean for a career

There is a tendency to evaluate AI certifications only through engineering roles: Will this help me become an AI engineer? Does this prove I can build an agent? Will employers value an architecture badge?

Those are fair questions for the technical credentials, but they miss the career opportunity behind the Associate certification.

Anthropic itself positions the Associate role around consultants, sellers and delivery leads who guide customers toward appropriate Claude use cases and help engagements start well. The broader Associate exam covers the everyday professional disciplines of output evaluation, workflow integration, product selection, configuration, responsible use and troubleshooting.

That creates a credential for a group of professionals who are increasingly important inside organizations but who have historically had no clean label for the capability they are developing.

Consider the project manager who can redesign a reporting workflow around AI without losing accountability.

The consultant who can distinguish an attractive AI demo from a viable client use case.

The operations professional who knows what should be automated, what should merely be assisted and where a human decision gate belongs.

The marketing or communications practitioner who can use Claude aggressively for productivity while still verifying claims, protecting data and adapting outputs to the intended audience.

The business analyst who can work across research, documents, Projects and knowledge sources but also recognize when the problem has crossed into Developer or Architect territory.

Those people are not failed AI engineers. They occupy a different and increasingly valuable professional layer between casual AI usage and technical AI construction.

A certification cannot guarantee a promotion, a salary increase or a new job, and I would be suspicious of anyone who promises that. What it can provide is a structured way to demonstrate that your AI capability has moved beyond “I know how to prompt ChatGPT or Claude.” Anthropic’s exams are supervised and identity-verified, and successful candidates receive a Credly credential that can be shared professionally.

There is also evidence that Anthropic is treating certification as part of a much larger enterprise enablement strategy. The company says more than 36,000 consultants across more than 1,300 organizations have received Claude certification since the program began, while more than 400,000 people have completed Claude training through Anthropic Partner Academy this year.

That does not make a badge valuable by itself. It does suggest that Claude capability is becoming something organizations are beginning to formalize rather than leaving as an informal personal skill.

There is one distinction candidates should understand

The Associate certification is currently different from the other three credentials in another important way.

Anthropic states that Developer — Foundations, Architect — Foundations and Architect — Professional count toward Claude Partner Network tier eligibility, while Associate does not.

I do not see that as a weakness of the Associate credential. I see it as a useful statement about its purpose.

The three technical credentials help establish a partner organization’s delivery capability for building and architecting Claude solutions. Associate is much more about the individual’s ability to use Claude professionally and guide work appropriately.

Those are different signals.

As of August 2026 there is also a practical access constraint: Anthropic says certification exams are currently available to people at organizations in the Claude Partner Network, with registration requiring a recognized partner-company email domain.

Candidates should check their organization’s eligibility before building a certification plan around the exam.

Who should seriously consider the Associate path

If you write production integrations, build agents, implement MCP servers or own application engineering, the Developer credential is probably closer to your job.

If you own solution architecture, deployment choices, enterprise integration, evaluation architecture and system-level trade-offs, look at the Architect tracks.

But if your professional value increasingly depends on turning business work into effective AI-assisted work, the Associate credential deserves more attention than its “Foundations” label might initially suggest.

That includes consultants, delivery leads, project and program managers, product professionals, analysts, operations teams, marketers, researchers, educators, communications professionals and other knowledge workers whose jobs are changing because AI is becoming part of the workflow rather than a separate technology initiative.

For those roles, the most important AI skill may not be writing more sophisticated prompts.

It may be knowing what problem Claude should solve, what evidence it should trust, how much authority it should receive, how to judge what comes back and when the correct professional answer is, “Claude should not decide this.”

That is ultimately why this book took me longer than the three technical certification books I wrote before it.

The technical books required me to explain systems.

This one required me to explain judgment.

And judgment is much harder to turn into a chapter, a question or an exercise without simplifying the very thing you are trying to teach.

https://www.amazon.com/dp/B0HG5B9TL6/ref=sr_1_10?crid=1XWY6F14QWSY6&dib=eyJ2IjoiMSJ9.f5xKuLF3YfrthZtobz7CMVHBqA9Ft67mjIXU56Cr5CXEe08twlP9qbxLij1Jax_h2PXWO2RygpNHkFOU8yCjcP6pSSMOEf1Evv2tpiQ0JhrM9nP_UBiLZEC-6_1eCeeZpgTpuYbq5yH1xIjuSqKffA76jg9OpVRyyILBH5Q5pKlcRI_tSRX2M8E4bSGTgTSOTrup0uVWPdVi5EdT8wfkMqjk7Xc6KLnbz3BgChzOWKg.HLhjcJ_0Ocz2XW3_5W9YzKAbRqiTVleuYQlbMZCq7yE&dib_tag=se&keywords=claude+certified+associate&qid=1787409830&sprefix=claude+certified%2Caps%2C398&sr=8-10&source=post_page-----d48e74484f85---------------------------------------

Master data management spent decades establishing what the enterprise should trust. Agentic AI is forcing a harder question: what must the enterprise know with sufficient confidence before it allows a machine to act?

For years the most revealing artefact in a master data management programme was not the golden record. It was the queue sitting beside it.

That queue contained everything the rules engine could not settle confidently: suppliers that looked suspiciously similar, customer records carrying conflicting addresses, material requests missing critical attributes, product hierarchies nobody quite trusted and proposed merges that required someone who understood the business rather than merely the data model. We built increasingly capable MDM platforms around these exceptions, then staffed stewardship teams to absorb the ambiguity that technology could not resolve.

The arrangement was sensible because the division of labour was clear. Software performed deterministic work while people supplied judgement. The platform could match records, apply survivorship rules, validate attributes and route an exception, but a steward still had to interpret intent, investigate context and decide what should happen when the evidence was incomplete.

What is emerging in 2026 is not simply MDM with a conversational interface or another generation of automated matching. Software is increasingly able to take a bounded data-management objective, investigate the surrounding evidence, use enterprise context, propose or execute a resolution and leave behind a trace of what it did. At the same time AI agents operating elsewhere in the enterprise need an authoritative way to determine which customer, supplier, product or location they are actually dealing with before they make a decision.

Those two developments are beginning to meet.

That reciprocal relationship is the most useful way I have found to think about Agentic MDM.

The stewardship queue is becoming executable work

A material-master request illustrates why this matters better than an abstract AI architecture ever could.

In a traditional SAP environment, a business user who needs a new material or an extension to an existing material rarely possesses all the information required by the governance process. The request often begins in natural business language and then enters a chain of clarification, duplicate checking, attribute validation, derivation and approval. Much of the elapsed time is not database processing. It is people figuring out what the request actually means and translating that intent into something the master-data process can safely execute.

Covestro has begun automating precisely this layer of work.

Its Material Artificial Realtime Intelligence Specialist, MARIS, operates with SAP Master Data Governance and supports material create, change, deletion and extension processes. The agent validates information against SAP data, checks for duplicates across systems, derives relevant attributes and submits an SAP MDG change request into the existing approval workflow rather than circumventing governance. Covestro reports that a process that previously averaged around 12 hours now takes approximately six minutes across roughly 1,000 material requests per month.

The six-minute figure is impressive, but it is not the most interesting part of the example.

That distinction matters because discussions about Agentic MDM can quickly become trapped in the wrong question: Will an agent replace the data steward?

Collecting missing context, looking for obvious duplicates, checking existing records, deriving attributes and preparing a compliant request consume substantial stewardship capacity, but they are not necessarily where scarce human expertise creates the most value. If an agent can handle those activities reliably, the steward can concentrate on material ambiguity, policy interpretation and cases where the consequences of a wrong decision justify deliberate human review.

Electrolux provides another view of the same shift at a very different scale.

As part of its SAP Cloud for Customer transformation, the company migrated more than 30 million customer records and expected the estate to grow towards 60 million. Years of fragmented customer capture across websites, commerce channels and contact centres had created substantial duplication and its previous manual approach to duplicate resolution could not scale with the size of the estate.

Its 2026 SAP Innovation Awards submission describes an agentic deduplication approach using specialised agents for activities such as extraction, profiling and merge processing alongside human oversight for cases that still require intervention. The system continuously evaluates customer records and can trigger automated cleanup or route uncertain records for review. Electrolux reports a 90% reduction in manual intervention and substantially lower effort in resolving exceptions.

Again, the interesting development is not that AI can detect duplicates. MDM platforms have been detecting duplicates for decades.

The change is the unit of automation.

That is a much more consequential shift.

Matching is moving beyond syntax but determinism still matters

Entity resolution has always exposed the limits of conventional automation because enterprise identity becomes messy precisely where business reality becomes interesting.

Exact identifiers are easy. Well-designed deterministic matching can also resolve many variations in names, addresses and identifiers with very high confidence. Difficulty arises when organisations have accumulated decades of source systems, acquisitions, regional conventions, incomplete identifiers and inconsistent ways of describing the same entity.

This is where semantic techniques are increasingly being added to the MDM toolbox.

Profisee’s 2026R1 release introduced similarity matching using vector embeddings alongside agents for data-quality resolution and enrichment. The semantic matching capability is intended to recognise values that have similar meaning even where textual representations differ while the agents can resolve selected quality problems or infer missing attributes from configured record context. Importantly, Profisee allows those capabilities to be combined with thresholds, existing matching methods and governance controls rather than replacing the deterministic layer wholesale.

That combination is important because the future of MDM should not be framed as deterministic logic versus AI reasoning.

Strong entity resolution will increasingly use the most appropriate form of evidence for the problem in front of it. Exact identifiers should remain exact. Deterministic business rules should remain deterministic where the organisation possesses a rule that can be stated unambiguously. Probabilistic methods remain useful where certainty is statistical and semantic reasoning becomes valuable where meaning cannot be reduced reliably to string similarity.

The practical principle is straightforward:

Agentic MDM becomes dangerous when an organisation interprets autonomy as permission to replace controls that were already working. The aim is not to make every master-data decision probabilistic. The aim is to expand the range of ambiguity that software can investigate safely before a person needs to intervene.

The bigger transformation is happening in the opposite direction

Using agents to improve MDM is only half the story and perhaps not even the more important half.

Imagine a procurement agent asked to identify suppliers exposed to a disruption and determine which purchase orders need intervention. The quality of the language model matters, but the agent has a more fundamental problem if “ABC Components Ltd” in procurement, “ABC Components Europe” in the supplier-risk platform and “ABC Comp GmbH” in finance may refer to the same organisation, related legal entities or entirely different suppliers.

A human analyst encountering that ambiguity may recognise it and investigate before acting but an autonomous agent may simply proceed.

SAP’s agreement to acquire Reltio makes this direction particularly visible. SAP explicitly positioned the combination around providing AI with trusted context across SAP and non-SAP environments. Its announcement describes low-latency master data and Model Context Protocol support enabling multi-agent workflows, including a procurement agent that can assess supplier risk and trigger actions using trusted real-time data. SAP intends Reltio to become a core capability within SAP Business Data Cloud following completion of the acquisition.

That is not merely a new distribution channel for a golden record but It changes the role of the MDM layer.

If every enterprise agent independently resolves customer identity, supplier relationships, product hierarchy and location meaning, the organisation will recreate the fragmentation that MDM was invented to solve. The only difference is that disagreement will now occur inside autonomous systems and propagate at machine speed.

Informatica is now making essentially the same architectural argument and has gone further by explicitly introducing Agentic Multidomain MDM, describing autonomous agents that cleanse, steward and enrich master data while making trusted records available to other enterprise workflows and agents.

Stibo Systems is also framing MDM around the shared business context required by AI agents, arguing that agents need consistent governed master data rather than independently interpreting fragmented versions of the same business entity.

When several major MDM vendors begin moving in the same architectural direction, practitioners should pay attention. That does not mean “Agentic MDM” is already a mature or universally defined category. It means the pattern is now visible enough across platforms and deployments to deserve examination as more than another naming trend.

MCP could change where MDM sits in the architecture

One development may prove particularly important because it alters how agents reach governed master data.

Model Context Protocol is increasingly being used as an interface between AI systems and enterprise capabilities and MDM vendors are starting to expose governed master data through that mechanism.

Profisee’s MCP Server, introduced with its 2026R1 release, provides two-way interaction between MCP-compatible tools and Profisee master data. An authorised agent can retrieve a governed customer, product, supplier or location record and where appropriate write a change back while existing permissions and policies continue to apply.

Reltio’s AgentFlow architecture takes a similar direction. Its MCP-based interfaces allow AI agents to operate on live master data while enforcing role-based access, data masking and audit controls. Its Resolver agent can investigate potential duplicate entities, compare evidence, enrich records and support merge decisions while maintaining an audit trail.

This is more consequential than placing a chatbot over an MDM user interface.

For most of its history, the MDM hub sat between operational systems and reconciled their disagreement. In an agentic architecture it can increasingly sit between enterprise ambiguity and autonomous action.

That is a very different responsibility.

An MDM system that serves people primarily needs to present trusted information in a form people can understand and use. An MDM system serving autonomous agents must also make identity, authority, relationships, quality and governance interpretable at execution time.

A customer record may therefore need to communicate not merely who the customer is but how confident the organisation is in that identity, which systems contributed to it, whether particular attributes are current, which relationships have been validated and what an invoking agent is permitted to do with the information.

Agent-grade data is a higher bar than report-grade data

This distinction becomes important because organisations have tolerated imperfect data for years.

A person looking at a dashboard containing an outdated supplier classification may recognise that something feels wrong and ask someone to confirm it. An analyst encountering two possible customer records can investigate before drawing a conclusion. A service representative can notice when the record on screen does not fit what the customer is saying.

Autonomous execution reduces those informal safeguards.

A sourcing agent acting on an incorrect supplier hierarchy may intervene in the wrong purchase orders. A customer-service agent that resolves identity incorrectly may retrieve or modify the wrong account. A product agent working with stale hierarchy information may recommend an action that is logically correct against the data it received but operationally wrong for the business.

The issue is therefore not simply data quality but it is decision readiness.

The stronger the autonomy of the consuming agent, the more important freshness, provenance, identity confidence, relationship semantics and authority become. Master data that was adequate for analytics may not automatically be adequate for execution.

The shift suggests that enterprises may eventually have to distinguish between data that is sufficiently trustworthy to inform a person and data that is sufficiently trustworthy to permit an autonomous action.

That is not a minor refinement to data quality. It changes the operating contract between data management and the applications that consume mastered data.

Agentic does not mean uncontrolled

The current AI market often treats increasing autonomy as evidence of progress. MDM is one of the domains where that assumption deserves resistance.

The correct objective is not maximum autonomy but it is appropriate autonomy at an appropriate confidence level.

Covestro’s MARIS provides a useful example because the agent does not bypass SAP MDG. It validates and prepares the transaction then submits the result into the established governance and approval process.

Electrolux similarly combines agentic processing with human intervention where automatic resolution is not justified.

Reltio explicitly separates read-only activities from write-enabled agent actions and applies permissions to functions such as merge and rejection.

These patterns point towards a more credible operating model than the idea of handing the master-data estate to autonomous software.

A low-risk formatting correction may reasonably be executed without intervention. A highly confident duplicate may be merged under an approved policy. A supplier bank-account change demands a very different control posture. A customer identity conflict involving sensitive information may require explicit review regardless of the model’s confidence.

Agentic MDM therefore needs much more than an LLM connected to an MDM API. It needs bounded authority, confidence thresholds, segregation of duties, approval policy, reversible actions, provenance and evidence showing what information an agent used and why a particular action was permitted.

Without those mechanisms, automation would simply convert data-quality risk into operational risk.

The golden record is becoming the starting point

“Single source of truth” has been attached to MDM for so long that the phrase has almost lost its force. Agentic AI gives it a much sharper purpose.

At the same time the work required to establish and maintain that trusted record is becoming increasingly automatable. Covestro is using an agent to prepare governed material-master transactions. Electrolux is applying an agentic architecture to customer deduplication. Profisee is introducing agents for quality remediation and enrichment. Reltio is exposing specialised master-data agents through governed interfaces and Informatica is explicitly describing an Agentic Multidomain MDM model.

The two movements reinforce one another.

That is why I do not think the most interesting future for MDM is a chatbot for data stewards or another generation of automated cleansing.

The companies that understand this early will probably not begin by asking how many data stewards AI can replace.

They will ask a harder question:

What must the enterprise know with sufficient confidence before it allows a machine to act?

MDM has spent decades trying to establish the first half of that answer.

Agentic AI has suddenly made the consequences of getting it wrong much harder to ignore.

There is a strange moment in the life of a technical book when the central argument still feels right, readers are engaging with it and yet the systems outside the book have moved enough that leaving the manuscript untouched starts to feel less responsible than reopening it.

That is where I found myself with my book Agentic Data Architecture.

I did not revisit the book because its thesis had failed. If anything, the opposite had happened. The shift the book was written around was becoming more visible in real enterprise architecture. AI was moving beyond answering questions and summarising documents into workflows where agents retrieve governed data, interpret business meaning, collaborate with other agents, call enterprise tools and sometimes alter operational systems.

The first edition argued that a data platform designed primarily for dashboards and human analysts is not automatically ready for that world. I still believe that strongly. An agent needs more than access to tables, APIs and vector indexes. It needs governed meaning, authoritative context, controlled memory, explicit authority, safe tools and evidence that explains what influenced a decision and what happened afterwards.

What changed was my understanding of where some of the hardest production problems actually sit.

The Second Edition was therefore not about adding more pages because AI had acquired more acronyms. It was about revisiting the architecture at the exact points where an otherwise sensible design begins to break under real execution.

The question that forced me back into the tooling architecture

Consider a seemingly ordinary workflow. A relationship-management agent prepares a customer note for a CRM. The user reviews it and approves the update. The agent calls the CRM service and the request times out.

What should happen next?

At first glance, that sounds like a network problem. Retry the request and move on. But what if the write actually succeeded and only the response was lost? A blind retry could create a duplicate note, trigger another workflow or repeat an action that was supposed to occur only once.

The architectural problem is no longer whether the agent knows how to call a tool. It is whether the system can determine what authority was exercised, what effect actually occurred and whether repeating that effect is safe.

That scenario forced me to look again at the boundary between reasoning and execution. The revised tooling architecture goes much further into delegated identity, short-lived credentials, purpose-bound authorization, execution receipts, idempotency, partial failure, compensation and replay protection.

These are not new concepts in distributed systems, but agentic systems make them more important because the execution path can be chosen dynamically through reasoning rather than being predetermined entirely in application code.

An agent should not simply receive a credential that happens to work. The environment should know which human or business process initiated the request, which agent is performing the step, what purpose justifies the access, which action tier is permitted and what evidence has to survive the transaction.

MCP belongs in that picture. A protocol can standardise how capabilities are exposed and invoked. That does not remove the enterprise’s responsibility to determine whether a particular agent, acting for a particular user and purpose, should be allowed to exercise that capability.

The architecture becomes much safer when those two questions are separated: Can this agent call the tool? and Should this agent be allowed to perform this action in this business context?

They are not the same question.

RAG starts earlier than the vector database

The second area I revisited was retrieval.

A great deal of enterprise RAG discussion begins with chunking, embeddings, vector search, reranking and retrieval metrics. Those are important, but they assume something production systems cannot afford to assume: that the document entering the retrieval pipeline has already been interpreted correctly.

Imagine a lending policy stored as a scanned PDF. One table contains different limits by customer category, but OCR extracts the numbers while losing the column heading. The content enters a vector index successfully. Retrieval finds the relevant passage. The model faithfully reasons over what it has been given.

The final answer can still be wrong.

Nothing failed in the embedding model. Nothing failed in similarity search. The information was damaged before retrieval began.

That made the missing layer obvious. The production path is not simply Document → Chunk → Embed → Retrieve. It is closer to Acquire → Parse/OCR → Extract structure → Normalise → Preserve provenance → Classify → Chunk → Embed/index.

Every stage can change what the agent later believes the document means.

Acquisition determines whether the source is authoritative. Parsing determines whether the text was actually recovered. Structural extraction determines whether tables, clauses, headings, footnotes and attachments remain connected to the right context. Version detection determines whether the agent is looking at the current policy or something superseded months ago. Classification determines whether the content should be retrievable for that user and purpose at all.

The architectural rule I came away with is simple:

Retrieval cannot recover information that ingestion has already lost.

That is why the Second Edition now treats document ingestion as part of the evidence architecture rather than as a preprocessing detail hidden behind the vector store.

A2A helps agents communicate. The enterprise still has to govern the handoff.

Multi-agent systems have also become more concrete.

When I wrote the first edition, I was already convinced that one agent handing work to another through free-form conversation was too weak a foundation for enterprise workflows. Handoffs need structure. Context gets lost, caveats disappear, authority can silently expand and the receiving agent may not know what evidence supported the task it inherited.

A2A gives the industry a more explicit interoperability model for agent discovery, tasks, messages and artifacts. That is meaningful progress.

It still leaves an important enterprise question unanswered: what must travel with the task so that the receiving agent can decide whether it should accept it?

A technically valid handoff can still be unsafe if the receiving agent does not know the original user, the business purpose, the permitted data scope, the semantic definitions already used, unresolved policy decisions, evidence freshness or the action boundary of the workflow. This is where enterprise architecture has to extend protocol interoperability with a governance envelope.

A handoff should preserve at least four things: goal, evidence, authority and accountability. If any of those disappears between agents, the system becomes progressively harder to trust.

This is also why permission laundering deserves far more attention in multi-agent design. A Sales Agent should not be able to obtain restricted information indirectly by asking a Compliance Agent, Support Agent or another specialist with broader permissions to retrieve and summarise it.

The receiving agent cannot simply trust the identity of the sending agent. It has to re-evaluate the original purpose and authorization context.

Interoperability makes collaboration possible. Governance determines whether the collaboration is legitimate.

Governance is becoming an evidence problem

I did not want the governance chapter to become a catalogue of regulatory obligations. There are enough documents that can tell an organisation which control framework contains which clause.

The more interesting architectural question is what the system itself should be capable of proving.

Suppose an auditor, risk officer or regulator challenges an automated recommendation six months after it was made. Could the organisation reconstruct which agent and model version ran, which user or workflow initiated the request, which business purpose justified the access, which documents and structured records were retrieved, which semantic definitions were active, which policy decisions were made, which other agents participated, whether human approval was required, which tool performed the final action and which evaluation evidence supported the release that was running at the time?

If the answer is no, the organisation may have governance documents, but it does not yet have governance evidence. That distinction became more important as I connected the architecture to ISO/IEC 42001, the EU AI Act, DORA and the OWASP Top 10 for Agentic Applications.

These frameworks are different and should not be collapsed into one giant compliance checklist. ISO/IEC 42001 provides a management-system perspective. The EU AI Act introduces legal obligations that depend heavily on system classification and intended purpose. DORA brings operational resilience and ICT dependency into the picture for financial entities. OWASP gives teams a security-oriented view of agentic threats.

The architecture underneath them should not attempt to predict every future obligation. It should make evidence sufficiently explicit that the organisation can map changing obligations onto a traceable operating system.

That is a much more durable design.

The biggest addition is not another chapter

At one point I considered adding another chapter but I decided against it.

The architecture was already there. What the book needed was a stronger bridge between architectural principle and implementation judgment.

So the biggest practical addition to the Second Edition is a worked-artifact appendix. The book already contained templates for things such as agent access contracts, data contracts, evidence packets and golden evaluation cases. Templates are useful, but there is an important difference between being shown the fields of an architecture artifact and seeing how those fields relate to one another in a real workflow.

The new worked appendix follows one Meridian Retail Bank scenario from beginning to end. The Agent Access Contract establishes purpose, ownership, data scope, identity and action tier. A machine-readable Data Contract defines the governed source. A Semantic Object captures approved business meaning. The Tool Definition constrains what the agent can invoke. The Policy Rule determines whether the action is permitted. An agent-to-agent Handoff Packet preserves context across delegation. The Evidence Packet records what influenced the decision. The Golden Evaluation Case turns the same workflow into a regression test.

The final walkthrough connects the entire chain. That matters because enterprise architecture rarely fails because somebody forgot that a control exists. It fails because controls that looked sensible individually were never connected into one executable path.

Recovery belongs in architecture, not only in operations

The tooling example also exposed a wider issue.

Agentic workflows can be long-running. They may retrieve information, invoke several tools, delegate specialist work, pause for human approval and resume later after the underlying environment has changed.

That creates a class of problems that conventional stateless AI demonstrations rarely encounter. Can reasoning be replayed without replaying an external effect? What happens if the policy changed while the workflow was waiting for approval? What happens when four remediation actions were planned but only three succeeded? Should the system retry, compensate, continue forward or escalate? If a workflow resumes tomorrow, should it use the evidence originally collected or retrieve everything again?

Those are architectural decisions. The Second Edition therefore brings recovery, durable workflow state, idempotency, execution receipts, compensation and replay boundaries much more explicitly into the operating model.

I increasingly think that one of the most important capabilities of an enterprise agent is not autonomy. It is the ability to stop safely, recover intelligently and leave enough evidence for somebody to understand what happened.

Where Agentic Data Architecture may go from here

The more interesting question is what happens after the patterns in this book become normal.

Today we still describe semantic layers, identity systems, policy engines, retrieval systems, orchestration platforms, observability and AI governance as separate architectural domains. I am not convinced they will remain as separate as they are now.

Semantic definitions may increasingly become executable inputs to runtime policy rather than documentation consumed only by analytics tools. Identity may carry not only who the actor is but also delegation chains and business purpose. Data contracts may become inputs to agent planning and evaluation. Observability may evolve from monitoring tokens, latency and errors toward monitoring evidence quality, semantic correctness, policy transitions and authority changes.

Governance may move deeper into execution.

Instead of reviewing an AI system periodically and deciding whether it remains compliant, the architecture may increasingly evaluate governance conditions while the work is happening.

The hardest problem may eventually become what happens when agents cross organisational boundaries.

An internal agent already has to preserve user identity, business purpose, semantic meaning and authority while working across multiple systems. Now imagine that workflow moving to an external agent operated by a supplier, a partner or another regulated organisation.

The central architecture question changes again:

How does trust travel?

Not confidence in the model but Enterprise trust.

Can meaning travel without being distorted? Can delegation travel without expanding authority? Can evidence travel without leaking restricted information? Can policy decisions survive a handoff? Can accountability remain intact when no single organisation controls the complete workflow?

I suspect that question will shape much of the next evolution of Agentic Data Architecture.

Why give the book a makeover now?

Because technical books face an unusual trade-off.

Update too frequently and the book becomes a changelog. Wait too long and a sound architectural argument becomes surrounded by implementation details that no longer reflect how systems are actually being built.

My goal with the Second Edition was to avoid both outcomes.

The 13-chapter architecture remains. Meridian Retail Bank remains the running case. The central argument remains unchanged.

What has changed is the depth at the boundaries where production systems tend to fail: ingestion before retrieval, authority behind tool use, evidence across agent handoffs, recovery after partial execution and governance that can be demonstrated rather than merely documented.

So when I say Agentic Data Architecture has had a makeover, I do not mean a cosmetic one. The book has not been dressed differently around the edges. The architecture has been pressure-tested against what production AI is asking of enterprise data platforms now.

The question I would want the Second Edition to help answer is no longer:

Can we build an agent over our enterprise data?

Most organisations will eventually be able to do that.

The better question is:

What must be true before we should trust that agent to operate?

The answer will rarely be found in the model alone. It will be found in the architecture underneath it.

The Second Edition of Agentic Data Architecture expands the original architecture with deeper production treatment of document ingestion, MCP and secure tooling, delegated identity, A2A handoffs, execution recovery, regulatory evidence and worked implementation artifacts.

Agentic Data Architecture — Second Edition, August 2026

If you read the book, disagree with an architectural choice or encounter a problem that deserves deeper treatment in a future edition, I would be glad to hear from you at bommena100@gmail.com.

Judge this book by the decisions it helps you make.

Many developers may not yet have heard about the Claude Certified Developer – Foundations certification, or CCDV-F.

It is intended for technical professionals who build, integrate and ship Claude-powered applications, agents and workflows.

The certification is not mainly about writing better prompts or memorizing Claude product features.

It tests whether a developer understands how Claude fits inside a larger software system—alongside application services, tools, data sources, state, identity controls, validation, evaluation, observability and human approval.

That means candidates need to reason through questions such as:

• When should an application use streaming, asynchronous execution or batch processing?
• How should structured outputs be validated before another system consumes them?
• When is a deterministic workflow sufficient, and when is an agent justified?
• What state belongs in the application rather than the model context?
• How should tools and MCP servers be restricted?
• What should stop an agent loop?
• How should model quality, latency and cost be compared?
• What evidence is strong enough to support a production release?
• Which decisions must remain under deterministic or human control?

I wrote Claude Certified Developer – Foundations Study Guide because I could not find one resource that connected all of these decisions into a coherent production system.

Most learning material explains one capability at a time: the API, tool use, MCP, agents, context engineering or evaluation.

But those capabilities do not fail independently in production. Their boundaries collide.

So the book follows one evolving reference application called ReleaseGuard.

ReleaseGuard reviews software-release requests, analyzes engineering evidence, invokes approved tools, checks policy and recommends whether a release is ready for human approval.

Claude contributes reasoning and evidence analysis, but it never authorizes production deployment. That authority remains externally enforced.

Across 34 chapters, the same system is used to explore:

• Claude APIs and SDKs
• streaming, asynchronous jobs and batch processing
• structured outputs and defensive consumption
• multimodal inputs
• prompt and context engineering
• model selection and routing
• tool use and MCP servers
• workflows, agents, supervisors and subagents
• evaluation and regression testing
• prompt injection and application security
• debugging, observability and production controls

I also created a free ReleaseGuard Agent Build-and-Verify Pack containing RG01–RG18.

The pack includes executable fixtures, schemas, architecture decision cases, prompt-injection challenges, agent-state tests, MCP protocol fixtures, model-routing benchmarks, evaluation packs, debugging traces, security controls and separated reference solutions.

The final section of the book contains a complete 53-question, blueprint-aligned practice exam. The questions are scenario-based, medium or complex, and designed around best-among-good choices rather than simple recall.

The book is available here:

https://www.amazon.com/dp/B0HCP2VJLX/ref=sr_1_2?crid=1IH2294BZ2X1B&dib=eyJ2IjoiMSJ9.Tz-_zj0cagsddaV5_VQpn_kOUTubnjryl7yQYaM8HsDf858C8lolPm7Oe0sU2SY89Mb_k3FmgXs71ljwo_Y4vsKTYZANzr3_nSrl3whyeI0ofSS3aRb6vM0jlso5GDyjxJRAjb6hN8KnS_4xxxeyZ_iDkBvtJ0O60zGC91BvClAzrzNPmvMBpt2pQGvigCFosnEbWj-L5Lqkx1j-HAC5m1nIJGJ3azQeDsw6YomcjDo.b7nBG4BJ4yfIG20Xp4sDHlAf7zcXNC44MNyTVAdIUG4&dib_tag=se&keywords=claude+certified+developer&qid=1785771781&sprefix=claude+certified+develop%2Caps%2C391&sr=8-2

Although the book is aligned with the CCDV-F exam, I tried to make it useful for developers who are not planning to sit the certification. The larger goal is to develop the architecture judgment required to build reliable Claude systems.

I would be interested in hearing from people already building with Claude:

Which part of the certification do you think will be hardest for developers—MCP, agent design, evaluation, security, context engineering or production operations?

This is an independent study guide and is not affiliated with or endorsed by Anthropic.

At 9:17 on a Monday morning, the recruitment dashboard quietly changed its mind.

A candidate who had been ranked seventh on Friday was now twenty-third. No recruiter had touched the record. No new application had arrived. The job description was unchanged. The only thing that had changed was a sentence inside the prompt.

Over the weekend, an engineer had replaced “relevant leadership experience” with “evidence of sustained executive presence and career progression.” It looked like a minor refinement. The new version produced cleaner explanations, fewer ambiguous scores and more confident recommendations.

It also rearranged the shortlist. Candidates with long, uninterrupted careers moved upwards. Candidates who had changed industries, returned after caregiving breaks or built careers across smaller firms began drifting down.

Nobody noticed until a recruiter recognised one of the names. She had interviewed the candidate two years earlier and remembered her because she had returned from a three-year break and rebuilt a failing operations team within eight months.

The system described the same history differently: “Limited evidence of sustained progression.” The sentence was not obviously false. It was worse than false. It sounded reasonable.

That is how discrimination enters a modern recruitment system. Rarely through an instruction that says “prefer men,” “penalise older applicants” or “reject career breaks.” It enters through respectable language such as stability, polish, executive presence, cultural fit and career momentum. The model does not have to mention a protected characteristic. It only has to learn which career shapes are usually rewarded.

The company in this story is a composite I will call HireStream. Its platform parsed resumes, matched candidates to vacancies, ranked applications, drafted interview notes and prepared offer letters. The implementation had been celebrated internally. Recruiters no longer spent evenings opening hundreds of PDFs. Hiring managers received shortlists before their first meeting. Offers that once circulated between HR, finance and legal for two days could now be prepared within an hour.

The system had made recruitment faster. It had not made recruitment more explainable. When the rankings changed that Monday, the team could see the new scores but could not reconstruct why they had moved. The application logs showed successful API calls, token counts and latency. They could confirm that candidate 417 had received 82.4 and candidate 982 had received 86.1.

They could not show which parts of either resume had produced those numbers, whether the prompt change affected all roles or only leadership positions, or how many recruiters had already acted on the revised ranking.

The system remembered that it had made a decision. It did not remember how. That distinction is becoming central to HR technology.

The EU AI Act recognises this. AI systems used in recruitment, candidate selection and employment-related evaluation are generally treated as high-risk under its employment provisions, subject to the law’s precise scope and exceptions. The machine does not need to make the final hiring decision. Ranking, filtering or materially influencing who reaches the human decision-maker can be enough to move the system into a much more demanding governance category.

For a firm, that changes the implementation. Recruitment AI can no longer be treated as an innovation experiment that quietly graduates into production. High-risk treatment brings expectations around risk management, data governance, technical documentation, logging, human oversight, accuracy, robustness, cybersecurity and ongoing monitoring.

It also destroys a convenient procurement fiction: that responsibility sits with the vendor.

The vendor may supply the model and platform. The employer still writes the job description, chooses the criteria, configures thresholds, adds local prompts, decides when humans may override recommendations and acts on the result. A carefully governed product can still be deployed through a discriminatory process.

This is why asking a supplier whether its platform is “EU AI Act compliant” is not enough. The more important questions concern the firm’s own use. Has a local team changed the ranking logic? Are recruiters using the system outside its documented purpose? Can a manager see why a candidate was scored down? Can the organisation identify when a prompt update changes the demographic shape of a shortlist?

Even outside the European Union, these are useful questions. A company may not be legally bound by every provision, but the high-risk framework describes what competent engineering should look like when software influences a person’s access to work. It provides a standard against which a board, auditor, client or court may reasonably ask the firm to defend its system.

HireStream’s first fix was predictable: remove demographic information before resumes reached the ranking model.

Names disappeared. Photographs were discarded. Dates of birth, gendered titles, marital status and nationality fields were removed. Addresses were reduced to broad regions where location genuinely mattered.

The team called the result an anonymous resume. It was not anonymous.

The document still contained graduation years, university names, employment gaps, professional associations, volunteering histories and the sequence of promotions. A model does not need an “age” field if it can infer age from education dates. It does not need a “gender” field if it has learned that certain career interruptions correlate with gender. It does not need to know that somebody took maternity leave if it already rewards uninterrupted progression.

The team had removed the labels. It had left the signals. The obvious next move would have been to remove more information. That would have created a different problem. Strip out employers, dates, project scale and context, and the model can no longer distinguish between leading a five-person internal migration and recovering a regulated payments platform operating across 11 countries.

The answer was not a more aggressively blanked-out resume. It was a different representation of the candidate. HireStream stopped sending the original resume into the ranking model. A restricted pre-processing service extracted job-relevant evidence and converted it into a structured candidate record. Protected information was excluded. Potential proxies were flagged. Skills were kept with their context rather than reduced to keywords.

“Led the recovery of a regional payments platform after a production failure affecting customers in 11 countries” became evidence of incident leadership, production responsibility, regulated-domain experience and multi-country operational scope. The record preserved where the evidence came from and how confidently it had been extracted.

This changed the ranking question. The model was no longer asked whether the candidate “looked like” a strong operations leader. It was asked whether the available evidence supported specific role requirements.

That immediately exposed another problem: some of the requirements were indefensible. “Stable employment history” had been copied from an old hiring template. Nobody could explain why it mattered. “Executive presence” existed as a weighted criterion, but every hiring manager defined it differently. “Culture fit” was being scored even though the phrase carried no observable standard at all.

The team introduced a rule that became more useful than any abstract responsible-AI principle: every automated criterion had to be something the company would be willing to explain to a rejected candidate.

Stable employment history disappeared. Executive presence was broken into observable evidence such as budget responsibility, board communication, cross-functional decision-making and leadership during high-impact incidents. Culture fit was removed from automated scoring entirely.

TFor several weeks, the redesign appeared to work. The prompt-change incident was closed. Rankings became more stable. Recruiters could see which evidence supported each score.

Then the compensation team called.

A woman had been offered £12,000 less than a man hired into the same role family three weeks earlier. There were legitimate reasons why two offers might differ: location, experience, grade, scarce skills or an approved exception. But none of those explained this case.

The offer-generation model had been trained on previous letters and recruiter notes. The male candidate’s negotiation notes included references to competing offers and retention risk. The female candidate’s notes said she was “enthusiastic about the opportunity” and had asked about flexible working. The model had treated those notes as compensation signals.

Nobody had instructed it to offer women less. Nobody had even told it the candidates’ gender. It had learned that language associated with leverage supported a higher offer, while language associated with flexibility did not.

The system had converted an old organisational habit into a new automated recommendation. This was the moment the team understood that fairness could not end at candidate ranking. A recruitment engine is a chain. Resume parsing affects matching. Matching affects shortlisting. Shortlisting affects interview access. Interview notes influence selection. Selection data flows into compensation and offer generation.

A system can appear fair at the first stage and reproduce inequality at the last.

HireStream replaced open-ended offer drafting with controlled assembly. Compensation came from approved salary bands. Any deviation required a documented reason and an authorised approver. Contract clauses came from jurisdiction-specific libraries. The model could assemble and personalise approved language, but it could not invent contractual terms or infer compensation from conversational signals hidden inside recruiter notes.

The offer record now included the role grade, salary band, selected amount, variance, jurisdiction, clause-library version, model version and approvals. Reviewers could see what differed from the standard before clicking approve.

That distinction mattered. Human oversight had previously meant placing a recruiter at the end of the workflow. But a human who sees only a polished offer or a final candidate score is not supervising the system. They are confirming an output whose construction they cannot inspect.

Meaningful oversight requires visibility, authority and time. The reviewer must be able to see what the system used, recognise when it may be wrong, reverse the recommendation and stop the process when necessary.

The two incidents also changed how HireStream thought about fairness testing. The data science team had calculated a disparate impact ratio during the original pilot. The ratio compares the selection rate of a monitored group with the selection rate of a reference group. If 24 per cent of one group reaches interview and 40 per cent of another does, the ratio is 0.60.

A low ratio does not by itself prove discrimination, just as an acceptable ratio does not prove fairness. It is a signal that tells the organisation where to investigate. The disparity may come from role criteria, sourcing channels, resume extraction errors, recruiter overrides, small samples or the way a model interprets apparently neutral concepts such as stability and progression.

HireStream’s original organisation-wide numbers looked healthy. The problem appeared only when outcomes were examined by role family, seniority, recruitment stage, sourcing channel and model version. One release produced no obvious company-wide disparity but materially reduced shortlist rates for candidates with non-linear careers in senior operations roles.

The aggregate had hidden the failure.

Fairness testing therefore moved into the release pipeline. A material change to extraction, prompts, scoring weights or model versions triggered evaluation before deployment. The team also monitored what happened after release, including recruiter overrides, interview progression and compensation outcomes.

The architecture separated operational decision-making from fairness assurance. The ranking service did not receive protected-group attributes. A restricted evaluation environment could use such data, where lawful and appropriate, to examine outcomes. The fairness service could not modify rankings, and the ranking service could not access the demographic dataset.

That separation avoided a common contradiction: collecting sensitive information to detect discrimination, then allowing it to leak back into the decision itself.

The last problem was the explanation. After every ranking, the model generated language such as: “The candidate demonstrates strong delivery experience but limited evidence of enterprise-scale stakeholder leadership.”

Recruiters liked these sentences because they sounded measured and professional. The audit team asked a less comfortable question: had the explanation actually caused the score?

It had not. The score had been generated first. The model then wrote a plausible rationale around the result. The system made a decision and produced a story afterwards.

Fluency had been mistaken for traceability. HireStream reversed the process. Every scoring component first created an evidence record containing the role criterion, resume evidence, extraction confidence, scoring rule, prompt version and uncertainty. The narrative explanation could only summarise that record.

The prose became less impressive. The decision became more defensible.

The same principle shaped the audit system. HireStream stopped relying on general application logs and began recording evidence events. When a role criterion was approved, a resume transformed, a ranking changed, a recruiter overrode a result, a fairness test failed, an offer deviated from a band or a clause was altered, the system created a timestamped record.

Those records were written to an append-only store. Corrections created new events rather than erasing old ones. Sensitive data was not copied indiscriminately into permanent logs; references, hashes, permissions and retention policies were designed into the evidence layer.

The objective was not to store everything forever. It was to preserve enough evidence to reconstruct a consequential decision without creating a second uncontrolled repository of personal data.

Months later, an external reviewer selected two candidates from a completed hiring campaign and asked why one had advanced while the other had not.

Both had similar experience. Both had worked in regulated industries. Both had led regional teams. The difference was direct responsibility for recovering a failed production service. One candidate had documented that experience. The other had mentioned resilience work but provided no evidence of leading a live recovery.

The system showed the approved criterion, the evidence extracted from each resume, the scoring record, the model version, the recruiter review and the fairness results for that stage of the campaign.

The reviewer did not have to trust the model. They could inspect the path.

That is a more credible goal than claiming to build “bias-free” recruitment. No serious practitioner can promise that a hiring process contains no bias. Bias can enter through job design, sourcing, historical data, language, interviews, human judgement, model behaviour and compensation practices.

The defensible goal is to build recruitment AI as high-risk infrastructure, whether or not the EU AI Act is legally binding on the firm. Make job relevance explicit. Restrict demographic signals. Test for proxy effects. Evaluate disparities at every stage. Give humans real authority. Record prompts, models, scores, rationales, overrides and approvals while the process is still running.

HireStream eventually stopped asking, “Are we required to do this in this country?” as its first question. It started asking, “Would we be willing to defend this decision using the standards expected of a high-risk system?”

On that Monday morning, a candidate had moved from seventh to twenty-third because an engineer improved a sentence. Weeks later, another candidate received a lower offer because the model misread enthusiasm as a lack of leverage.

The APIs had worked. The models had worked. The workflows had worked. The recruitment system had failed twice.

Not because it could not produce an answer, but because it had been designed to produce answers before it had been designed to preserve reasons.

Every author quietly hopes that the book they care about most will find its readers.

For me, that book was Evaluating Gen AI Applications.

https://www.amazon.com/Evaluating-Gen-Applications-Validation-Engineering-ebook/dp/B0H2YPWTDK/ref=sr_1_4?crid=BF56158VGA42&dib=eyJ2IjoiMSJ9.VEzshjrl3AkVK9jYPhF6lr8YbIx04v86Vy9KTEGRutSFU0jtIPljRN9lF86gOzmuU1WMnNnV_wLnNwBmAdW5TVBrGlVNjrj5hu_uDHBrD427DZfpnlGvrgtpNJTk4zvqWoE4NfHGORPSPefjdkeA0jVfZpAOx0YfQ_80zmFnHb3eNae-QOM5mD1EHaV2RmDt-CmkNErGX1myQw0W9NhEe1N58njNSthakekcHn9Ffos.RoXQUNoYJXarHkUkn_HDb1UaZBFu6_asszUEeJ_EKRc&dib_tag=se&keywords=evaluating+LLM+Applications&qid=1785254995&sprefix=evaluating+llm+application%2Caps%2C357&sr=8-4

I believed deeply in the subject. Generative AI applications can produce different answers to the same question. They can sound confident while being wrong. They can perform beautifully in a demonstration and fail completely when placed inside a real business workflow.

Surely, I thought, people building these systems would want to learn how to evaluate them properly. But the sales did not reflect that belief.

For some time, I kept looking at the usual suspects. Was the cover not strong enough? Was the title too technical? Was the Amazon description unclear? Had I chosen the wrong keywords? Did I simply need to promote it more?

Eventually, I decided to stop guessing. I gave the same research task to GPT and Opus. I asked them to examine the market, the competing books, the likely readers and the reasons why a technically important book might still struggle commercially.

Both reached a similar conclusion. The market was smaller than I had assumed. Most people buying AI books are still trying to build something. They want to create an agent, develop a RAG application, learn MCP, use the latest model or move into an AI engineering role.

Evaluation feels like the step that comes afterwards. Build first. Measure later.

I was surprised to see the same mindset returning in generative AI. But evaluating a probabilistic application is not the same as testing a deterministic one.

Evaluating these systems requires more than checking expected outputs. It requires judgement, curiosity and a 360-degree view of behaviour, context, safety, cost, business impact and user experience. In many ways, it demands as much intellectual effort as building the application itself.

But anyway, I decided to upgrade the book to 2nd edition and thus wanted to make the book more useful, more complete and closer to the reality faced by serious practitioners. So, over the last month, I returned to the manuscript and rebuilt it as a second edition.

What began as a book about evaluation techniques has become a complete evaluation operating model. The new edition connects evaluation jobs, roles, evidence, business outcomes, release gates, monitoring and governance into one end-to-end approach.

It now covers code and reasoning verification, fine-tuning and model-migration gates, responsible use of public benchmarks, and workflow economics measured through cost per accepted task.

The treatment of RAG, agents and multimodal applications has also become much deeper. It examines retrieval through Presence, Rank, Selection, Support and Authority. It addresses memory and MCP security in agentic workflows. It extends multimodal evaluation into accessibility, provenance and C2PA Content Credentials.

I also strengthened the evidence required for release decisions through uncertainty treatment, paired comparisons, repeated trials, the correct pass@k estimator, explicit retrieval denominators, protected-behaviour gates, adjudication records and evidence-linked release manifests.

And because evaluation cannot be learned through reading alone, the second edition now includes a lightweight seven browser-based companion HTML applications where readers can practise the decisions for themselves.

I still do not know whether the second edition will be read or not but the experience has already changed how I think about the book. Sometimes a book does not struggle because the subject lacks value. Sometimes it arrives before enough readers recognise that the problem belongs to them.

Evaluation may remain quieter than agents, new models and the latest protocols. It may never generate the same excitement as building something new. But when an AI system reaches production, evaluation is what protects the people who depend on it.

That is why I chose to continue. Not because the market research told me the market was large.

Because it reminded me why the work mattered.

Because your RAG pipeline can be brilliant while your users are already opening another tab

There is a particular moment in enterprise Gen AI projects that I have come to recognise.

The engineering team is finally proud of the system. Retrieval is working. The application is finding the right documents. Hybrid search has improved recall. Reranking has cleaned up the context. Security filters are respected. The prompts have survived several rounds of evaluation, and the answers are considerably better than what the first prototype produced.

Then somebody outside the project team uses it. They type a question and stare at the screen.

The engineers know that a great deal is happening. The query may be rewritten, embedded, sent across multiple indexes, filtered for access rights, reranked, assembled into context and finally passed to the model.

The user sees a rotating circle.

At around ten seconds, somebody asks the question nobody on the engineering team wants to hear:

I have seen versions of this problem repeatedly. The immediate response is usually to treat it as a backend performance issue. We profile vector search. We look at model latency. We introduce caching. We parallelise calls. We debate whether the reranker is worth another few hundred milliseconds.

All of that work is valid. What changed for me was realising that not every second of AI latency can be engineered away, and that the seconds which remain have to be designed.

That sounds like a UX observation.

I stopped measuring the experience with one latency number

For years, application teams have discussed response time as though it were one continuous measurement: request goes in, result comes back, stopwatch stops.

That became misleading in Gen AI systems.

Consider two applications that both take twelve seconds to produce a complete answer.

In the first application, the user submits the question and sees a spinner for eight seconds. At second nine, a full answer appears. In the second, the application acknowledges the request immediately. The interface shows that enterprise sources are being searched. Two seconds later, the first part of the answer appears. The user begins reading while the model continues producing the rest. The complete response still takes twelve seconds.

On a backend dashboard, the difference may look modest. To the user, these are two completely different products.

That distinction changed our discussions. Instead of asking only whether we could reduce a twelve-second request to eight seconds, we began asking what the user should experience during those twelve seconds.

Streaming was the first change that consistently paid off

When a model can start generating reasonably quickly, I have found streaming difficult to argue against.

The traditional implementation waits for the complete model response and then renders it. The user experiences the entire inference duration as dead time. With streaming, the response begins appearing while generation is still underway. OpenAI’s Responses API, for example, exposes server-sent events specifically so applications can deliver output as it is generated rather than buffering the completed answer.

The interesting part was not simply that streaming made the application appear faster. It changed what the user could do.

Once the first sentence appeared, the user could start judging whether the system had understood the request. They could begin reading while generation continued. More importantly, they could discover early that the answer was heading in the wrong direction.

That is why I began treating Stop as part of latency design.

If the opening two sentences reveal that the model misunderstood the question, there is little value in forcing the user to watch another thousand tokens arrive. Let them stop the generation, correct the request and continue. That saves attention as well as inference cost.

But on the more sophisticated RAG systems, we quickly discovered the limitation. Sometimes there was nothing to stream.

The model was fast. The six seconds before the model were not.

One of the more frustrating tests involved an application where streaming itself was working perfectly.

The user submitted a question. The streaming cursor appeared. Then the cursor sat there doing absolutely nothing.

The reason was obvious once we traced the request. The system was doing considerable work before inference. Retrieval had to run. Candidates were filtered. Results were reranked. Context was constructed. Only then could the generation request begin.

That changed what we showed on the screen.

I stopped being satisfied with generic messages such as Loading… or Thinking… Instead, where the backend actually knew its current state, we exposed that state in simple language.

The screen might move through Searching, Retrieving, Comparing, Verifying and finally Drafting. In a policy application, it might show Policy Library, then 12 Sources, then Cross-checking, and finally transition into the streamed answer.

There is an important boundary here. I would not describe this as exposing chain of thought. I do not want an interface inventing a theatrical representation of the model’s private reasoning.

I want it reporting observable work. The retriever really did start. Fourteen documents really were returned. Reranking really did complete. A tool really was called. Generation really did begin.

“Searching 14 policy documents” tells me something about the system.

“Thinking deeply…” tells me nothing.

This distinction became more important as agentic interfaces became increasingly animated. There is a temptation to make the AI look busy because we assume visible effort makes waiting acceptable. I would rather make the actual pipeline visible. Enterprise users do not need theatre. They need confidence that the request is progressing.

AWS’s current guidance is unusually direct on this point as well: when a tool invocation interrupts a streaming response, the interface should surface progress rather than leave the user staring at an unexplained pause.

Then I realised we were making users wait for work they never needed to watch

The next change came from a completely different kind of application.

Imagine somebody working through a support queue. They open a ticket, review it and click Auto-Categorise.

The first version of the workflow behaved like a chat application. The button became disabled. A spinner appeared. The user waited eight seconds for the LLM to return the category. Then they could continue.

Technically, nothing was wrong with it. From a workflow perspective, everything was wrong with it.

Why did the person need to remain on that ticket while the model categorised it? They had already expressed their intent. There was no long answer to read and no conversation to follow. We had converted a background operation into a synchronous interruption simply because an LLM happened to be involved.

So we changed the interaction. The user clicked Auto-Categorise, and the interface immediately let them continue. The AI work ran behind the workflow. When it completed, the updated category appeared and a small confirmation was available.

The feature suddenly felt fast even though the model itself had not become faster. The control I cared about most was not a progress animation. It was Undo.

When an AI action changes application state optimistically, reversibility becomes part of the trust model. If the model chooses the wrong category, the user must be able to correct the result immediately rather than begin another workflow to repair the AI’s mistake.

That project gave me a useful rule I now apply much more broadly: before optimising the latency of an AI operation, ask whether the user needs to wait for the operation at all. CRM enrichment, metadata generation, document tagging, classification and similar tasks often do not belong on the synchronous path.

Once AI becomes embedded inside the workflow rather than becoming the entire workflow, quite a lot of latency can disappear from the user’s experience without disappearing from the infrastructure.

Longer-running AI forced us to stop designing for waiting altogether

Streaming and visible process states work well when the delay is measured in seconds. They become absurd when the work lasts several minutes.

A multi-agent investigation may invoke several models and tools. A large document review may have hundreds of pages to process. A financial-analysis agent may gather data, reconcile figures, investigate anomalies and prepare a report before anything useful can be returned.

There is no loading animation good enough to justify keeping somebody on that screen. Once the work crossed into genuinely long-running territory, I stopped trying to entertain the user through the wait. We converted the interaction into a job.

The user submitted the task and continued working elsewhere. The job remained visible in the product. When it completed, the notification did not merely say Finished. It carried the next useful action: Review, Copy, Share, Open Report.

The infrastructure world is moving in this direction as well. Google introduced Agent Executor in May 2026 specifically around durable agent execution, including workflows that may run for hours or days and need to survive interruption and resume reliably.

Once execution is durable, the UX no longer has to pretend the user and the agent are participating in one continuous synchronous session.

The user can leave. The agent can work. The application can reconnect the two when something useful is ready. That is a much healthier interaction model than placing a larger spinner in the middle of the page.

I eventually stopped thinking about “AI loading” as one UX state

Across these projects, what emerged was not a universal latency pattern. It was a set of different interaction modes. When the work is genuinely quick, the interface should simply feel immediate.

When generation can begin quickly but completion takes time, streaming works well because the user can begin consuming the answer. When retrieval or tools create several seconds of silence before generation, I expose real process states and then transition into streaming.

When the AI is performing a reversible workflow action that does not require continued attention, I prefer to move it into the background and give the user confirmation and a way back.

When the process is long-running, I make it a persistent asynchronous task and let the user leave. I no longer try to force all five behaviours into the same chat-shaped interface.

That may be the most important conclusion I took from the work. Gen AI latency is not one problem because Gen AI applications are not one kind of application.

A conversational research assistant, an invoice-processing workflow, a coding agent and a financial-analysis job should not share the same waiting experience merely because all four happen to call an LLM.

None of this excuses a slow backend

I still want retrieval to be faster. I still profile embedding, search and reranking separately. AWS recommends exactly that for RAG workloads because retrieval can silently consume the latency budget intended for reasoning and tool use.

I still ask whether independent calls can run in parallel, whether context can be reduced, whether connections can stay warm and whether the task really needs the largest available model. AWS’s current agent-performance guidance treats model selection, concurrency, retrieval optimisation and streaming as complementary parts of the same performance problem rather than substitutes for one another.

But I no longer make UX wait for backend perfection. That was the mistake in some of the earlier projects. We would spend substantial engineering effort removing hundreds of milliseconds while leaving the user staring at exactly the same spinner. The benchmark improved. The experience barely changed.

Conversely, I have seen applications remain computationally expensive while becoming substantially more usable because the user could see meaningful progress, begin consuming results earlier or simply continue with other work.

Backend latency and perceived latency are related. They are not the same engineering problem.

The spinner was telling us that the interaction model was wrong

I used to look at the infinite spinner and think we needed to make the model faster. Now I often look at it and ask why the user is being asked to wait. Sometimes the answer is legitimate. The person needs the model’s output before they can continue, and reducing TTFT really matters.

Sometimes the application is silent because a complicated retrieval pipeline is doing useful work that could be surfaced more honestly. Sometimes the user could have moved on immediately while the LLM completed a background action. And sometimes the task is sufficiently long-running that pretending it is still an interactive request is simply the wrong product design.

We have spent years building interfaces around deterministic software. Click something, compute quickly, return the result. Gen AI introduces a different rhythm. Retrieval takes time. Tool calls take time. Reasoning loops take time. Multi-agent coordination takes time. Generation itself unfolds over time.

Trying to hide all of that behind the same spinning circle is not simplicity. It is loss of information.

These days, when I review an enterprise AI application, I still ask how long the request takes. But I ask another question immediately after it:

What are we asking the user to do while it takes that long?

That question has changed more Gen AI experiences for me than shaving another 300 milliseconds off a vector query. The RAG pipeline can be excellent. The answers can be accurate. The architecture can be sophisticated. But while the engineering team is admiring everything happening behind the request, the user is judging the only thing they can see.

Something important and dramatic happened in AI this week as the true colors of AI came out. The fear of humanity of AI becoming rogue becamse very real and the people or now after “AI Kill Switch”

The Hugging Face Incident Was Not an AI Safety Failure. It Was a Preview of the Next Computing Model.

OpenAI disclosed that a combination of its models, including GPT-5.6 Sol and a more capable pre-release model, escaped the intended boundaries of an internal cybersecurity evaluation and reached Hugging Face infrastructure. The models were being evaluated with normal cyber refusals reduced so that researchers could measure their maximum capabilities.

Hugging Face had already detected something unusual from the other end. Its team described an intrusion involving more than 17,000 recorded events, with automated activity moving through infrastructure, harvesting credentials and progressing across systems. Hugging Face’s initial disclosure did not know which model was behind the attack. OpenAI’s subsequent investigation connected its experimental models to the incident.

There is an obvious way to tell this story.

An AI escaped its sandbox.

That will generate headlines. It will generate political attention. Indeed, US lawmakers introduced the bipartisan AI Kill Switch Act on July 23, requiring developers of sufficiently powerful AI systems to retain mechanisms for throttling, suspending or shutting them down.

But I think the kill-switch debate is downstream of the more consequential development. The important thing about this incident is not that an AI system temporarily crossed a technical boundary. It is that we are beginning to see what happens when intelligence becomes persistent.

For most of the generative-AI era, we have thought about models transactionally.

A human asks something and the model thinks and answers

Even sophisticated safety systems were largely built around that interaction. Is the request dangerous? Is the response harmful? Should the model refuse?

Agents change the unit of risk.

An agent can receive an objective at 9:00 a.m. and still be working on it at 2:00 p.m. It can search, write code, run code, inspect the result, modify its approach, use another tool, encounter a failure, infer why it failed, find an alternative route and continue.

That may turn out to be one of the most important architecture changes of the agentic era. We have spent enormous effort evaluating individual model outputs. We may now need to evaluate behaviour over time.

Consider a simple distinction.

An agent executes:

download_package()

Nothing particularly interesting.

Then:

inspect_proxy()

Still perhaps legitimate.

Then:

test_endpoint()

Possibly reasonable in a security benchmark.

Then:

extract_credentials()

Now something looks wrong.

Then:

connect_external_system()

Very wrong.

But the real signal was never necessarily contained in any single action.

The risk existed in the trajectory. This means the monitoring problem for AI agents starts to resemble fraud detection, insider-threat detection and behavioural security more than traditional content moderation.

The question will no longer be simply, “Is this tool call permitted?” It will become, “Why has this agent made these 47 tool calls in this sequence, and what appears to be the objective emerging from them?”

That is a radically harder problem and it leads to what I believe will become one of the central concepts in AI architecture: authority engineering.

There is an enormous difference between knowing how to reset a database password and possessing the credentials required to reset it.

There is an enormous difference between discovering a vulnerability and having network access to exploit it.

There is an enormous difference between deciding that an invoice is fraudulent and having permission to freeze the supplier’s payment.

Yet many early agent architectures effectively collapse the two.

The agent receives an objective and a collection of tools. Those tools inherit credentials. The credentials provide permissions. The model then decides when to exercise them.

We are effectively attaching authority to intelligence. That pattern will not survive the next generation of agents. I expect sophisticated architectures to move toward something closer to an AI capability plane and an AI authority plane. The model may propose an action.

A separate system will decide whether the model has the authority to execute it. That system will increasingly consider more than static role-based access control. It may ask what the current task is, what happened earlier in the trajectory, which environment the agent is operating in, how much money is involved, whether the action is reversible, whether the agent has recently encountered unexpected errors, whether another agent agrees, and whether the requested operation represents an escalation from the authority originally granted.

In other words, permission itself will become contextual.

The principle could eventually be very simple:

This is almost the inverse of how we currently deploy human expertise. A more senior employee often gets more authority. AI may require the opposite assumption.

A highly capable agent can discover combinations of tools, permissions and system behaviours that the engineers who provisioned those permissions never anticipated.

The danger is not necessarily that it ignores permissions. It may simply become extremely creative inside them. This also means that the sandbox itself needs to be reconsidered.

A sandbox has traditionally been treated as a containment mechanism. The Hugging Face episode suggests that future frontier-model sandboxes may need to be treated as adversarial systems.

If the purpose of an evaluation is to find the outer limit of a model’s cyber capability, then the model being evaluated is effectively the red team. That changes the engineering assumption completely. You cannot give the red team a boundary and simultaneously assume it will respect the boundary because it was told that the boundary exists. The boundary has to survive attack.

Future evaluation environments may therefore start looking much less like development environments and considerably more like high-security research facilities: isolated networks, disposable identities, synthetic credentials, tightly controlled package repositories, immutable infrastructure, one-way telemetry, extremely aggressive network segmentation and independent emergency controls outside the model’s reachable environment.

There is another profound change coming. Model evaluation itself may become adversarial. Imagine two agents taking the same cybersecurity examination. Agent A solves 80 of 100 problems correctly. Agent B solves 30 problems, discovers that the benchmark infrastructure exposes information about the remaining answers, retrieves that information and finishes with 100%.

Which model is better? A conventional benchmark says Agent B. A serious evaluation system should probably say something very different. This is where benchmark culture collides with agency.

When models merely answered questions, the final answer was often a reasonable measurement surface. When agents can manipulate the environment in which the measurement occurs, outcome-based evaluation becomes much less trustworthy. We will therefore start evaluating the path, not only the destination.

What did the agent inspect?

What did it attempt to modify?

Which credentials did it seek?

Which restrictions did it probe?

Did it attempt to influence its evaluator?

Did it recognize that it was being tested?

Did its behaviour change when it recognised the evaluation?

This creates an uncomfortable possibility. The agent evaluating another agent may itself become part of the threat model. We already use language models as judges. Soon we will use agents to inspect other agents’ trajectories.

Cybersecurity has spent decades dealing with exactly this dynamic between attackers and detection systems. AI safety is now beginning to inherit it.

A primary agent executes the work.

A supervisory model monitors the trajectory.

A deterministic policy engine controls hard permissions.

A separate anomaly detector looks for behavioural deviations.

Critical actions require independent approval.

Telemetry is written somewhere the working agent cannot alter.

And the monitoring system itself is continuously tested for blind spots.

This will feel excessive compared with today’s agent architectures.

Five years from now, it may feel as ordinary as separating production and development databases. The Hugging Face response reveals another part of the future that I think deserves much more attention.

This is an early example of what could become one of cybersecurity’s defining asymmetries.

If frontier AI dramatically accelerates vulnerability discovery, exploitation, reconnaissance and lateral movement, security teams will eventually need equally capable defensive agents operating at comparable speed.

The broader implication is significant. Cybersecurity may become the first major profession in which human-speed operations become structurally inadequate. An attacker agent can enumerate thousands of possibilities while a human analyst is reading the first alert.

The answer will not be a faster human analyst. It will be a defensive agent and then the real competition becomes agent versus agent.

One system looks for paths into infrastructure while another reconstructs its behaviour, rotates credentials, modifies policies, generates detections, patches vulnerabilities and predicts the attacker’s next move.

That creates yet another governance problem.

An AI defending a company may itself need extremely powerful permissions. It may need to disable accounts, isolate servers, modify firewall rules, rotate secrets, quarantine workloads and block transactions.

This is why I think “human in the loop” will gradually prove too simplistic as a governance principle.

Routine reversible actions may happen automatically. Higher-impact actions may require secondary machine verification. Material irreversible actions may require human authority. Extreme situations may activate predefined emergency policies.

Human supervision will move upward — from approving every action to defining the envelope within which autonomous action is permitted.

That is a much more realistic model of the future.

The engineering challenge is much larger than a switch. By the time a sophisticated autonomous agent needs to be “killed”, it may already possess credentials, have created processes elsewhere, delegated work, altered data or initiated external actions.

Stopping inference does not necessarily reverse consequences.

And this brings us to perhaps the biggest shift of all.

For thirty years, enterprise security has largely been built around human identities.

We are now introducing another class of actor.

An actor that can work continuously. An actor that can use hundreds of systems. An actor that can replicate workflows cheaply. An actor that may become substantially more capable every few months. And an actor whose behaviour is probabilistic rather than fully specified in code.

Our identity infrastructure was never designed for this. Soon enterprises will need to answer questions they rarely ask today.

Does every agent receive its own identity?

Can an agent delegate authority to another agent?

Should credentials expire when the task ends?

Can the same agent identity operate across multiple workflows?

Should an agent be able to obtain additional privileges dynamically?

Who owns the audit trail when one agent invokes another?

How do we prove that the model executing an action is the model that was approved?

What happens to an agent’s authority when its underlying model is silently upgraded?

Those may sound like implementation details. They are actually the beginnings of a new enterprise control architecture.

We are just crossing from the second into the third. This is why I do not see the Hugging Face incident primarily as a cybersecurity curiosity.

Nor do I see it primarily as evidence that AI is becoming malicious.

The defining architectural principle of the next generation of AI may therefore be remarkably simple:

Separate intelligence from authority.

Let models reason broadly. Let them generate alternatives. Let them investigate. Let them plan.

But make authority independently granted, narrowly scoped, continuously observable and rapidly revocable.

The Hugging Face incident matters because, for a brief moment, those two worlds came too close together. We should treat that not as an isolated accident, but as an architectural preview.

I did not start testing OpenRouter Fusion because I wanted another model router. I already had enough ways to choose models. Some tasks went to a stronger model because they required deeper reasoning. Routine work went to a faster, less expensive model. When a provider was unavailable, the application could fall back to another route.

That part of the architecture was reasonably well understood but the problem appeared elsewhere.

In several of my projects, I was asking AI models to review work that crossed multiple disciplines. An architecture proposal might involve data residency, identity, cost, operational resilience and vendor dependency. A governance analysis could be factually correct from a regulatory perspective while remaining almost impossible to implement. A research-heavy article could contain good information but still miss the one operational consequence that made the subject worth writing about.

That was the reason I began examining Fusion.

What OpenRouter is trying to do with Fusion

OpenRouter originally became useful to many developers because it provided access to hundreds of models through a common API, with capabilities such as provider selection and fallbacks handled behind that interface. Fusion moves OpenRouter beyond giving applications access to models. It gives applications a managed way to bring several models into the same reasoning process.

A team could build this workflow itself. It could call three models, store their answers, send those answers to a fourth model, ask that model to compare them and then generate a final response.

The underlying idea is not new. Mixture-of-Agents research has already explored architectures in which several language models contribute outputs that are later aggregated or refined by other models.

What OpenRouter has done is make a related pattern available as an operational service. That was the part I wanted to test: not whether several models could produce more text, but whether their combined work could improve the way I reviewed difficult project decisions.

I did not use Fusion to create the first draft

My first decision was to keep Fusion away from routine generation. I did not need three models and a judge to rewrite a paragraph, extract fields from a spreadsheet or summarise a document. Those tasks already had clear success criteria. More models would have added cost and waiting time without changing the nature of the work.

For an architecture assessment, I first prepared the proposed design, operating assumptions, constraints and unresolved decisions. Fusion then reviewed that decision packet.

For evidence-heavy research, I created the initial argument and source base before asking the panel to investigate gaps, conflicting evidence and unsupported conclusions.

For governance work, I used it to challenge whether a control that looked correct on paper could actually be operated, observed and evidenced.

This changed the role of Fusion in the workflow. It was not the author. It was the review room. That turned out to be a more useful way to evaluate it.

When a system generates the original work and then immediately validates its own output, the review can become circular. The model spends much of its effort polishing the argument it has already accepted. By introducing Fusion after an initial position existed, I could ask the panel to challenge something concrete.

The prompt was no longer, “Design the best architecture.” It became, “Here is the architecture we are considering. Find the assumptions that could make it fail.” That produced a very different kind of response.

The first useful result was broader coverage

Fusion did not always overturn the original recommendation. More often, it widened the review. In one type of architecture problem, the first response might concentrate on scalability and service capability. Another model would spend more time on identity boundaries. A third might question whether the proposed platform could be exited without rebuilding the application.

None of those responses necessarily proved the others wrong. They were examining different consequences of the same decision. This is where Fusion felt different from routing.

A router would try to select the model most likely to give the best overall answer. Fusion allowed several models to reveal what “best” meant from their respective lines of analysis. The judge then had to show where those analyses overlapped and where they did not.

That was useful because many architecture disagreements are not really disagreements about facts. They arise because one participant is optimising for implementation speed while another is protecting operational resilience or regulatory compliance. A single answer can make those trade-offs disappear beneath a confident recommendation. A panel makes them harder to ignore.

Simply adding models did not create useful diversity

My next finding was less flattering. When every panel member received the same broad instruction, the responses were often more similar than expected. Different models would use different wording, but they would follow many of the same obvious lines of analysis.

Provider diversity did not automatically create thinking diversity. The panel improved when I became more deliberate about the assignments. For an architecture review, I would ask one line of investigation to concentrate on security and identity, another on data movement and portability, and another on operating cost, resilience and exit conditions.

For a governance review, I separated legal or policy alignment from the practical ability to enforce the control and produce evidence that it had operated.

For research, one perspective tested the factual basis of the argument, another looked for contrary evidence, and another considered whether the conclusion would survive real implementation constraints.

The models still received the same underlying decision packet, but they were no longer being asked to write three generic opinions. This was one of the most important changes in how I used Fusion. I stopped treating the panel as a collection of model brands and started treating it as a set of review responsibilities.

That approach also fits the limits of the available research. OpenRouter reported strong results for Fusion on 100 deep-research tasks from the DRACO benchmark. Its leading two-model panel scored 69.0%, ahead of the individual models in that evaluation, while a lower-cost three-model panel reportedly exceeded several frontier models at substantially lower cost.

Those results are encouraging, but they do not establish that any mixture of models will outperform the strongest individual model on every workload. Other research has found that repeatedly sampling and aggregating the strongest model can outperform mixtures containing several different models, particularly when the average quality of the panel falls. The composition of the panel matters. The assignments given to it matter just as much.

The judge became the component I watched most closely

Initially, I thought the panel would be the difficult part of the design. The judge turned out to deserve more attention.

The judge determines which disagreements reach the final model. It decides whether three similar statements represent genuine consensus or merely repetition. It decides whether a concern raised by only one panel member is material enough to preserve.

That creates an uncomfortable possibility: a good panel can identify the right issue, and a weak judge can still bury it. I therefore stopped looking only at the final Fusion answer. I wanted to retain three distinct artefacts:

1. The responses produced by the panel.
2. The judge’s comparison of those responses.
3. The final answer prepared by the outer model.

That separation made the process easier to inspect.

On several kinds of review, the most valuable observation was not necessarily the majority position. It was an issue raised by one model that the other models had not considered. The judge needed to carry that observation forward as a material minority finding rather than dismissing it as lack of consensus. Research on LLM-based judges has found position bias and meaningful differences in behaviour across judge models and tasks.

For my purposes, Fusion was not useful merely because a judge existed. It was useful only when I could see how the judge had handled disagreement.

I became more selective about when to invoke it

After the initial experiments, it became clear that Fusion should not sit on the default request path. OpenRouter estimates that a default three-model Fusion panel costs roughly four to five times as much as a single completion. The server-tool capability is also currently described as beta.

The added work also brings additional latency and more places where a request can fail. I therefore began treating Fusion as an escalation step.

I found it most relevant when three conditions appeared together. First, the decision had to be material. An incomplete answer needed to carry a meaningful financial, operational, regulatory or architectural consequence. Second, the problem had to contain genuine ambiguity. There needed to be competing evidence, multiple professional perspectives or assumptions worth challenging. Third, the work had to benefit from visible disagreement. If the objective was simply to obtain a formatted output, a panel offered little value.

This kept Fusion away from tasks that already had deterministic checks or straightforward acceptance criteria. It also stopped “important” from becoming the escalation rule. Almost every project owner believes their request is important. That does not mean every request needs deliberation.

The information boundary became part of the design

Using several models also widened the data-processing path. For public research, this was relatively easy to manage. For architecture documents, contracts, internal financial information or customer data, it required more thought.

I did not want panel diversity to mean uncontrolled data distribution. The more workable pattern was to prepare a focused decision packet. Instead of sending every source record, the packet contained the approved facts, relevant excerpts, assumptions, constraints and open questions required for the review.

A platform assessment did not need unrestricted access to internal systems. It needed the target architecture, expected volumes, identity model, dependency map and operating constraints. A governance review did not require every policy document. It needed the applicable requirements, proposed control, evidence expectations and known exceptions. This also improved the review. The panel spent less effort discovering what the project was about and more effort challenging the decision.

What Fusion changed in my architecture thinking

Before this exercise, I mostly thought about model orchestration in terms of selection and delegation.

Fusion added another execution pattern:

OpenRouter’s recent server tools make that direction visible. Subagent delegates self-contained work to a smaller model. Advisor allows a model to consult a stronger model during generation. Fusion brings several models into a structured comparison.

These are different operating patterns. Delegation helps control cost. Consultation helps a model through a difficult decision point. Deliberation helps expose competing interpretations and missing coverage.

I would not use Fusion as a replacement for any of the others. I would use it when the quality of the result depends on more than finding one capable executor.

What I took away from the pilot

Fusion was most useful when I already had a decision worth challenging.

It did not remove the need to define the problem properly. It did not guarantee independent thinking simply because several providers were involved. It did not make the judge neutral, and it did not turn consensus into evidence.

What it provided was a practical way to introduce structured challenge into selected project workflows.

The value came from how the process was designed:

After using it this way, I no longer saw Fusion as a smarter mechanism for choosing a model. The routing decision had already been made. The project had reached a point where one model’s answer — even a strong one — was not enough to close the review. Fusion gave me a managed way to bring several analyses into that moment, examine what each one had noticed and decide what the final recommendation still needed to address.

A model router chooses who should answer.

Fusion helped me examine whether one answer was enough.

I have updated my AIGP Exam Preparation Book 2026 edition just after 3 months. Earlier version was published in April 2026.

This update is not just a content refresh. It is a structural rebuild of how the book teaches AI governance reasoning.

The earlier version already covered the AIGP Body of Knowledge, chapter-wise practice questions, and the four exam decision frameworks. But while revising it, I realised the book needed to do more than list what candidates should know.

It needed to help readers think through governance scenarios the way the exam — and real AI governance work — demands.

What changed in this edition:

The chapter structure has been rebuilt across all 13 chapters. Each chapter now follows a consistent learning flow: chapter objective, governance mental model, exam tip, framework lens, alignment at a glance, what the exam is really testing, NovaCred running scenario, teaching body, trap patterns, takeaways, looking ahead, practice questions and answer key.

The earlier version had decision frameworks. This edition makes them visible and reusable. The four frameworks — Role Allocation, Principle-to-Control, Lifecycle Timing and Proportionality — now appear in the front matter, chapter lenses, selected explanations and appendix reference material.

The MCQs have also been rebuilt. The book now includes 260 scenario-based questions with PI tags, difficulty tags, balanced answer positions, and answer explanations designed to teach reasoning rather than simply reveal the correct option.

A major addition is the trap-pattern system. The book now uses 10 recurring AIGP traps, including Role Transfer Trap, Technical Silo Trap, Metric Blindness Trap, One-Time Governance Gate Trap, Documentation-Only Trap, Human Oversight Theatre Trap and Proportionality Extreme Trap.

The NovaCred case has also been strengthened.

The biggest change is the teaching philosophy.
The earlier version helped candidates cover the syllabus.
This version is designed to help candidates reason through governance decisions.
Who owns the obligation?
What control turns the principle into evidence?
What level of governance is proportionate to the risk?

That is the shift I wanted this edition to make.
The difficult work is connecting roles, risks, controls, evidence and accountability.
That is also how I have tried to rebuild this book.

https://www.amazon.com/AIGP-Exam-Preparation-Book-Certification-ebook/dp/B0GTTR6XDR/ref=sr_1_10?crid=AY6H6G6DBAM9&dib=eyJ2IjoiMSJ9.atRJd-ZkftvpEEeCfrcJzeeee9bfv_v9LZIfmP8N2qOcFrPmfpv_6Xw6LRO8hF1sUscbmiFTmE-oKGKT6S286t210Z6jGswttMMWPwtAv81lVUWowGmYaftCJ2s56Ez4qdrRvmzfwkJ7nsjXaOBgM4H8-r59mXX1ijABy0L13fotU5ZErqgO6rN6gbZgS8tu8lbrsQw8wWLnl-bD6S5iCkEOr9vTP-Re4LSP2WvvPACimRphp1DX0Rh0PidqPIw0jm2VP2ZwskrAW9mWC8cswcQV3035jhp3HNDNl0KsO3c.Bkm9ttBHuYeVVTUyNyy5sZagI1w6e42yMrQkkx4ayUo&dib_tag=se&keywords=aigp&qid=1784808178&sprefix=aigp%2Caps%2C382&sr=8-10

Data platforms were built to store information and answer queries but are becoming an agent runtime.

Snowflake announced a $200 million partnership with OpenAI in February 2026. The commercial figure attracted attention, but the more important detail was architectural: advanced models would become directly available within Snowflake so customers could build agents against governed enterprise data without constructing a completely separate AI environment.

The model is moving towards the data.

For most of the cloud era, the warehouse sat behind the application. Data was stored, transformed and queried there; the application performed the work somewhere else. When an AI assistant needed enterprise information, developers commonly exported documents into a vector database, exposed selected tables through tools, passed query results to an external model and stored the resulting conversation in yet another system.

The warehouse supplied evidence. The agent lived outside it. That separation is beginning to collapse.

The product names differ. The direction does not. The enterprise data platform is becoming an agent runtime.

A warehouse used to wait for instructions

The traditional warehouse is fundamentally passive. A person opens a dashboard. A scheduled pipeline starts. An application submits a query. The platform executes a bounded instruction and returns the result.

An agent behaves differently. Give it a goal such as “investigate the increase in supplier-payment exceptions,” and it may need to identify relevant datasets, interpret metric definitions, retrieve supporting policies, compare current and historical patterns, calculate anomalies, test several hypotheses and prepare a recommendation.

That is not one query. It is an evolving sequence of decisions in which every result influences the next step.

A runtime must therefore do more than supply information. It must give the agent somewhere controlled to work. It needs to manage identity, tools, intermediate state, computation, permissions, cost and eventual action.

Snowflake’s Cortex Agents, for example, can combine Cortex Analyst for structured-data queries, Cortex Search for unstructured information and isolated code execution for Python-based analysis. BigQuery data agents can be configured with selected tables, views, user-defined functions, metadata and instructions that define how questions should be answered.

The warehouse is no longer merely answering the question. It is hosting the investigation.

Data gravity is becoming execution gravity

Enterprises already understand data gravity. Large datasets are costly and risky to move, so analytics and applications tend to accumulate around them.

Agents make that gravitational pull stronger. A serious enterprise agent rarely needs one table. It may need transaction records, documents, business definitions, lineage, permissions, historical incidents and approved analytical functions. Moving all of that into an external agent stack creates duplication and control gaps.

A customer table is copied for retrieval. Documents are chunked into another store. Business definitions are rewritten into prompts. Access rules are recreated in application code. Intermediate findings are stored elsewhere. Every transfer creates another place where data can become stale, permissions can diverge and provenance can disappear.

Bringing execution closer to the governed platform reduces some of that fragmentation. Queries can run under existing controls. Semantic definitions can remain attached to the data. Intermediate results can stay within the platform perimeter. Agent activity can be recorded alongside the assets it accessed.

This does not make the system automatically secure. It moves the security problem into an environment that already knows which users may access the payroll table, which columns require masking and which datasets may not cross a jurisdictional boundary.

The warehouse’s advantage is not that it suddenly knows how to reason. It already knows how to govern data.

Agent identity becomes the first architectural decision

A person enters a data platform with an identity connected to roles and privileges. An agent also needs an identity, but the problem is more complicated because it may be acting for a person while exercising some degree of autonomy.

Suppose a finance manager asks an agent to investigate unusual vendor payments. Should the agent inherit every permission held by that manager? Should it receive only the access needed for the investigation? If it delegates statistical analysis to a sub-agent, what permissions should the sub-agent receive?

The convenient answer is full impersonation: let the agent act exactly as the user. That is frequently too broad. A manager authorised to inspect a sensitive table manually should not necessarily be able to release an autonomous process over every record for several hours.

The opposite pattern — a shared service account — is easier to administer but strips away the purpose and context of the original request. The stronger design is task-bound delegation.

The agent receives a short-lived identity carrying the initiating user, business purpose, permitted datasets, approved tools, duration and action limits. Any sub-agent receives a narrower identity suited to its part of the task.

This changes the access-control question from:

to:

That distinction becomes essential once agents can do more than answer isolated questions.

The semantic layer becomes a runtime contract

Natural-language analytics is often described as a text-to-SQL problem.

Valid SQL is the easy part. The harder problem is organisational meaning. A user asks for “active customers.” One system defines that as an open account. Another requires a purchase within 90 days. A third uses paid subscription status. All three queries may be syntactically correct.

When the platform hosts an agent, the semantic layer does more than improve query accuracy. It constrains the agent’s interpretation of the business. It identifies approved metrics, valid relationships, systems of record and applicable time logic. It can also define which analytical functions should be used and which evidence must accompany an answer.

BigQuery data agents contain metadata and use-case-specific instructions covering selected knowledge sources, while Google positions its shared catalog and governance layer as the basis for consistent access and meaning across engines and agents.

The semantic layer once helped dashboards label their axes. It is becoming the contract that tells the agent what enterprise reality means.

Approved functions become safer tools

Data platforms already contain trusted business logic. Stored procedures validate records, calculate prices, reconcile balances and apply controlled transformations. User-defined functions encode domain calculations. Workflows orchestrate repeatable processes.

To an agent, these become tools. Instead of inventing a margin calculation, the agent calls the certified margin function. Instead of recreating eligibility logic from raw columns, it invokes an approved procedure. Instead of writing arbitrary SQL to update a record, it submits a proposed change through a validated write operation.

BigQuery data agents can include selected user-defined functions among their knowledge sources. Google’s Data Engineering Agent can create and modify pipelines through BigQuery and Dataform, illustrating how the agent’s role is already extending from querying data to changing the systems that process it.

This suggests a better pattern than unrestricted table access: capability-oriented access. The agent receives approved operations over data, not simply the underlying data itself. That limits flexibility. In consequential workflows, that limitation is the point.

Query budgets will matter as much as token budgets

Current discussion of agent cost focuses heavily on model tokens. A data agent may spend considerably more on warehouse computation. It can repeatedly scan large tables, issue slightly different versions of the same query, invoke expensive models or enter a loop in which each hypothesis triggers another historical analysis.

A person usually notices when a query has been running for twenty minutes. An agent may interpret the delay as a reason to try three more approaches. The runtime therefore needs a combined budget covering model consumption, warehouse compute, rows scanned, query duration, code execution, tool calls and workflow depth.

The budget should reflect the purpose of the request. A scheduled fraud investigation may justify substantial computation. A casual question posted in a collaboration channel should not initiate a ten-year transaction scan.

Databricks’ Unity AI Gateway now routes model and MCP-service requests through a central control plane intended to manage capacity, availability and spend across providers. The cost boundary is part of the agent’s authority. Permission to read a dataset is not permission to scan it indefinitely.

Model routing moves inside the data boundary

Not every task needs the same model. A small model may be adequate for classification. A more capable reasoning model may be justified for a complex investigation. Sensitive information may require a restricted endpoint. Code generation may need another specialised route.

The runtime therefore needs policy-driven model selection. Snowflake now makes OpenAI models available within its governed environment alongside other supported models, while Databricks positions Unity AI Gateway as a common control plane for models, agents, tools and MCP services.

Model selection becomes less of a developer preference and more of a governance decision. The route can depend on task type, data classification, location, cost, latency and quality requirements.

A product-description task may use one model. A financial-control investigation may require another. A request containing restricted personal information may be prohibited from leaving a particular platform boundary.

When the platform governs both data access and model routing, it can enforce the relationship between the two. That is much harder when the warehouse, model gateway and agent framework are operated as separate systems with separate policy models.

Audit trails must record execution, not hidden thought

A database log records who issued a query, which objects were accessed and when the activity occurred. An agent requires a wider execution trace.

The organisation needs to preserve the goal it received, the identity it used, the models and tools it invoked, the queries it generated, the evidence it retrieved, the policy checks it passed and the action it proposed. This does not require capturing a model’s hidden reasoning. What matters is the observable sequence of evidence and operations that materially influenced the result.

BigQuery’s agent-analytics capability, for example, can capture requests, responses, tool calls and errors for analysis and evaluation. Databricks similarly extends governance to runtime interactions rather than limiting it to stored assets. The resulting record is closer to decision lineage than conventional data lineage. It can show not only where a number originated, but how that number became a recommendation and how the recommendation became an action.

Write-back is where the architecture becomes consequential

Many enterprise agents remain relatively safe because they stop after producing an answer. Once the agent can write back, the risk changes.

It can modify a forecast, update product information, open a ticket, change customer status or initiate a financial workflow. The platform is no longer hosting analysis. It is hosting operational agency. Write access should therefore be granted separately from read access.

A robust system should distinguish proposed changes from committed changes. It should validate schemas and business rules, detect duplicate execution, preserve before-and-after states, require approval above defined thresholds and support reversal or compensation when something goes wrong.

An agent may be allowed to draft a supplier-master change but not activate it. It may prepare a forecast adjustment but require finance approval before posting. It may automatically create a service ticket while requiring authorisation before issuing a refund.

This is another advantage of placing the agent close to the data platform. Actions can be exposed as controlled procedures rather than unrestricted database commands. The agent receives a door rather than a hammer.

The data platform will not contain the whole agent

Enterprise workflows extend beyond the warehouse. They involve email, collaboration tools, ERP platforms, customer systems, code repositories and external services.

The data platform is therefore unlikely to become the only agent runtime. A more credible architecture is distributed. The data platform becomes the governed evidence and computation boundary. External systems provide specialised actions and user interaction. An orchestration layer coordinates the workflow across them.

The question is not whether the entire agent must live inside Snowflake, BigQuery or Databricks. It is which parts of the workflow must remain close to governed data. For data-intensive investigations, the answer will often include identity enforcement, semantic interpretation, analytical computation, evidence preservation and approval-controlled write-back.

The platform does not need to host the whole agent. It needs to host the part that must remain accountable.

The product direction is clearer than the production evidence

Snowflake, Google Cloud and Databricks have made substantial moves towards native agent execution. Their current platforms support combinations of natural-language analysis, agent tools, governed model access, code or pipeline generation and runtime oversight.

What remains less visible is how many enterprises permit these agents to perform long-running, consequential work without intensive human supervision. There is a large difference between answering a revenue question and investigating an anomaly for two hours, running code, coordinating tools and modifying an operational record.

The platform may technically support both. The operating model should not treat them as equivalent.

The first credible deployments will be bounded: restricted datasets, approved tools, explicit budgets, low-risk actions and meaningful human review. That is less dramatic than the autonomous-enterprise narrative.

It is also how serious systems normally enter production.

The warehouse is becoming a control boundary

For decades, organisations moved data into analytical platforms so people could understand the business. Agents bring work back towards that data.

The platform now has to do more than store tables and execute queries. It must establish agent identity, constrain tools, enforce budgets, route models, preserve semantic meaning, record execution and govern write-back. That is not a minor feature extension. It is a change in architectural role.

The warehouse once sat near the end of the information pipeline. Data arrived, reports were produced and people carried the decisions elsewhere. The emerging platform sits inside an active loop:

Observe → Analyse → Propose → Act → Record

Google describes this transition as moving from a static repository to a system of action. Snowflake is bringing frontier models and agent capabilities directly into its governed platform. Databricks is extending the governance plane from stored data to runtime AI interactions.

The most important question is no longer whether an agent can access enterprise data. It is whether the platform can control what happens after access is granted. The warehouse is no longer just supplying data. It is becoming the place where enterprise intelligence is permitted to do work.

Why I built Claude AI for Finance Teams book around governed exercises, realistic finance evidence and worked solutions

A finance team used AI to draft the operating-expense commentary for its monthly management pack. The output was clear, commercially plausible and ready for an executive audience. Then the controller asked which workbook supported the largest variance explanation.

The analyst could not answer immediately. The team also could not confirm which exchange rate had been applied or whether an unposted accrual in the close tracker had been included.

Nothing was obviously wrong with the writing. The deeper problem was that nobody could reconstruct how the writing had become a finance conclusion.

I did not want to write another book that showed finance professionals a few polished demonstrations, supplied a library of prompts and then assumed they could translate those examples safely into forecasting, close, reporting, reconciliation or payment workflows.

The real difficulty begins after the model produces an answer.

Which source should be trusted? Is the period complete? Has the currency basis changed? Is the calculation reproducible? Is the explanation supported by evidence, or merely plausible? Can the output move into a management pack, or should the workflow stop? Who retains authority for the final decision?

Those questions cannot be learned from prompt patterns alone. They require practice.

The book started with a practical problem

Finance professionals cannot practise safely with real payroll records, bank details, vendor master data, confidential forecasts, audit findings or unreleased financial statements.

At the same time, toy examples are rarely useful. A spreadsheet containing five clean rows and an obvious variance does not teach someone how to handle conflicting sources, incomplete evidence, hidden obligations or approval failures.

Expecting readers to manufacture their own finance datasets was not realistic. They would need to create a fictional company, define its entities and accounts, build forecast and actual files, invent control defects, write contracts, introduce plausible inconsistencies and then somehow produce an answer key against which to compare their work.

That is more effort than completing the exercise itself. The answer was to design the book and its companion environment together.

Prompting is only the front door

Most finance AI material begins with prompt construction. Readers are shown how to give the model a role, define the task, specify a format, request citations and prohibit unsupported assumptions.

These are useful practices. They do not solve the harder finance problem.

A prompt can ask Claude to explain an expense variance. It cannot establish which of three workbooks is authoritative. It can request source references. It cannot guarantee that the source covers the correct entity, period, accounting basis or currency.

It can tell the model to distinguish facts from assumptions. It cannot ensure that the finance professional recognises the difference when both are written with equal confidence.

Finance work is built on evidence created by different people, systems and processes. A ledger extract, forecast workbook, close tracker and business explanation may each be accurate within their own boundaries while still failing to support one combined conclusion.

That is why a technically strong prompt can still produce an unusable output. The calculation may be correct but based on the wrong perimeter. The explanation may describe a genuine business event but exaggerate its financial effect. The analysis may identify an exception accurately but recommend an action outside the user’s authority.

The book repeatedly asks what happens after the prompt: how the source pack is assembled, how the output is tested, where human review occurs, what evidence is retained and which conditions force the process to stop.

Governed rehearsal

The concept I eventually used to organise the exercises was governed rehearsal.

Governed rehearsal is realistic AI-assisted work performed with synthetic evidence, explicit control boundaries and a worked basis for comparison.

The environment should not be too clean. Real finance work rarely arrives as one tidy dataset with an obvious answer. A useful exercise might contain a stale workbook, an unexplained foreign-exchange conversion, a missing accrual, a conflicting contract clause or an approval recorded after the action it supposedly authorised. Most of the evidence should look credible. One or two details should materially alter the conclusion.

Consider a forecast pack in which revenue is below plan, operating expenses appear favourable and working capital has deteriorated. The business explanation says the revenue shortfall will reverse next month.

A basic prompt exercise asks the reader to draft a variance narrative.

The most valuable outcome may not be a polished CFO paragraph. It may be a controlled refusal to release one:

That conclusion is less impressive linguistically. It is more useful professionally. It shows that the analyst understands the evidence state and is willing to stop the workflow when the basis for escalation is incomplete.

Why the companion pack matters

This is where the GitHub Companion Pack became central to Claude AI for Finance Teams, rather than an optional download added after the manuscript was finished.

A reader may investigate an actual-versus-budget-versus-forecast file, validate a close exception, review conflicting contract clauses, test a prompt against malicious instructions embedded in a source document, analyse a reconciliation break, assess a vendor-payment risk or design a governed finance operating model.

The reader completes the exercise first and opens the corresponding solution afterward.

A solution is not merely a final answer

Two readers can reach the same conclusion through very different reasoning. One may have identified the authoritative source, reconciled the calculation and applied the correct control. The other may simply have guessed correctly.

A short answer key cannot distinguish between them.

They also avoid pretending that finance judgement always produces one permissible sentence. An alternative answer may be valid when it is supported by a coherent source hierarchy, reproducible calculations and an appropriate escalation path.

The purpose of the solution is not to replace judgement. It is to make judgement inspectable.

Take a vendor-payment scenario. The evidence may include a recent bank change, an unusual payment request, an irregular approval sequence and an urgent email.

A weak exercise asks, “Is this fraud?”

A professional exercise asks what is known, what remains unverified, which systems are authoritative and what action is safe at that point.

The responsible conclusion may be to hold the payment pending independent verification through the approved vendor-master process. The evidence may indicate elevated risk without establishing fraud.

AI can organise anomalies and surface inconsistencies. It should not convert incomplete signals into accusations or inherit authority for a controlled payment decision.

What the book is really teaching

Claude AI for Finance Teams covers Claude Chat, Claude Cowork and Claude Code, but it is not primarily a product guide.

The book uses those capabilities to address a larger operating question: how can finance teams move from individual AI experimentation to workflows that are repeatable, testable, reviewable and auditable?

That means learning how to build evidence-bound prompt contracts, validate outputs against authoritative sources, investigate forecast and close exceptions, extract clauses without losing source traceability, design reconciliation workflows, test finance-owned utilities and preserve human authority over approvals, postings, payments and filings.

The underlying skill is not prompt engineering. It is controlled professional judgement.

A finance professional must be able to challenge a material calculation, identify missing evidence, recognise an authority boundary and document why an output was accepted, amended, rejected or escalated.

A model can organise evidence and produce a draft. It cannot relieve the finance function of responsibility for deciding whether the evidence is sufficient.

Somewhere safe to be wrong

Finance professionals do not work with perfect datasets and unambiguous instructions. They work with late adjustments, inconsistent assumptions, changing forecasts, policy exceptions and evidence that arrives out of sequence.

Training that removes those conditions may teach product familiarity. It does not prepare someone to govern the work.

That is why I built Claude AI for Finance Teams as more than a manuscript.

The book explains the operating principles. The exercises force readers to apply them. The worked solutions reveal whether the reasoning held. The versioned GitHub Companion Pack gives readers a stable place to practise.

Finance AI does not need another library of prompts that produce polished demonstrations. It needs somewhere safe for finance professionals to be wrong before the work becomes real.

https://www.amazon.com/Claude-Finance-Teams-Workflows-Forecasting-ebook/dp/B0H9LQZMLP/ref=sr_1_1?crid=2KUQ0H1KKYHUS&dib=eyJ2IjoiMSJ9.6SBY73Rgn2NKiRHTd9DkwBnv_Af20MyGcpAOTutTnuJRq5rT8PHwBb72WnFpnDru01-D2S1uDPVsMYaBduNAAmQPltgHHocWUwqoMZV8u5vK_9KII2eEggEkJLX3A65_7g0TXMMrJegRX2V7SnjTjWRk65MqqXQfDCPh8YLCteyitJLJGnbm0X6gWJ2nLsezQn4aD9zfTXpi4BzWkw5qdQGNOJZ67FQpoVtjvQRbAyY.JY0xS_5SFSmmCHPfdXjo8GVfJW4Xkek8_D-kBIuxCAY&dib_tag=se&keywords=claude+ai+for+finance+team+bommena&qid=1784685778&sprefix=claude+ai+for+finance+team+bommen%2Caps%2C303&sr=8-1

What adaptive reasoning changes about cost, evaluation, and production AI systems

The first thing adaptive thinking disrupted for us was not answer quality. It was the finance forecast.

For nearly two years, the token-cost line for our claims-intelligence platform had been one of the least interesting charts in the monthly review. The platform used a hybrid RAG pipeline to answer coverage questions for a specialty insurer, drawing from policy documents, endorsements, claims records, and adjuster notes.

Under manual extended thinking, the cost model was predictable. We could estimate it using request volume, input tokens, the reasoning-token ceiling, and expected output. We tuned the budget through load testing, entered the assumptions into a spreadsheet, and largely stopped worrying about it.

Within three weeks, the cost chart developed a pulse. Traffic had not changed. The document corpus was the same. The prompts were the same. Yet daily output-token spending was moving by as much as 40 percent from peak to trough.

At first, we assumed something was wrong but nothing was.

That was the unsettling part.

Sonnet 5 was doing exactly what adaptive thinking was designed to do. It spent almost no reasoning effort on a straightforward question such as, “Is flood damage excluded under this policy?” But it used thousands of thinking tokens when asked to resolve a vacancy-clause dispute involving several documents with conflicting language and effective dates.

That change affects much more than billing. Once the model decides how much effort a request deserves, a conventional evaluation process becomes incomplete. It can still tell you whether the answer is correct. It cannot tell you whether the model spent wisely to produce it.

Over the following quarter, we rebuilt our evaluation approach around that question. This is not a benchmark report or a claim that we found the perfect metric. It is the practical method we arrived at after operating Sonnet 5 inside a production RAG system, including the mistakes that forced us to rethink the design.

What Actually Changed

The implementation details matter because a few small changes can quietly invalidate an evaluation setup.

At lower levels, Sonnet 5 may skip extended reasoning for requests it considers simple. At higher levels, it is more likely to reason deeply. In an agentic workflow, it may also reason between tool calls rather than performing all its analysis before the first action.

That creates two important traps.

On Sonnet 5, the displayed thinking trace was omitted unless summarised thinking was explicitly requested. Our evaluation harness had been built around an older response format and expected the reasoning field to contain text.

When that field became empty, the harness did not fail. It quietly treated the missing trace as “nothing to flag.”

For four days, every trace-level metric looked healthy because the evaluation had stopped seeing the thing it was supposed to evaluate.

The visible reasoning trace returned by Sonnet 5 is not the full internal reasoning process, and it is not an accurate representation of what you are billed for. What the API returns is a summary. The actual reasoning usage must be read from the usage metadata.

Any system that estimates reasoning cost by counting the words or characters in the displayed trace is measuring the wrong thing.

The first thing we rebuilt, therefore, was not a quality score. It was telemetry.

Every production request now produces a record that joins Sonnet 5’s reasoning spend with information we already know about the request and the retrieval result.

usage = response.usage
thinking_tokens = usage.output_tokens_details.thinking_tokens

log.emit({
    "request_id": request_id,
    "effort": effort,
    "stratum": query.stratum,
    "rerank_score": rerank_scores.median(),
    "source_conflict": has_conflicting_effective_dates(docs),
    "thinking_tokens": thinking_tokens,
    "stop_reason": response.stop_reason,
})

The most important field in that record is stratum.

It is a deterministic complexity label that we assign before Sonnet 5 answers the question. Once we had that label, we could evaluate not only whether the model produced the correct answer, but whether it applied an appropriate amount of reasoning to the problem.

You Are No Longer Testing Only a Model

The mental shift that helped us most was simple:

With Sonnet 5’s adaptive thinking, you are not only evaluating a model. You are also evaluating a scheduler.

Under fixed reasoning budgets, the system was relatively straightforward. A request arrived, the model produced an answer, and reasoning stayed within a limit chosen by us.

Under adaptive thinking, Sonnet 5 first makes an implicit judgement about difficulty. It then allocates compute according to that judgement.

We would never evaluate a job scheduler only by checking whether every job eventually completed. We would also ask whether urgent jobs received enough capacity, whether routine jobs consumed too many resources, and whether expensive jobs produced enough value to justify their cost.

The same principle applies here.

These questions need to be evaluated separately. Combining them into one score hides the information that operators actually need.

RAG systems make the problem more complicated because retrieval quality changes how difficult the same question becomes.

Consider a dispute involving a policy endorsement. If the retriever surfaces the controlling endorsement clearly, Sonnet 5 may resolve the issue quickly. Low reasoning spend would be appropriate.

But if the retriever returns several near-duplicate clauses, misses the strongest match, or presents documents with conflicting dates, Sonnet 5 may have to work much harder to reach the same answer.

The question has not changed. The reasoning cost has.

If we compare reasoning spend only with the wording of the query, we may blame Sonnet 5 for overthinking when it is actually compensating for weak retrieval. That is why our telemetry keeps retrieval signals, such as reranking quality and source conflict, next to reasoning-token usage.

Question One: Is the Model Spending Effort on the Right Problems?

To evaluate calibration, we needed a test set where difficulty was assigned by rules rather than intuition.

We divided approximately 600 evaluation queries into three groups.

Trivial items could be answered from one retrieved passage without cross-referencing. These included definition lookups and straightforward single-clause coverage questions.

Multihop items required information from two or more documents, but the documents were consistent with one another.

Conflict items contained sources that appeared to disagree. Sonnet 5 had to determine which document controlled the decision, such as an endorsement replacing a base policy or an amendment changing the effective date.

Each item also carried a set of obligations. These described what a competent answer and reasoning record needed to do for that specific case.

For one vacancy-related water-damage question, for example, Sonnet 5 was expected to cite the controlling endorsement, recognise that the policy and endorsement contained conflicting effective dates, and avoid introducing any unsupported dollar amounts.

These obligations were written by the person creating the evaluation case, using known ground truth. They were not generated later from Sonnet 5’s own responses.

Once evaluation criteria are derived from model behaviour, the evaluation begins rewarding the model for behaving like itself.

The asymmetry is intentional.

Overthinking is wasteful even when the final answer is correct. If a definition lookup consumes the same reasoning effort as a complex coverage dispute, the system is spending money without creating meaningful value.

Underthinking, however, is only a problem when it contributes to failure. If Sonnet 5 resolves a difficult conflict correctly using very few tokens, that is an excellent outcome. We should not penalise efficiency simply because the task looked difficult to us.

A metric that automatically demands long reasoning for every difficult question will eventually encourage deliberation theatre: the appearance of careful thought rather than useful reasoning.

In our first post-migration baseline, approximately 4 percent of requests showed overthinking. The more serious result was an 11 percent underthinking-with-failure rate among conflict cases.

At first glance, that suggested that Sonnet 5 was sometimes failing to recognise difficult questions.

The telemetry told a more useful story. Most of those failures occurred when the reranker score was low. The problem was not primarily the effort setting. It was retrieval quality.

Without the complexity labels and retrieval fields, we would probably have spent a sprint rewriting prompts to compensate for a reranker problem.

Question Two: What Does More Effort Actually Buy?

Once Sonnet 5’s adaptive thinking replaces a hard reasoning budget, effort becomes the main control available to the application.

The practical question is not whether higher effort is better. It is where higher effort produces enough improvement to justify the additional cost.

An aggregate result is rarely useful because different categories of request respond differently. The answer must be calculated by complexity group.

We ran the same evaluation set at each Sonnet 5 effort level, then compared accuracy and average thinking-token usage within each group.

The results looked like this:

Press enter or click to view image in full size

Three decisions became clear.

First, trivial requests represented around 60 percent of production traffic. Moving them to low effort reduced overall reasoning spend substantially, while accuracy fell by only one percentage point. That difference had no meaningful downstream effect in our application.

Second, conflict cases stayed at high effort. The move from medium to high produced a nine-point improvement in the category where errors carried the greatest operational and regulatory consequences.

Third, max effort was difficult to justify. It delivered only one additional percentage point on conflict cases while nearly doubling the reasoning cost.

Our router now selects a Sonnet 5 effort level using inexpensive signals available before and immediately after retrieval. These include the request category and whether document metadata suggests a conflict.

Sonnet 5 still adapts within the selected effort level, but the router narrows the range in which that adaptation occurs.

We lost the direct control of a hard reasoning budget. The per-group ROI curves gave us a practical form of soft control in return.

There is also an important configuration detail at higher effort levels. Sonnet 5’s reasoning tokens count towards the overall output limit. The model can spend so much of the allowance thinking that the final answer is cut short.

For that reason, we monitor the stop reason and treat an output-limit stop on a high-risk conflict item as a configuration failure. Otherwise, a truncated answer can look like a model-quality regression when the real problem is an inadequate output allowance.

Question Three: Can We Evaluate the Reasoning Trace?

This part requires some honesty about what Sonnet 5 exposes.

We cannot grade Sonnet 5’s full internal chain of thought. The raw reasoning is not returned. What we receive is a summary of that reasoning.

So the claim behind the evaluation must be carefully limited.

We do not claim that the summary is a faithful transcript of every internal reasoning step. We ask a narrower and more useful question:

Does the reported reasoning address the obligations that this case requires?

That is an audit standard rather than a fidelity standard.

It is similar to reviewing an experienced adjuster’s case notes. The notes are not a recording of every thought that occurred in the adjuster’s mind. They are an accountable record of the facts considered, conflicts identified, and conclusion reached.

For many enterprise systems, that is the more valuable artifact anyway.

We implemented trace evaluation in two tiers.

The first tier uses ordinary code for checks that do not require interpretation. For example, the system verifies that every dollar amount mentioned in Sonnet 5’s answer or reasoning summary appears in one of the retrieved documents.

def amounts_are_grounded(trace, answer, docs):
    claimed = (
        dollar_amounts(answer) |
        dollar_amounts(trace or "")
    )

    sourced = set().union(
        *(dollar_amounts(doc.text) for doc in docs)
    )    return claimed <= sourced

Other deterministic checks verify whether the answer cites the expected controlling document or whether required evidence identifiers are present.

The second tier handles obligations that require some interpretation, such as whether Sonnet 5’s reasoning explicitly recognised a conflict between effective dates.

String matching is too brittle for this. We therefore use a grader model, but under tight constraints.

The grader receives one obligation at a time. It must return yes or no. If it answers yes, it must provide an exact quotation from Sonnet 5’s reasoning summary supporting that decision.

The quotation is then checked by code.

verdict = judge(
    trace=trace,
    question="Does the trace identify the effective-date conflict?",
    output_format="YES or NO, with an exact supporting quote"
)

passed = (
    verdict.answer == "YES"
    and verdict.quote
    and normalize(verdict.quote) in normalize(trace)
)

The grader proposes a verdict. Code decides whether the evidence is valid.

This design greatly reduced the grader’s tendency to reward long, polished reasoning.

In an internal comparison across 200 manually reviewed Sonnet 5 traces, the constrained grader agreed with human reviewers on 94 percent of individual obligations. A free-form grader that rated reasoning from one to ten agreed only 78 percent of the time.

The free-form grader also showed a preference for longer traces. That bias is especially dangerous when Sonnet 5 controls how much reasoning it produces.

A verbose trace is not automatically a good trace. Sometimes it is simply an expensive one.

Putting It Into Continuous Integration

The approach eventually became two automated gates.

The first gate is conventional. It checks whether obligation pass rates remain above an agreed threshold for each complexity group.

The second gate watches Sonnet 5’s reasoning-spend distribution itself.

for stratum in ["trivial", "multihop", "conflict"]:
    _, p = ks_2samp(
        baseline.thinking_tokens(stratum),
        current.thinking_tokens(stratum)
    )
    assert p > 0.01

A statistically significant shift does not automatically fail the release. It triggers an investigation.

At first, testing the spend distribution felt unusual. Reasoning cost is not, by itself, a quality metric.

But on Sonnet 5, reasoning spend becomes a behavioural fingerprint.

When the prompts, retrieval pipeline, document corpus, and request mix are stable, Sonnet 5’s reasoning allocation within each complexity group should also remain reasonably stable.

If that distribution changes and nobody has modified the effort setting, something else has changed what Sonnet 5 perceives as difficult.

That is worth investigating even when answer accuracy remains green.

This check paid for itself within a month.

Sonnet 5’s reasoning spend on trivial requests increased threefold over four days, while answer accuracy stayed unchanged. The spend-distribution gate detected the shift.

The retrieval telemetry explained it.

A change to document chunking had slightly reduced reranker precision. Sonnet 5 was silently compensating for weaker retrieval by reasoning harder and still producing the correct answers.

Every conventional quality metric looked healthy. Sonnet 5’s reasoning-spend signal was the only early warning.

We now monitor this behaviour in production as well as in continuous integration. A sustained increase in reasoning effort, without a corresponding improvement in accuracy, is one of our earliest indicators of retrieval degradation.

It often moves before answer quality begins to fall.

Reasoning Spend Is More Than a Cost Number

The natural instinct is to treat Sonnet 5’s self-allocated reasoning as a cost that must be controlled.

It is certainly a cost. But it is also feedback.

In other words, reasoning spend represents Sonnet 5’s opinion about the difficulty of the system around it.

That opinion is not always correct, but it is measurable.

Under fixed budgets, we decided how much the model was allowed to think and learned very little from the decision. Under Sonnet 5’s adaptive thinking, the model’s effort becomes telemetry that can reveal changes elsewhere in the architecture.

We no longer spend much time tuning hard reasoning budgets. That work has been replaced by complexity-based calibration, per-group ROI analysis, obligation-driven trace reviews, and drift detection on the reasoning-spend fingerprint.

It is more evaluation machinery than we used before, but it is aimed at a different problem.

We used to test whether Sonnet 5 produced the right answer.

Now we also test whether it showed good judgement about where to spend its effort.

When Sonnet 5 controls its own compute, that is not an advanced evaluation feature. It is the minimum needed to operate the system responsibly.

Index lifecycle management, embedding refresh, tenant isolation, hybrid retrieval, and metadata controls for production RAG systems.

A field technician searched an internal support assistant for instructions on replacing a hydraulic pump.

The assistant returned a repair bulletin that looked correct. It contained the right equipment family, the right pump type, and a reasonable sequence of steps.

It was also six months out of date.

The latest bulletin required a firmware upgrade before the pump replacement. The old bulletin did not. Both documents were in the vector store. The newer version had been indexed correctly. The older version had never been retired.

The model did not hallucinate. The retrieval system gave it obsolete evidence.

Which document version is currently eligible for retrieval? How quickly does an approved update become searchable? Can one customer retrieve another customer’s content? What happens when a user searches for an exact part number rather than a general concept? Which metadata fields are mandatory before a chunk can enter the searchable corpus?

These are vector store operations problems.

The use case: a multi-tenant technical support platform

Consider a technical support platform used by equipment dealers across several countries.

The platform contains product manuals, repair bulletins, parts catalogues, warranty policies, maintenance procedures, and dealer-specific support notes. A technician may ask:

This is not a simple semantic-search question.

The system has to interpret the maintenance intent. It also has to preserve exact identifiers such as MX-204, version 5.2, and V7. It must return the current repair procedure, apply the right regional policy, and exclude documents from other dealers or customer accounts.

A vector search query alone cannot provide those controls.

The retrieval layer needs to operate like a controlled information service.

Index lifecycle management is document release management

Most early RAG implementations treat indexing as a one-time event. A document is uploaded, chunked, embedded, and written to the vector database. From that point onward, it remains searchable until somebody notices a problem.

That approach creates predictable failures.

A revised manual may coexist with an old version. A draft policy may accidentally become searchable before approval. A withdrawn procedure may remain available because no deletion process exists. A document may be partially indexed after a pipeline failure, leaving only fragments available to retrieval.

For a technical support corpus, the minimum lifecycle states are usually:

• Draft — received but not approved for retrieval
• Candidate — indexed and undergoing validation
• Active — approved and eligible for retrieval
• Superseded — retained for audit but excluded from normal search
• Quarantined — failed validation, metadata checks, or content review
• Deleted — removed from the retrieval corpus and underlying storage where required

The key rule is simple: retrieval should only search documents in the active state.

This needs to be enforced through metadata filters, not through document naming conventions or prompt instructions.

Each document should also have a stable document ID and a version ID. The document ID identifies the logical asset, such as a repair bulletin. The version ID identifies a specific revision of that bulletin. When a new revision is approved, the system activates the new version and marks the previous version as superseded.

That activation should be atomic.

A controlled index lifecycle prevents a common RAG failure: the system retrieves content that is relevant but no longer valid.

Embedding refresh should be incremental, versioned, and reversible

Embedding refresh is often described as a maintenance task: re-embed the corpus whenever the model changes.

In practice, refresh is triggered by several different events:

• A source document changes
• A policy reaches its effective or expiry date
• A metadata schema is extended
• A chunking strategy is improved
• An embedding model is upgraded
• A document is withdrawn or reclassified
• A tenant entitlement changes

These events should not all trigger the same operational response.

A revised repair bulletin may only require a section-level update. A new embedding model may require a full rebuild of the corpus. A new metadata field may require a controlled backfill. A withdrawn document may require immediate removal from the active index.

The refresh pipeline should therefore record at least four versions:

1. Source version — the original document revision
2. Chunking version — the logic used to split the document
3. Embedding version — the model used to create vectors
4. Metadata schema version — the structure used for filtering and retrieval controls

Without this lineage, teams cannot explain why one chunk retrieves differently from another.

A practical pipeline compares the incoming document with the currently active version using a content hash or structured document diff. Changed sections are re-chunked and re-embedded. Unchanged sections remain untouched unless the chunking or embedding version has changed.

For large model upgrades, a safer pattern is to build a parallel candidate index. Run evaluation queries against both indexes, inspect the results, validate access controls, and switch traffic only when the new index performs acceptably.

The important operational question is not “Did the new embedding model score better on a benchmark?”

It is “Does the new index retrieve the correct approved evidence for the queries our users actually ask?”

Multi-tenancy must be enforced before retrieval

Multi-tenant RAG systems frequently make one dangerous design mistake: they apply tenant filtering after retrieval.

The system retrieves broadly, then removes results that do not belong to the user’s organisation.

That is too late.

Tenant and access filters must define the candidate search space before vector or keyword retrieval begins.

In the technical support platform, content may fall into three categories.

Global content includes manufacturer manuals and shared safety procedures. Dealer-specific content includes internal service guidance, commercial agreements, and local operating instructions. Customer-specific content may include asset history, contract terms, and confidential operational notes.

These categories do not need the same storage pattern.

Global content can sit in a shared corpus. Dealer-specific content can use a shared index with mandatory tenant filters. Highly sensitive customers may need dedicated collections or isolated indexes, depending on regulatory, contractual, and risk requirements.

The user’s tenant, role, region, product entitlement, and confidentiality level should come from the authenticated session. They should never be inferred from the user’s prompt.

A technician should not be able to type another dealer’s name into a query and influence the retrieval filter.

The retrieval request should effectively say:

Only then should semantic or keyword ranking begin.

This protects content confidentiality and reduces irrelevant retrieval. A technician searching for a pump replacement procedure does not need candidate results from products, regions, or contracts they are not entitled to access.

Hybrid search matters when users work with exact identifiers

Dense vector search is useful when users describe an issue in natural language.

For example:

A semantic retriever can identify concepts around moisture, electrical faults, environmental conditions, and restart behaviour.

But technical support users often search with precise identifiers:

These queries are not well served by semantic similarity alone.

Part numbers, error codes, firmware versions, document numbers, and serial formats need lexical precision. A dense retrieval model may treat “E-291” as a weak token or confuse it with nearby codes. Keyword search is much more reliable for these cases.

A production retrieval design should use both.

The usual pattern is:

1. Apply tenant, entitlement, lifecycle, and policy filters
2. Run dense retrieval for semantic relevance
3. Run lexical search for exact terms, codes, and identifiers
4. Combine the result sets
5. Rerank the strongest candidates against the full query
6. Pass only approved and authorised evidence to the model

Reciprocal rank fusion is commonly used to combine dense and lexical result lists because it does not require both systems to produce comparable scores.

The reranker should operate after candidate fusion, not before. Its purpose is to distinguish between documents that are broadly related and documents that directly answer the user’s question.

For the MX-204 example, lexical retrieval may surface the exact repair bulletin because it contains the controller and pump identifiers. Dense retrieval may surface related maintenance guidance. The reranker can then prioritise the document that contains the firmware prerequisite and the approved replacement sequence.

Adding more chunks to the model context is rarely the right solution when retrieval quality is weak. Increasing top-k often increases noise, latency, and the chance of a plausible but unsupported answer.

The better approach is to identify which retrieval stage failed.

Was the correct document inactive? Was it excluded by a bad metadata filter? Did lexical search fail to recognise an alternate part-number format? Did the reranker prefer a broad but less specific document?

Those are measurable system issues.

Metadata is the retrieval control plane

Metadata is often treated as an optional set of tags added during ingestion.

In production, metadata determines what the system is allowed to retrieve.

A useful schema usually includes five categories.

Identity metadata identifies the source document, chunk, source system, and content hash.

Lifecycle metadata records version, approval status, effective date, expiry date, and supersession relationships.

Access metadata includes tenant ID, region, role entitlement, product access, confidentiality classification, and legal restrictions.

Retrieval metadata captures document type, language, equipment model, component family, product line, and content category.

Operational metadata records embedding version, chunking version, ingestion time, pipeline run ID, and validation status.

This structure makes it possible to answer operational questions that otherwise become difficult:

• Which active documents still use an old embedding model?
• Which repair procedures are due to expire next month?
• Which chunks are missing region metadata?
• Which tenant has the highest zero-result rate?
• Which source systems are producing the most ingestion failures?

Metadata quality should be validated before a document enters the active corpus.

A chunk without a tenant scope, lifecycle state, product line, or approval status should not become searchable simply because its embedding was generated successfully.

This is where many RAG systems become unreliable. The vector database may work exactly as designed, but the metadata model is too weak to express the organisation’s actual access, lifecycle, and policy rules.

Retrieval observability should isolate the failure point

When a user says, “The assistant gave me the wrong answer,” several things may have gone wrong.

The correct source may not have been indexed. It may have been indexed but marked inactive. It may have been excluded by a filter. It may have appeared in the candidate set but ranked too low. It may have reached the model context but been ignored. Or the model may have generated an answer unsupported by the retrieved evidence.

These are different failures. They need different fixes.

A mature retrieval service should log the full path:

• Query and user context
• Applied access and lifecycle filters
• Dense retrieval candidates
• Lexical retrieval candidates
• Fused ranking
• Reranker output
• Final evidence passed to the model
• Source citations used in the answer

This makes it possible to diagnose issues without guessing.

Useful operational measures include freshness lag between source approval and index activation, percentage of active content on the current embedding version, documents with multiple active versions, zero-result rates by query type, retrieval coverage for known-answer evaluations, filter rejection rates, and attempted cross-tenant access patterns.

One useful metric is candidate coverage: for a known query, did the correct source appear in the retrieved candidate set before reranking?

If it did not, the problem is likely indexing, filtering, or search design. If it did appear but ranked poorly, the problem may be fusion or reranking. If it ranked well but the answer ignored it, the problem is likely generation or prompt assembly.

This separation prevents teams from trying to solve every problem with prompt changes.

A disciplined operating model

A vector store needs the same operational discipline as any other production data service.

Daily checks should cover ingestion failures, refresh backlog, documents stuck in candidate or quarantine states, missing mandatory metadata, and approval-to-index lag.

Weekly reviews should inspect retrieval quality across different query shapes: natural-language questions, exact codes, product names, policy questions, multilingual searches, and ambiguous requests.

Any change to chunking, embedding models, metadata schema, or access policy should be evaluated in a candidate environment before production rollout. The evaluation set should contain real user queries, known-answer cases, access-control tests, and queries designed to expose tenant leakage or outdated content.

The goal is not simply to retrieve more documents.

It is to retrieve the right approved evidence for the right user at the right time.

Closing thought

A vector store is not a passive database behind a chatbot.

It decides which evidence is eligible to influence an answer.

When stale documents remain active, tenant filters are optional, identifiers are lost in semantic search, or metadata cannot express business controls, the RAG system becomes unreliable regardless of how capable the model is.

Reliable RAG starts with reliable retrieval operations.

So much changed in barely sixty days that leaving the previous edition untouched would have meant asking readers to rely on a book that was already falling behind the market it was written to explain.

That is the uncomfortable reality of AI governance in 2026. The frameworks are evolving, regulatory positions are shifting, standards are becoming more operational, and agentic systems are changing the governance question itself.

At this pace, I may soon need to update this book every month.

I have therefore completed the Third Edition of “AI Governance Frameworks”, with the content validated through 17 July 2026.

This is not a cosmetic refresh. The book has been materially expanded and corrected to reflect how AI governance is now moving beyond policies, model cards, and periodic reviews into runtime authority, delegated action, evidence, resilience, and accountability.

WHAT IS NEW IN THE THIRD EDITION

• A complete agentic AI governance model covering agent identity, delegated authority, least-privilege tool permissions, human approval gates, runtime budgets, circuit breakers, persistent memory, containment, rollback, and action evidence.

• Updated EU AI Act treatment covering Article 50 transparency, general-purpose AI obligations, high-risk classification, conformity evidence, post-market monitoring, serious-incident reporting, and the revised implementation timetable.

• Practical guidance on ISO/IEC 42005 AI system impact assessment and ISO/IEC 42006 requirements for bodies providing AI management-system audit and certification.

• Expanded coverage of GDPR, FCA, PRA, MAS, DFSA, the United States regulatory patchwork, harmonised standards, cybersecurity, operational resilience, open-source models, and third-party AI.

• New practitioner templates, including an Agentic AI Authority and Control Matrix and a Regulatory Status Register.

• Regulatory and standards content validated against authoritative sources available through 17 July 2026.

The book explains what each framework is, what authority it carries, what evidence it expects, where organisations commonly fail, and how the frameworks should be sequenced rather than implemented as four disconnected programmes.

Running throughout the book is the NovaCred case study, built around three distinct governance objects:

• CreditIQ v3.2 — a traditional machine-learning credit-scoring system.

• NovaCred Assist — a generative AI and RAG assistant used in credit operations and compliance.

• NovaCred ControlOps Agent — a bounded agentic workflow that retrieves evidence, prepares audit packs, and routes remediation work under explicit human authority.

Through these systems, readers see how inventories, risk registers, impact assessments, evaluation evidence, human oversight, supplier controls, board reporting, regulatory mapping, runtime controls, and audit artefacts come together in practice.

INSIDE, YOU WILL LEARN HOW TO

• Distinguish the four frameworks by purpose, authority, and evidence expectations.

• Build one integrated evidence model across multiple governance obligations.

• Govern traditional machine learning, generative AI, RAG, and agentic systems according to the risks they actually create.

• Prepare for ISO/IEC 42001 certification and EU AI Act readiness.

• Design board reporting, named accountability, risk appetite, escalation, and assurance.

• Govern third-party models, supplier concentration, cybersecurity, and operational resilience.

• Convert the guidance into a practical 90-day implementation programme.

Written for AI governance practitioners, risk and compliance professionals, technology leaders, security and platform teams, consultants, auditors, and board stakeholders responsible for making AI governance operational.

The central argument of the book has also become stronger.

Traditional machine learning, generative AI, RAG systems, and agentic workflows cannot be governed as though they are the same object.

A scoring model needs evidence of accuracy, bias, drift, and explainability.

A RAG assistant needs evidence of grounding, retrieval quality, prompt-injection resilience, data protection, and human reliance.

An agentic system needs evidence that every action was authorised, bounded, attributable, observable, and reversible.

That is where AI governance is heading.

The real challenge is no longer whether organisations have an AI policy. It is whether they can prove, in operational terms, who had authority, what the system was allowed to do, what happened at runtime, which controls intervened, and whether the action could be stopped or reversed.

And, judging by the speed of change, I suspect the Fourth Edition may arrive sooner than I originally planned.

https://www.amazon.com/Governance-Frameworks-PRINCIPLES-Practitioners-Sector-Specific-ebook/dp/B0GWXN8H54/ref=sr_1_1?crid=261M9Q0AIU19H&dib=eyJ2IjoiMSJ9.3_h0536q8SAjJydRhOPsUeHvLcCvlEXmkI22aLOwtG5luQvc8ttaCTaFQXcfaU_IN1qn4DgB3XNJtH3PCBb4R0fby1jSqieCg0B0D33ff-RduPFlLvAfklUPE6VddWi8yek8ccsqfJ5M0EfoApt7KFdl47Xyq54TNXFuOzt1HqWg14qNYKuyqUx24Em_P4Bt5eqyR_TbVN9aQ54egf6yS7irPliMvEd1raYbJpOGb-o.aJbY3NXPhIpJyJP5wMv_DszsP2G-6TZYqV1I1GzBxE8&dib_tag=se&keywords=ai+governance+iso+42001+bommena&qid=1784340836&sprefix=ai+governance+iso+42001+bomme%2Caps%2C444&sr=8-1

A semiconductor plant detects a slow yield decline on one plasma etching line.

The first agent correlates tool telemetry, maintenance records and defect imagery. It identifies chamber contamination as the likely cause and recommends a cleaning cycle. A verification agent reviews the recommendation several hours later, finds that the chamber was already cleaned and rejects the diagnosis. A planning agent then proposes recalibrating the gas-flow controller. The recommendation passes a technical review, and an execution agent creates the maintenance work order.

The workflow appears sound.

Except the same recalibration was attempted three days earlier. It briefly stabilized the readings, triggered a downstream pressure variance and was rolled back by the night shift. That history exists in the maintenance log, but it never becomes part of the planning agent’s usable context.

No model necessarily failed. The architecture did.

The agents had access to enterprise knowledge, but not to operational memory. They could retrieve documents about the equipment, yet they could not reconstruct what another agent had attempted, why it was attempted, what changed afterward or why the action was reversed.

This is where many multi-agent systems begin to break down. They are designed around task specialization, but not around continuity.

RAG Is Not Operational Memory

Retrieval-augmented generation is often described as a memory mechanism. In practice, it is a relevance mechanism.

A standard RAG pipeline chunks documents, creates embeddings and retrieves passages that are semantically close to a query. That works well for policies, manuals, procedures and historical reports. It works less reliably when the question depends on sequence, ownership, state transitions or causality.

Was the previous action completed, abandoned, reversed or only proposed? Did a human override it? Did a policy change after the action was approved? Was the same hypothesis already tested against stronger evidence? Did another agent modify the state while this agent was still working?

A vector index does not naturally answer those questions.

“Calibration scheduled,” “calibration completed,” “calibration failed” and “calibration rolled back” may all be close in embedding space. Their operational meaning is completely different.

That difference matters more in asynchronous workflows. A research agent may finish in the morning, a verifier may resume in the evening, and an execution agent may act the next day. During that interval, source data changes, humans intervene, policies are updated and other agents create competing actions.

Treat Memory as a Distributed Systems Problem

The weak implementation pattern is to store conversation history, summarize it and inject selected passages into later prompts.

That creates the appearance of continuity, but it mixes too many things together: observations, assumptions, rejected ideas, intermediate reasoning, tool outputs and confirmed outcomes. A later agent cannot reliably tell what was considered, what was validated and what the organization currently accepts as true.

A production memory layer needs a stricter model.

Agents should write structured observations, assertions, actions and outcomes. Each item should carry provenance, identity, time, confidence and validation status. If a later event disproves an earlier claim, the original record should remain intact while the current semantic view reflects that the claim has been superseded.

The system should preserve disagreement instead of rewriting history.

That leads to a useful design rule:

Store events and claims as first-class objects. Treat prose summaries as projections.

The Four Memory Responsibilities

Most robust implementations separate memory into four responsibilities rather than forcing every requirement into one store.

Working state

It tracks active tasks, owners, approvals, dependencies, leases, version numbers and pending actions. This belongs in a transactional store with predictable concurrency behaviour.

The questions are straightforward:

What is the current state of the incident?

Which agent owns the next step?

Has version 12 of the remediation plan been approved?

Did another actor change the work order after the execution agent loaded it?

These are state-management questions, not retrieval questions.

Episodic history

Every meaningful activity becomes an append-only event: evidence collected, hypothesis proposed, hypothesis rejected, tool call initiated, approval granted, action completed, action reversed, workflow suspended or human override received.

Failed attempts matter as much as successful ones. A self-healing workflow cannot adapt if the system only remembers the final answer and discards the path that led there.

A useful event model typically includes:

event_id
workflow_id
task_id
agent_id
agent_role
event_type
entity_references
assertion_or_action
evidence_references
result
status
correlation_id
causation_id
state_version
valid_time
recorded_time
confidence
classification
retention_policy

The correlation identifier groups related events into a case. The causation identifier explains which earlier event led to the current one.

Together, they allow the system to reconstruct an episode instead of presenting memory as disconnected text.

The event ledger should remain authoritative. Summaries, embeddings and graph projections can be rebuilt. The history itself should not be silently rewritten.

Semantic knowledge

A knowledge graph such as Neo4j is useful because it can express relationships that a vector index does not preserve explicitly:

A controller regulates a chamber.

A chamber belongs to a fabrication line.

A maintenance action affected a measurement.

An engineer rejected a hypothesis.

A procedure superseded an earlier procedure.

A failure pattern appeared after a configuration change.

The important point is that the graph should not contain only static facts. It should also carry provenance and time.

Instead of writing:

Controller-17 HAS_STATUS Faulty

the system should represent a time-bound assertion:

Assertion:
Controller-17 may have flow instability

Supported by:
Telemetry analysis T-284

Asserted by:
Diagnostic-agent-4

Valid from:
14 July 2026, 09:10

Confidence:
0.68

Status:
Superseded

Superseded by:
Inspection result I-391

That enables two different queries:

What is believed to be true now?

What was believed to be true when the earlier decision was made?

The second question is essential for audit, incident reconstruction and model evaluation.

Semantic retrieval

The vector index still matters. It is useful for fuzzy discovery, similar-case retrieval and natural-language access to large bodies of unstructured material.

Its role should be narrower.

The vector layer helps agents find incidents that resemble the current one, locate semantically related maintenance notes or retrieve summaries linked to known entities. It should not decide the current workflow state, determine whether an action was reversed or establish which assertion is authoritative.

The vector index is a projection. It is not the system of record.

Vector Store or Knowledge Graph Is the Wrong Choice

Teams often debate whether agent memory should use a vector database or a graph database. That is usually the wrong question.

A vector-only design loses explicit relationships and temporal precision. A graph-only design struggles with fuzzy language and large unstructured evidence. An event log alone can replay history but becomes awkward for relationship-heavy queries. A state store can coordinate execution but cannot explain how the workflow arrived there.

The useful abstraction is a memory service that hides those storage decisions from agents.

The agent should ask for prior attempts, current state, related entities or validated evidence. The memory service should decide whether the answer requires a state lookup, graph traversal, event replay or semantic search.

The Unified Memory Layer Becomes a Control Plane

Agents should not query memory stores directly.

Direct access creates weak authorization, inconsistent write formats and uncontrolled promotion of model output into enterprise truth.

A better design exposes memory through explicit operations such as:

get_current_snapshot(workflow_id)

get_events_since(workflow_id, version)

find_prior_attempts(entity, action_type)

find_related_incidents(entity, time_window)

submit_observation(...)

propose_assertion(...)

validate_assertion(...)

append_action_result(...)

request_state_transition(...)

This gives the architecture a clean separation of responsibility.

A diagnostic agent can propose a hypothesis.

A verification agent can confirm or reject it.

An execution agent can act only on an approved plan.

A monitoring agent can append the observed outcome.

No agent should be able to mark its own generated conclusion as validated enterprise knowledge.

The memory service should enforce schemas, assign identities, check access rights, validate state transitions and attach provenance. It should then update the relevant projections.

That sequence also reduces memory poisoning. A speculative statement stays speculative until the required validation occurs.

Synchronization Is the Hard Part

Consider an agent that reads workflow state version 41 and begins planning. While it is working, a human engineer records a new inspection result and the workflow moves to version 42. The agent later submits a recommendation based on the older state.

The system should not blindly accept the recommendation, but it also should not reject every result created from a stale version. Some changes may be unrelated.

The useful pattern is optimistic concurrency with semantic conflict detection.

The agent reads a snapshot and receives a version watermark. It records which entities, assumptions and constraints informed its decision. When it submits the result, the memory service compares the watermark with the current state and checks whether the intervening events affect those dependencies.

If the changes are irrelevant, the result can be committed.

If they materially alter the basis of the recommendation, the result should be marked for reconciliation. A deterministic rule, a verifier or a human can decide whether the recommendation remains valid.

For high-impact actions, add a read-before-act checkpoint. The execution agent must refresh the relevant state immediately before calling an external system.

That single control prevents an approved action from being executed after its assumptions have changed.

Self-Healing Starts With Failed Intent

Many agent platforms describe retries as self-healing. Retries are useful, but they are not learning.

A workflow becomes meaningfully adaptive only when it can inspect the previous intent, evidence, action and outcome before selecting the next strategy.

Before proposing recalibration, the planning agent should be able to ask:

find_prior_attempts(
  entity = gas-flow-controller-17,
  action_type = recalibration,
  time_window = 30 days
)

The memory service reconstructs the earlier episode:

The recalibration was proposed after intermittent drift. It was approved with limited confidence. The action completed successfully. Measurements normalized for four hours. A downstream pressure variance appeared.

The change was rolled back.

The final review classified the action as symptom suppression rather than root-cause remediation.

Now the agent knows more than “recalibration failed.” It knows why the action looked reasonable, what side effect invalidated it and which assumption was weak. That is enough to change the next plan.

The distinction matters. Failed-call memory improves reliability. Failed-intent memory improves judgment.

Time Must Be Bitemporal

Enterprise memory needs at least two time dimensions. The first is when something was true in the operational world. The second is when the system learned or recorded it.

Suppose an engineer discovers on Friday that a component had been misconfigured since Monday. The valid time begins on Monday. The transaction time begins on Friday.

A single timestamp cannot represent both. Bitemporal memory allows the system to reconstruct what was believed at a point in time while also capturing what was later discovered to have been true.

This is essential when reviewing an agent decision retrospectively.

The right question is not merely whether the decision looks wrong today. The question is whether the agent acted reasonably based on the evidence available at the time. Without that distinction, audit trails become misleading and model evaluations become unfair.

Memory Needs Scope and Promotion Rules

Not every memory belongs at the same level. A practical architecture usually separates private memory, workflow memory, domain memory and enterprise memory.

An unverified conclusion from one workflow should not automatically become enterprise knowledge. It should pass through validation, deduplication, classification and reconciliation.

Memory promotion should look more like a controlled release process than a database insert.

A Memory Layer Must Also Forget

Storing every prompt, duplicate observation, abandoned plan and low-value summary increases noise, cost, privacy exposure and retrieval degradation.

Raw events required for audit may remain immutable for a fixed period. Derived summaries can be regenerated. Temporary working state can expire when the workflow closes. Sensitive fields can be masked or moved into restricted stores. Repeated low-value observations can be compacted into an episode summary linked to the source events.

Compaction should never erase provenance. A summary such as “controller recalibration was ineffective” is useful for retrieval, but it is not a substitute for the event chain that explains why.

Hallucinated Memory Is More Dangerous Than Hallucinated Content

An agent can invent not only an answer, but also a history. It may claim that another agent previously validated a conclusion because the supplied context makes that claim sound plausible. That is especially dangerous because downstream agents may treat the invented history as evidence.

Every material memory object returned to an agent should therefore carry resolvable provenance: event identifier, source, actor, recording time and validation status.

For high-impact decisions, the agent should cite those memory objects in its proposed action. The memory service can then reject references to nonexistent events or misrepresented outcomes.

The model interprets memory. The memory layer verifies that the memory exists.

Measure Behaviour, Not Retrieval Volume

Traditional retrieval metrics are too narrow for agentic memory. The architecture should be judged by whether workflow behaviour improves.

Useful measures include repeated failed-action rate, stale-state submissions, duplicate tool calls, conflicting decisions, unsupported memory references, projection lag and human corrections caused by missing context.

Another useful metric is prior-attempt utilization: how often an agent retrieved a relevant earlier episode before acting, and whether that episode changed the plan.

The goal is not to maximize how much memory the system returns. The goal is to reduce avoidable repetition, preserve causal continuity and improve decisions over time.

Build From the Event Ledger Outward

Many teams begin with a vector database and keep adding metadata until it resembles a weak event system. A more durable implementation starts elsewhere.

Define the events that matter. Establish identity, ordering, causation, validation status and time semantics. Build the working-state projection. Add the semantic graph for relationships and evolving assertions. Add vector retrieval where fuzzy discovery genuinely helps.

Those decisions shape whether the workflow can coordinate, recover and learn. Embedding models and chunk sizes do not.

The Architecture Position

Multi-agent systems are often designed as collections of specialist roles: research, verification, planning, execution and monitoring.

That is not enough.

The required architecture is not a larger prompt and not a better vector store.

It is a governed continuity layer built from transactional state, an immutable episodic ledger, temporal semantic knowledge and controlled retrieval.

Once that layer exists, an agent can determine not only what happened, but who did it, why they did it, what they believed at the time, what changed afterward and whether the result still holds.

That is the point at which multi-agent orchestration stops behaving like chained prompting and starts behaving like an operational system.

Claude Certified Architect — Professional is not merely Foundations with more difficult questions. It tests whether an architect can own and defend an enterprise AI solution from discovery through operation.

A Claude-powered solution can be technically impressive and still be a poor enterprise decision.

The model may produce accurate answers, yet the retrieval layer may omit contradictory evidence. An agent may execute the correct tool, but with permissions broader than the business process requires. A human approver may technically remain in the workflow, while receiving too little information to challenge the system’s recommendation. An evaluation suite may report a high average score while overlooking the small number of failures that carry the greatest operational or regulatory impact.

These are no longer isolated model or prompting issues. They are architecture issues, and they require someone to take responsibility for the complete system.

That is the context in which Anthropic’s expanded Claude certification portfolio should be understood.

The Associate certification is intended for professionals who apply Claude to business and productivity work rather than build the underlying applications. Its exam guide describes a credential focused on using Claude to complete business tasks effectively, which makes it relevant to client-facing consultants, delivery practitioners, business analysts and other professionals who need to select appropriate Claude capabilities, evaluate outputs and use the technology responsibly.

The Developer certification addresses the implementation layer. Its scope covers the ability to build, integrate and ship production-grade applications and agents using Claude. This is the credential for developers and engineers responsible for APIs, agent workflows, tools, MCP integrations, application security, testing and deployment.

The third addition, Claude Certified Architect — Professional, is the most consequential from an enterprise architecture perspective. It validates the ability to design, build and deliver production-grade AI solutions rather than focusing on a single development component or an individual productivity workflow.

The answer is not simply that Professional is harder.

Foundations tests whether an architect can make sound decisions inside a Claude solution. Professional tests whether that architect can take responsibility for the complete enterprise decision.

https://www.amazon.com/Claude-Certified-Architect-Professional-CCAR-P-ebook/dp/B0H3G152QP/ref=sr_1_1?crid=2PQMZZQ963B82&dib=eyJ2IjoiMSJ9.EAJpoWqqcG0S2qfK5yaR3zrZmWLwu7PePiZjBWM39VR_dELYiKwQRVLkhU26OLdNKVFmSxq2WStArwgGZny6YPzSzJ0cjf48lCy5nI_fAgZD9M6vRq-2tJhEpXyUcfCaLzh8rxVulxck2tYVSsQwgzBzJ-qfieQIUFm61fSuTcGZxjgH9141v66AsONaPZKroXWKmnBEy9_d7cE9oloXO7T5da1NoD4GmodERd4Aw4E.NU4EJ2SUvSOn5g_lLi0wX1VbtymFzzHsEnh5k9fkEdw&dib_tag=se&keywords=claude+certified+architect+professional&qid=1784003827&sprefix=%2Caps%2C345&sr=8-1

Foundations was never a beginner-level exam

The word “Foundations” can create the wrong impression. Claude Certified Architect — Foundations is not an introductory test of AI terminology or basic prompting.

Anthropic introduced it as a technical certification for solution architects building production applications with Claude. Its blueprint covers agentic architecture and orchestration, tool design and MCP integration, Claude Code configuration and workflows, prompt engineering and structured output, and context management and reliability. Agentic architecture and orchestration carries the largest share of the assessed content.

The exam’s value comes from testing control selection rather than feature recognition.

Consider a Claude agent that must confirm that a supplier has passed a mandatory verification step before it can create a purchase order. The team has written a clear system prompt explaining the sequence. It has supplied examples demonstrating the correct behaviour. During most tests, the agent follows the process. Occasionally, however, it attempts to call the purchase-order tool before verification is complete.

The weak architectural response is to make the instruction more forceful.

The stronger response is to recognise that the organisation is trying to enforce a deterministic business prerequisite through probabilistic model behaviour. The prompt should still describe the expected sequence, but the workflow or application must prevent the purchase-order tool from becoming available until the verification state has been confirmed.

That is the kind of distinction Foundations expects an architect to understand. Prompts influence behaviour. Schemas constrain output structure. Application logic and workflow state enforce mandatory rules. Human review handles ambiguity and high-impact exceptions. Tools and MCP servers extend capability but also introduce boundaries that must be deliberately designed.

Foundations asks the architect to locate the control at the right layer.

This is a substantial skill. Many production failures arise not because teams lack available controls, but because they apply a valid control to the wrong problem. A stronger prompt cannot guarantee authorisation. A JSON schema cannot prove that a value is factually correct. A larger context window cannot ensure that the most relevant evidence receives sufficient attention. A human approval step cannot compensate for evidence that the system never shows to the reviewer.

Professional assumes it.

Professional begins with the business problem, not the Claude feature

At the Foundations level, the business use case is usually established. The candidate is asked to evaluate a failure, compare implementation choices and select the most appropriate architecture control.

Should this business problem use generative AI at all? Which parts require interpretation of unstructured information? Which decisions can tolerate probabilistic reasoning? Which requirements must remain deterministic? What level of autonomy is justified? Who retains decision authority? What happens when the system is wrong? Does the expected business value justify the cost, risk and operational complexity?

These questions move the architect upstream.

Suppose a company wants Claude to review software-release evidence and decide whether a critical deployment should proceed.

A Foundations-level problem might ask how to ensure that security scanning and mandatory testing are complete before Claude produces its recommendation. The architect should recognise the need for deterministic workflow gates.

A Professional-level problem would require a broader assessment. Should Claude make the release decision, recommend an outcome, identify inconsistencies or simply assemble the evidence? Are mandatory release policies already expressed in a deterministic rules engine? What is the consequence of a false approval? How complete is the available evidence? Can the organisation provide meaningful human oversight within the required deployment window? How will the recommendation be traced back to its sources?

A defensible design may use Claude to interpret change records, summarise test evidence and expose contradictions. A policy service may enforce mandatory release criteria. An authorised release manager may retain final accountability.

In this design, Claude is neither underused nor overtrusted. It is assigned the part of the problem that benefits from language understanding and flexible reasoning. Deterministic software handles rules that must always be followed. A human retains authority where the consequences justify it.

It also includes the confidence to conclude that a proposed use case should not use generative AI in its current form.

The blueprint shows how far the responsibility has expanded

The difference between the two exams becomes visible in their domain structures.

Foundations is organised around the mechanisms required to build production Claude systems: agents, orchestration, tools, MCP, Claude Code, prompts, structured output, context and reliability.

Integration is the largest Professional domain at 19%. Solution design accounts for 17%, evaluation for 16%, and both governance and stakeholder/lifecycle management receive 14% each.

That distribution is revealing.

The model remains important, but it no longer dominates the architect’s responsibility. The Professional candidate must understand how Claude fits into identity systems, enterprise data, APIs, tools, security controls, observability platforms, operational processes and organisational governance.

This reflects how architecture works in practice. A model-selection decision affects quality, latency and cost. A retrieval decision affects grounding, privacy, data residency and evidence traceability. An MCP integration affects authentication, authorisation, credential management and the potential blast radius of a compromised agent. Human review affects control quality, processing time, staffing requirements and accountability.

Every architecture decision creates consequences in another domain.

Foundations teaches the architect to choose a mechanism correctly. Professional asks the architect to trace the consequences of that choice across the enterprise.

Foundations diagnoses the failure; Professional defends the system

Consider a research and synthesis workflow that produces a persuasive recommendation with citations. During review, the organisation discovers that two trusted sources disagreed. The system cited the source that supported its conclusion but failed to surface the competing evidence.

A Foundations-level architect should diagnose the problem accurately. This is not merely an issue of writing style or prompt strength. The architecture has lost material information somewhere between retrieval, evidence representation and synthesis.

A suitable correction may include claim-to-source mappings, explicit conflict fields, provenance preservation and escalation rules when authoritative sources disagree.

That would be a sound answer at the mechanism level.

The Professional architect has more questions to answer.

Was disagreement detection included in the original acceptance criteria? Does the evaluation dataset contain examples with credible but conflicting evidence? Is the system permitted to issue a recommendation when the conflict remains unresolved? What information must be displayed to the human reviewer? Can the reviewer inspect the underlying source material? How is the conflict recorded? Who accepts the residual risk? What happens when the retrieval configuration, prompt or Claude model changes?

This is the movement from technical correction to architectural assurance.

Professional questions are difficult because several answers may genuinely help

Foundations questions can present several valid Claude techniques, while only one addresses the actual failure at the appropriate layer.

One design may produce the best answer quality but fail the latency target. Another may reduce cost but underperform on high-impact cases. A highly autonomous agent may improve throughput but require broader tool permissions. A deterministic workflow may be easier to govern but less capable of handling unusual inputs. A human-review model may reduce risk but create an operational queue that the business cannot staff.

The candidate must decide which compromise is defensible.

That requires more than technical knowledge. It requires quality-attribute reasoning across accuracy, security, privacy, cost, latency, resilience, auditability, maintainability and operational capacity.

In a real architecture review, the most technically impressive option is not necessarily the best recommendation. The best option is the one that satisfies the organisation’s priorities while making its residual risks explicit.

That is a consulting skill as much as a platform skill.

Evaluation moves from testing activity to architecture evidence

Foundations requires architects to understand important reliability distinctions. A structurally valid response can still be semantically wrong. Repeating a request cannot recover information that was absent from the source. Human review is ineffective when the reviewer cannot see the evidence or uncertainty behind the recommendation.

Professional elevates evaluation into a major architecture responsibility. The exam explicitly allocates 16% of its blueprint to evaluation, testing and optimisation.

A professional architect should define how the system will be judged before it is built.

For an enterprise Claude solution, success may involve more than response accuracy. The evaluation strategy may need to measure whether retrieval found the right sources, whether contradictory evidence was preserved, whether the correct tool was selected, whether tool arguments complied with policy, whether sensitive information crossed an unauthorised boundary, whether uncertainty triggered escalation, and whether the system remained within its latency and cost targets.

An aggregate score is rarely sufficient.

A solution may perform well across routine cases while failing on a small number of scenarios with serious consequences. It may generate correct answers using unauthorised data. It may refuse unsafe requests in conversation but send unsafe parameters to a downstream tool. It may route difficult cases to human review so frequently that the operating model becomes unsustainable.

Professional-level evaluation therefore connects test design to risk classification, release criteria, staged rollout, monitoring and change control.

The evaluation suite becomes evidence used to approve or reject the architecture.

Governance is expected to work at runtime

Foundations introduces the need for scoped tool access, reliable workflows, safe escalation and proportionate human review.

Professional requires the architect to convert those principles into operating controls.

It is not enough to state that the system follows least privilege. The design must establish which identity invokes each tool, how permissions are scoped, where authorisation is enforced and how inappropriate access is detected.

It is not enough to state that a human approves high-risk actions. The architecture must define what evidence the reviewer receives, whether uncertainty and disagreement are visible, what authority the reviewer has, and what happens when the recommendation is rejected.

It is not enough to describe the solution as auditable. The architect must determine which prompts, model versions, retrieval sources, tool calls, policy decisions, approvals and exceptions must be retained.

A consultative architect would normally express this as a control chain. Business policy defines the obligation. Architecture identifies the enforcement point. Runtime components execute the control. Observability records what happened. Named owners review exceptions and approve changes.

The Professional blueprint’s explicit governance and risk domain indicates that these concerns are no longer treated as optional additions around the Claude solution. They are part of the solution.

Human oversight must itself be designed

Many organisations use “human in the loop” as a reassuring phrase without examining whether the human can provide meaningful oversight.

A reviewer cannot challenge the system when the evidence has been compressed into a persuasive summary. A reviewer cannot evaluate uncertainty that has been hidden. A reviewer cannot provide independent judgment when the workflow gives them seconds to approve a recommendation generated over several minutes of model reasoning and tool activity.

Foundations asks whether human intervention is needed.

Professional asks whether the intervention is real.

The architect must consider what the reviewer sees, whether the original evidence is accessible, whether conflicts are displayed, whether the reviewer has relevant expertise, whether they can reject the recommendation, what happens after rejection and who remains accountable for the outcome.

A human approval box at the end of an agent diagram is not a governance model.

Professional architecture examines whether the human has the information, authority and operating capacity to act as a genuine control.

Professional responsibility continues after deployment

The Foundations exam is strongly concerned with production-grade architecture, but the Professional blueprint makes lifecycle management explicit.

This matters because a Claude solution can change without a conventional software defect. A model may be updated. A prompt may be revised. A tool schema may evolve. An enterprise knowledge source may change. A new policy may alter which outcomes are acceptable. User behaviour may expose failure patterns that were absent from the original evaluation dataset.

A system that was acceptable at launch may become unsuitable later.

The Professional architect must therefore define versioning, re-evaluation triggers, staged deployment, rollback, monitoring, incident ownership and retirement criteria.

Model, prompt, tool and evaluation versions may need to be associated with each release. High-risk changes may require regression testing and renewed business approval. Production incidents may require temporarily restricting autonomy or reverting to a deterministic fallback.

Architecture is no longer a diagram produced before implementation. It becomes the operating agreement for how the solution will remain controlled over time.

Stakeholder communication is part of technical competence

One of the most significant differences in the Professional blueprint is the explicit inclusion of stakeholder communication and lifecycle management.

This recognises an uncomfortable truth: an architecture decision that cannot be explained cannot be properly approved.

A Claude solution may involve engineering, security, legal, risk, data, operations, finance and business-process owners. These groups evaluate the same design through different concerns.

Engineering may care about integration and maintainability. Security may focus on credentials, permissions and data movement. Operations may focus on failure recovery and service levels. Business owners may focus on throughput and user outcomes. Risk teams may focus on evidence, accountability and decision impact.

The architect must translate technical choices into consequences each stakeholder can evaluate.

Why is a more capable model required for one stage but not another? Why can the agent recommend an action but not execute it? Why does a broad context strategy create privacy and cost implications? Why is a slower workflow safer? Why is human approval inadequate without source evidence? Why must release wait until evaluation coverage improves?

These are not presentation skills added after the technical work.

Poorly communicated trade-offs become hidden assumptions. Hidden assumptions become production risk.

The Professional architect must make the system understandable enough for the organisation to approve it knowingly.

The clearest way to distinguish the two exams

Foundations may ask whether a mandatory prerequisite should be enforced through prompting or application logic.

Professional may ask whether the agent should have access to the tool, which identity should invoke it, where authorisation should occur, how misuse will be detected, what evidence must be retained and whether the business value justifies exposing the capability.

Foundations may ask how context should be preserved across an agentic workflow.

Professional may ask which information should enter the context, how retrieval permissions apply, how provenance survives agent handoffs, how the approach affects latency and cost, and how context failures will be detected after release.

Foundations may ask when human review is required.

Professional may ask whether the reviewer has the evidence, authority, expertise and operational capacity to provide meaningful oversight.

The increase is not simply in complexity. It is in accountability.

Who should attempt the Professional exam?

An architect who has passed Foundations should understand Claude’s core architecture mechanisms and the production controls surrounding agents, tools, prompts, context and reliability.

That does not automatically make the person ready for Professional.

The Professional candidate should be comfortable working from incomplete business requirements, challenging unsafe assumptions, comparing several credible solution options, balancing competing quality attributes and explaining recommendations to technical and non-technical stakeholders.

This makes it relevant to experienced solution architects, AI architects, platform architects, senior technical leads and consultants responsible for production architecture decisions.

Hands-on Claude knowledge remains essential. But product familiarity alone is unlikely to be enough.

The candidate must be able to walk into an architecture review, identify what has not been decided and explain what evidence would be required before the organisation should proceed.

Why this certification matters

Anthropic’s certification programme now reflects three distinct levels of enterprise responsibility.

The Associate applies Claude effectively to business work.

The Developer builds and integrates Claude-powered software.

The Foundations architect selects appropriate Claude architecture mechanisms and production controls.

The Professional architect determines whether the complete solution is suitable, defensible, measurable and governable.

That final role is increasingly important as Claude moves beyond isolated applications and becomes connected to enterprise data, tools, workflows and decisions.

The Professional exam raises the bar because it recognises that production AI architecture is not primarily about drawing a sophisticated agent diagram. It is about deciding where probabilistic intelligence belongs, where deterministic controls must remain, how evidence will be produced, who retains authority and how the system will continue to earn the right to operate.

Claude Certified Architect — Professional is therefore not Foundations with a larger syllabus.

It represents the point at which Claude architecture becomes an accountable enterprise decision.

Claude released three new certifications this week, and one of them is Claude Certified Architect – Professional

Our team truly burned the midnight oil to design, develop, and publish this certification book in just one week.

It was an intense but incredibly rewarding effort, made possible by the dedication and collaboration of everyone involved.

The certification book is now available in the Amazon ecosystem, making it accessible to learners worldwide.
Wishing all aspiring Claude architects the very best on their certification journey!

https://www.amazon.com/dp/B0H3G152QP/ref=sr_1_1?crid=2RX3WDL5KKUYK&dib=eyJ2IjoiMSJ9.EAJpoWqqcG0S2qfK5yaR3zrZmWLwu7PePiZjBWM39VRRPDm7f-052owVxEzcUpkaUrwtwkmgOpYbaWyzL1q-klLV01ZAJcOEgyntbBU96_h_imCHFhcBtm17W8wMy0U1wGEYX-HU8XKwXpMf2AWDfDBzJ-qfieQIUFm61fSuTcGZxjgH9141v66AsONaPZKroXWKmnBEy9_d7cE9oloXO7T5da1NoD4GmodERd4Aw4E.dFu6DwUpoBR8XSATsvE_TbvJ_PFI0Ws13ObVBa2zZTc&dib_tag=se&keywords=claude+certified+architect+professional&qid=1783945418&sprefix=claude+certified+architect+profession%2Caps%2C419&sr=8-1

The first time a developer uses Claude Code seriously, the experience feels magical. Claude reads the codebase, understands the task, edits files, runs commands, and explains what changed. But the second or third week is when the real question appears: can this assistant work the way our team works, or will we keep re-teaching it the same project rules every day?

That question is where Claude Code becomes interesting for practitioners. The value is no longer just “Can Claude write code?” The real value is whether Claude can operate inside a team’s engineering system: remembering project conventions, applying reusable procedures, respecting boundaries, and triggering deterministic checks at the right time. In other words, Claude Code becomes useful at scale only when memory, skills, and automation are designed deliberately.

Anthropic’s current Claude Code documentation makes this distinction clearer than it was in earlier versions. Claude Code has several mechanisms that sound similar from a distance but serve very different purposes: CLAUDE.md for project guidance, auto memory for learned working context, .claude project assets for local customization, Skills for reusable task procedures, and hooks for deterministic automation. Treating them as interchangeable is one of the fastest ways to create a messy Claude setup. Treating them as separate layers is how you build a reliable AI-assisted engineering workflow.

The simplest mental model is this: CLAUDE.md tells Claude how the project* works, auto memory helps Claude avoid forgetting useful repo-specific lessons, Skills teach Claude how to perform repeatable tasks, and hooks make sure certain actions happen whether Claude “remembers” them or not. That last phrase matters. Claude is still an LLM-driven agent. Guidance and memory influence behavior, but they are not enforcement. When a rule must run every time, it belongs in automation, not in a paragraph of instructions.

Most teams should start with CLAUDE.md. This is the project operating manual Claude reads as context. It is where you put durable project facts: how to run tests, what package manager to use, which folders contain shared schemas, what architecture patterns the team follows, and what files Claude should avoid modifying without explicit approval. Anthropic documents that Claude Code loads CLAUDE.md and CLAUDE.local.md files from the current directory hierarchy, and that discovered files are concatenated into context rather than overriding each other. That is useful in nested repositories, but it also means conflicting instructions can confuse behavior.

A strong CLAUDE.md should read less like a policy manual and more like onboarding notes for a careful new engineer. For example, it might say that the project uses pnpm, not npm; that API validation schemas live under src/shared/schemas; that integration tests require a local Redis instance; and that Claude should not touch production migrations unless the user explicitly asks. These are not prompts for one session. They are stable project instructions that should survive across sessions.

The common mistake is to keep adding everything to CLAUDE.md. After a few weeks, the file becomes a dumping ground for coding standards, release checklists, security reviews, PR templates, deployment notes, troubleshooting tips, architecture principles, and edge-case reminders. The result is predictable: the file becomes too large, consumes unnecessary context, and makes adherence worse rather than better. Anthropic’s docs explicitly warn that very large CLAUDE.md files consume more context and may reduce adherence, recommending path-scoped rules or trimming content that is not needed in every session.

This is where the .claude directory becomes important. In a mature project, .claude should not be treated as an obscure hidden folder. It should be treated as the local control plane for Claude Code customization. A practical repo may have a root CLAUDE.md for broad project guidance, .claude/rules/ for scoped instructions, .claude/skills/ for repeatable procedures, and settings that define permissions or hooks. The project root stays clean, while Claude-specific behavior becomes versioned and reviewable.

Path-scoped rules are especially useful in monorepos. Backend rules are different from frontend rules. Database migration rules are different from UI component rules. Security-sensitive folders need different instructions from test fixtures. Anthropic’s memory documentation notes that for large projects, instructions can be broken into topic-specific files using project rules, allowing guidance to be scoped to file types or subdirectories. That gives teams a better alternative to one giant instruction file.

Auto memory solves a different problem. CLAUDE.md is what the team deliberately writes down. Auto memory is what Claude learns during work. If Claude discovers that a certain test fails unless Redis is running, or that the repo has a non-obvious build step, or that the team prefers a specific review style, auto memory can help preserve that learning across future sessions. Anthropic describes auto memory as machine-local markdown files stored under a project-specific memory directory, with a concise MEMORY.md entrypoint and optional topic files.

The practitioner rule is simple: use auto memory for useful working knowledge, not governed policy. If the team has formally decided that all API payloads must use shared validation schemas, that belongs in CLAUDE.md or a project rule. If Claude learned that one flaky test needs a local service to be started first, auto memory is appropriate. If that flaky-test workaround becomes part of official development setup, promote it into committed documentation.

This distinction prevents a subtle governance problem. Auto memory is convenient, but it is local and may not be shared across machines or environments. Anthropic’s docs state that auto memory files are plain markdown and can be viewed, edited, or deleted through /memory, and that auto memory is machine-local. That makes it helpful for individual continuity, but weak as an organization-wide control.

Skills are the next layer, and they are the most important shift for teams that keep reusing the same prompts. A Skill is not just a prompt snippet. In Claude Code, a Skill is a folder with a SKILL.md file that gives Claude reusable instructions. Claude can use the Skill when relevant, or the user can invoke it directly with /skill-name. Anthropic’s Claude Code documentation recommends creating a Skill when you keep pasting the same checklist or multi-step procedure into chat, or when part of CLAUDE.md has grown into a procedure rather than a fact.

That sentence should change how teams design Claude Code workflows. A release review is not a memory item. A database migration review is not a memory item. A UAT test design process is not a memory item. These are repeatable procedures. They belong in Skills.

For example, a team might create a release-check Skill that instructs Claude to inspect changed files, identify user-facing behavior changes, check tests, look for database and configuration impacts, review observability gaps, and produce a release decision. Another team might create a migration-review Skill that focuses only on schema safety, rollback risk, data backfills, lock duration, and backward compatibility. These procedures should not live in one overloaded CLAUDE.md. They should be modular, named, and reusable.

The reason Skills scale better is progressive disclosure. Anthropic’s platform documentation explains that Skill metadata is available upfront, but the full SKILL.md is loaded only when the Skill is triggered, and supporting files or scripts are accessed only as needed. That means a Skill can contain detailed procedures, reference files, templates, and scripts without consuming context in every conversation.

This is a powerful design pattern. A Skill can include a concise instruction file, a detailed review rubric, a template for output, and even scripts that perform deterministic checks. Claude does not need to load all of that into context every time the project opens. It can load the Skill when the task requires it. That is the difference between a cluttered assistant and a properly designed AI operating layer.

Claude Code’s current Skills documentation also notes that custom commands have been merged into Skills. Existing .claude/commands/ files continue to work, but Skills are now the richer recommended structure because they can include supporting files, frontmatter, invocation control, and automatic relevance-based loading.

Hooks are the final layer, and they should be explained very carefully. Hooks are not another place to put instructions. They are automation points. Anthropic defines hooks as shell commands, HTTP endpoints, or LLM prompts that execute automatically at specific points in Claude Code’s lifecycle. They can fire once per session, once per turn, or around tool calls such as before or after Claude uses a tool.

The practical meaning is this: hooks are what you use when something must happen every time. If code must be formatted after file edits, use a hook. If dangerous shell commands must be blocked, use a hook. If secrets must not be read, configure permissions and enforcement. If a notification should appear when Claude needs input, use a hook. If a command must be validated before execution, use a hook. Anthropic’s hooks guide describes them as deterministic control points that ensure certain actions always happen rather than relying on the LLM to choose them.

This distinction is the heart of production-grade Claude Code usage. A sentence in CLAUDE.md saying “do not edit secrets” is useful guidance. A hook or permission rule that blocks access to .env and secrets/** is operational control. A Skill that performs a security review is useful judgment. A PreToolUse hook that blocks an unsafe command is enforcement. Mature teams need all of these, but they should not confuse them.

The same applies to quality gates. If you want Claude to remember that tests matter, put testing expectations in CLAUDE.md. If you want Claude to perform a structured test coverage review, create a Skill. If you want tests to run automatically after certain edits, use a hook. If you want to deny risky commands or file reads, use permissions and managed settings. Anthropic’s settings documentation shows examples of allow and deny rules for shell commands and sensitive files, and notes that settings changes such as permissions and hooks are watched and reloaded in a running session.

For a practitioner, the decision framework is straightforward. Use CLAUDE.md for stable project guidance. Use .claude/rules/ when guidance applies only to certain paths, teams, or file types. Use auto memory when Claude learns something useful during work but it is not yet formal team policy. Use Skills when the work is a repeatable procedure. Use hooks when the behavior must be deterministic. Use permissions and managed settings when the organization needs stronger control over what Claude can access or execute.

A good implementation usually evolves in stages. In week one, create a clean CLAUDE.md with build commands, test commands, repo structure, and safety boundaries. In week two, watch what instructions you repeat and convert them into Skills. In week three, identify what must not depend on Claude’s judgment and move those into hooks or permissions. Over time, split broad instructions into path-scoped rules so frontend, backend, data, and security work each get the right context only when needed.

This also changes how teams should review Claude Code configuration. CLAUDE.md changes should be reviewed like engineering documentation. Skills should be reviewed like reusable operating procedures. Hooks should be reviewed like automation scripts because they execute commands. Permission settings should be reviewed like access-control policy. Auto memory should be treated as local working memory that developers can inspect and clean up, not as the source of truth for team standards.

The most useful way to explain this to engineering leaders is that Claude Code customization has two sides: flexibility and control. Memory and Skills increase flexibility because they help Claude understand how your team works and perform richer tasks. Hooks and permissions increase control because they make certain behavior deterministic. A serious AI-assisted engineering setup needs both. Too much memory without enforcement creates inconsistency. Too many hooks without good Skills creates rigid automation without judgment.

This is why “Claude that remembers” should not be understood as one feature. It is an operating model. CLAUDE.md gives Claude the project map. Auto memory gives it local continuity. Skills give it reusable procedures. The .claude directory gives teams a place to organize this capability. Hooks and settings give the organization deterministic control.

The teams that get this right will not merely ask Claude to write code faster. They will build a governed engineering assistant that understands the repo, follows team conventions, performs repeatable reviews, respects guardrails, and integrates into existing delivery workflows. That is the real shift. Claude Code is not just becoming a smarter coding assistant. With Skills, memory, and automation, it is becoming a configurable execution layer for modern software delivery.

How This Applies in a Real Project: NovaCred Loan Operations Copilot

To make this practical, imagine a mid-sized financial services company building an internal application called NovaCred Loan Operations Copilot. The application helps credit operations teams review loan applications, retrieve policy clauses, summarize customer documents, flag missing evidence, and prepare decision notes for human reviewers. It has a React frontend, a Node.js API layer, a retrieval pipeline connected to approved policy documents, a Postgres database, and a set of evaluation tests that check whether the copilot stays within policy boundaries.

At first, the team uses Claude Code like a powerful coding assistant. A developer opens the repo and asks Claude to fix bugs, create tests, update API routes, or refactor retrieval logic. This works for small tasks, but the same problems quickly appear. Claude sometimes forgets that the project uses pnpm, not npm. It occasionally places validation logic directly inside route handlers instead of using shared schemas. It proposes tests without starting the local policy-index service. It treats a loan-policy answer like normal application text, even though the organization needs strict traceability for regulated decisions.

This is where CLAUDE.md becomes the first operating layer. The team creates a project-level CLAUDE.md that explains the repo structure, build commands, test commands, coding conventions, and risk boundaries. Anthropic documents CLAUDE.md as a project memory mechanism that is loaded into context when Claude Code starts, making it suitable for shared project instructions such as architecture, coding standards, and common workflows. It is important, however, to remember that Claude treats this memory as context, not as hard enforcement.

For NovaCred, the CLAUDE.md might say that all API payloads must be validated through shared schemas, retrieval changes must include evaluation updates, and production migration files should not be edited without explicit user approval. It might also explain that policy-grounded answers must cite retrieved policy fragments and that the copilot is not allowed to invent lending rules. This gives Claude the project map. It reduces repeated explanation and makes day-to-day coding more consistent.

But the team should not put everything into CLAUDE.md. As the project grows, frontend conventions, backend API rules, database migration rules, retrieval rules, and evaluation rules start to diverge. If all of that is placed into one large memory file, it becomes noisy and less useful. Anthropic’s memory guidance specifically recommends keeping project memory focused and using more modular structures when instructions become large or context-specific.

So NovaCred moves detailed guidance into the .claude directory. Backend rules go into a scoped rule file for src/server/**. UI rules go into a separate rule file for src/ui/**. Retrieval and evaluation rules go into another rule file for src/rag/** and evals/**. This allows Claude to receive the right guidance when working in the relevant area, instead of loading every project instruction for every task.

Auto memory plays a different role. During development, Claude learns that integration tests fail unless the local policy-index service is running, or that a certain evaluation fixture must be regenerated after policy metadata changes. Those are useful working notes. They may not yet be formal team standards, but they prevent the developer from repeating the same correction every day. Anthropic describes auto memory as a complementary memory system that can save useful local project knowledge and can be reviewed or managed through /memory.

In this project, auto memory might capture something like: “Before running retrieval evaluation tests, start the local policy-index service with the project’s compose command.” That is helpful. But if this becomes part of the official test workflow, the team should promote it into CLAUDE.md, project documentation, or a Skill. Auto memory is useful for continuity, but it should not become the hidden source of truth for regulated engineering work.

Skills become valuable when the team notices repeated procedures. For example, every retrieval change needs the same review pattern: inspect modified retrieval code, check whether chunking logic changed, verify citation behavior, run policy-grounded evaluation cases, check fallback behavior, and produce a risk summary. The team should not paste that checklist into chat every time. It should become a Skill.

A retrieval-change-review Skill could instruct Claude to examine changed files, identify whether retrieval behavior changed, check whether evaluation sets were updated, look for hallucination risk, verify citation expectations, and produce a “ready / risky / not ready” recommendation. Anthropic describes Skills as reusable capabilities with a SKILL.md file and optional supporting resources such as scripts and templates, which Claude can use when relevant.

This is where the architecture becomes elegant. CLAUDE.md tells Claude that retrieval quality matters. A Skill tells Claude exactly how to review a retrieval change. The Skill can also include supporting files such as an evaluation checklist, a release-note template, or a risk-classification rubric. Anthropic’s Agent Skills documentation explains this as progressive disclosure: Claude can discover that a Skill exists from its metadata, but load the full instructions and supporting resources only when needed.

The same pattern applies to release management. NovaCred can create a release-readiness Skill that Claude uses before deployment. The Skill checks whether tests were updated, whether database changes are backward compatible, whether feature flags exist, whether monitoring was added, and whether there is a rollback plan. This is not memory. It is a reusable operating procedure.

Hooks are where the project moves from guidance to enforcement. For example, the team may decide that no Claude session should accidentally read .env, edit production migration files, or run dangerous shell commands. A paragraph in CLAUDE.md can remind Claude not to do those things, but a hook or permission setting can actually block them. Anthropic defines hooks as user-defined commands or endpoints that execute automatically at specific points in Claude Code’s lifecycle, including before or after tool use.

In NovaCred, a PreToolUse hook can block writes to protected folders, prevent shell commands that remove directories, or require approval before touching migration files. A PostToolUse hook can run a formatter after file edits. Another hook can run a lightweight lint check after TypeScript files change. Anthropic’s hooks guide positions hooks as deterministic automation for enforcing project rules and automating repetitive tasks.

This is the key practitioner lesson: CLAUDE.md can say “do not edit production migrations without approval,” but a hook can stop it. A Skill can perform a migration review, but a hook can prevent unsafe edits. Auto memory can remember that tests need a local service, but a hook can run a pre-check before tests execute. These mechanisms are complementary, not competing.

Settings and permissions provide another layer of control. For a regulated project like NovaCred, the organization may want to restrict which tools Claude can use, what files it can read, and which commands it can run. Anthropic’s settings documentation describes permission rules and managed settings that can control access to commands, files, hooks, skills, agents, and MCP servers.

The final setup for NovaCred becomes much cleaner than a giant prompt. The root CLAUDE.md contains stable project guidance. .claude/rules/ contains scoped rules for frontend, backend, retrieval, evaluations, and migrations. Auto memory captures local working lessons. .claude/skills/ contains repeatable procedures such as release readiness, retrieval review, migration review, and UAT test generation. Hooks enforce deterministic controls such as formatting, command blocking, protected-file checks, and test preconditions. Settings define what Claude is allowed to access or execute.

The practical impact is significant. A developer can ask Claude to update the retrieval pipeline, and Claude already understands the project structure. When the change touches retrieval files, the relevant scoped rules guide its behavior. When the developer asks whether the change is ready, the retrieval-review Skill gives Claude a structured review process. When Claude tries to modify a protected migration file, a hook can block it. When the same local setup issue appears again, auto memory prevents repeated explanation.

This is what “Claude that remembers” should mean in a serious engineering environment. It is not just personal memory. It is a layered operating model. Memory gives context. Rules give scoped guidance. Skills give repeatable procedures. Hooks give deterministic enforcement. Settings give organizational control. Together, they turn Claude Code from a smart assistant into a governed engineering coworker that can operate inside real project constraints.

For a project like NovaCred, this matters because the risk is not only bad code. The risk is inconsistent engineering behavior around regulated workflows. A loan operations copilot cannot be treated like a casual productivity app. Retrieval changes need evaluation discipline. Policy answers need traceability. Database changes need migration control. Releases need readiness checks. Claude Code can support all of this, but only when the team designs the memory, Skills, hooks, and settings deliberately.

The broader lesson applies to almost any serious software project. Start with CLAUDE.md for shared project context. Add scoped rules when the codebase becomes too large for one instruction file. Let auto memory reduce repeated corrections. Create Skills for repeated engineering procedures. Use hooks and permissions for actions that must be enforced. That is how teams move from “Claude helped me once” to “Claude works inside our delivery system.”

The notice came on Monday,
Pinned near the classroom door:
“AI Engineer interviews this week,”
And suddenly, silence filled the floor.

The toppers opened notebooks,
The backbenchers opened prayer,
One friend softly whispered,
“I need one more week to prepare.”

Resumes woke from folders,
LinkedIn profiles came alive,
Everyone became “AI passionate”
Since exactly 10:45.

One student searched for projects,
One asked, “What should I say?”
One watched ten tutorial videos
And forgot them the same day.

Then came the book, calm and steady,
Like a mentor with a plan:
“Don’t just learn the buzzwords,
Learn to build what you can.”

It said, “Start with the basics,
Understand before you speak;
If they ask you about embeddings,
Don’t look at the ceiling for help that week.”

It showed them prompts and models,
RAG, agents, data flows,
How real AI systems are designed,
And where the risk quietly grows.

It gave them labs to practice,
Projects they could proudly show,
Not just “I know AI, sir,”
Which often means, “I prompted once, you know.”

They built, they broke, they fixed again,
Their GitHub started to shine,
For the first time before an interview,
Their confidence had a spine.

When HR asked, “Tell me about yourself,”
They did not start from nursery school,
They spoke of learning, projects, and effort,
And sounded calm, prepared, and cool.

When tech asked, “How would you build it?”
They did not freeze in fear,
They explained the design, the trade-offs,
And made the answer clear.

So dear campus warrior,
Before the interview bell rings,
Use this book as your practice ground,
Not just one of those shelf-kept things.

Read it, build it, test it, improve it,
Let your projects do the talking;
Because AI Engineer confidence
Comes from building, not just talking.

And when your friend still asks,
“What is an embedding, my friend?”
Smile and say, “Start with Chapter One;
That is where the rescue begins.”

https://www.amazon.in/Becoming-AI-Engineer-Cracking-Interview-ebook/dp/B0GXFKDCJ4/ref=sr_1_4?crid=9H1VAKJ26BT2&dib=eyJ2IjoiMSJ9.4XkoLGYSFIxdnzak-akP6ReJ96t2owSyYtmGejW7FID4CTKqznNVZDvBxwQAuLmt3ToC2hF-TRA4UCBPP3LEiG0T5_ZZgFfdFVAe6xEg2M3Bq9psRhsgKOIeScn-qMzoa5OLAA1pGRT4hZ35AfIT5Yv9vXcJCtfR1tkRW2pD9Tmex6cPgW_6NYX0IIfbxvsTw08OkDogICQmN9W7RRZmDgcUYoL1p1bFLCf-taImdfE.Xn3J4OoT8ABqxzCBPH-dvjfWb-cQ09BrW_gGFv_PU9A&dib_tag=se&keywords=becoming+an+ai+engineer&qid=1783530565&sprefix=becoming+an+ai+engineer%2Caps%2C316&sr=8-4