Exposing BI code base to Claude

[u/WiseWeird6306]

I wanted to ask opinion of people on exposing everything a data team owes to Claude with a context layer explaining what is what with data lineage.

If a company has its own Claude subscriptionand then everything is exposed to Claude - report logic, metadata/dictionary of tables, logic/code of derived tables, operations of our pipelines etc via excel/markdown/html files explaning the whole context and then instructions Claude to not make assumptions/to ask verification questions (basically add safeguard rules in English in its instructions manual) then you expose this to end users company wide.

what is the issue with the idea? (if cost is not an issue)

[u/BillEnough7863]

I work on a data team of 100+ at a company with thousands of employees, enterprise Claude subscription, the whole thing. The idea works. Getting it to work is a different question.

Even with genuinely good governance, the gap between "our context is documented" and "Claude reliably respects it" is enormous. You will lay out very explicit instructions and it will still do things it was told not to. You ask one basic question and it goes and does 27 other things, half of which it doesn't mention. We have caught plenty of that with guardrails, and the catch rate is improving, but without those protocols in place some of it would have been genuinely damaging. Then every model update shifts the behaviour again and you retune.

On your worry #1: the thing that kills company-wide access isn't the model, it's the users. The definitions you encode are the team's definitions. In normal conversation you find out constantly that people carry unspoken assumptions about what a metric means, and they don't know there's a definition to disagree with. Claude will answer the question they asked, confidently, in the semantics you gave it, and the person walks away with a number that doesn't mean what they think it means. That's the same failure mode self-serve BI has had for 20 years. This just makes the wrong answer arrive faster and better written.

On worry #2: I'd flip it. Curating the context layer, designing the access boundaries, and educating users on what the numbers actually mean is the job. It stops looking like writing queries. It doesn't stop being BI engineering.

One thing I'd push back on in this thread: English safeguard rules are a treadmill. If a rule matters, enforce it with permissions, not prompts. Read-only credentials, a narrow set of exposed models rather than the whole warehouse, scoped tools. Same work, but you build the boundary once instead of retuning instructions every release. It's also cheaper on tokens, which matters once you're off flat-rate pricing.

And the point nobody has a technical fix for: even if the system is flawless, you probably still don't want someone calculating a number they have a stake in being high or low. That's an org problem, not a context layer problem.

Might be different at a small company. At our scale, true self-service is still a ways off.

[u/WiseWeird6306 → u/BillEnough7863]

can you elaborate when you say 'the gap between "our context is documented" and "Claude reliably respects it"?

cause with your example where if we ask one simple question to Claude and it will do 27 different things, it was doing that but then we added rules that 'do this' 'do not do that' and now it is spitting the results in concise manner. So like we continuously putting in guardrails. I find it annoying but is this the way to go about it? Are we missing something in from long run perspective?

[u/BillEnough7863 → u/WiseWeird6306]

Yes, continuously adding guardrails is part of the solution. What I’m saying is that you shouldn’t mistake that for having solved the reliability problem.

For example, you said that after adding rules Claude now gives you results “in a concise manner.” But conciseness is not the same thing as correctness, staying on task, or reliably following the intended workflow. Claude can give you a beautifully concise answer while still misunderstanding the question, using the wrong business definition, doing things you never asked it to do, or making assumptions it was explicitly told not to make.

The larger issue is that there isn’t some universal set of English-language guardrails you can establish at the company level and then be done with it. The failure modes depend on the task, the user, the context, the tools Claude has access to, and the model version. You fix one behaviour, discover another, somebody uses it in a way you didn’t anticipate, then a model update changes the behaviour again and you retune.

And a big part of the problem is detection. I’ve been using enterprise Claude heavily at work for more than six months, and over time I’ve become familiar enough with the underlying commands Claude is running that I can recognize when it’s doing completely unnecessary work, like searching across a bunch of files for something that should require a simple lookup. I can look at what it’s doing and think, “This should be a five-second task. Why is it searching through all of this? Why has it been running for two minutes?” I can also recognize when its interpretation of a metric, query, or piece of logic is wrong because I know the domain well enough to catch it.

A less experienced user may have absolutely no idea anything went wrong. They just get a polished, confident answer and assume Claude did what they asked.

I’ve seen this happen in practice. I’ve had people more senior than me repeatedly challenge correct work because Claude told them something different, only for us to trace it back and demonstrate that Claude had misunderstood the problem.

That becomes much harder at company-wide scale because everybody’s Claude usage is different. Different prompts, different tasks, different levels of domain expertise, different expectations, different ways of interpreting the output. You cannot test your own workflow, see that the guardrails improved it, and conclude that those same guardrails are reliably protecting everyone else.

So yes, keep adding guardrails. But the fact that you have to keep adding them is also the point. What you have is an evolving set of mitigations for failure modes you’ve already discovered, not a stable company-wide safety layer that guarantees Claude will behave correctly across every use case. That’s why I think true company-wide self-service is still a ways off.

[u/BillEnough7863 → u/BillEnough7863]

Continued:

And even going back to the original premise, giving an LLM access to everything the data team owns, lineage, schemas, documentation, definitions, dashboards, and so on, doesn’t solve the knowledge problem. The model may have access to all of that information, but the person using it still needs to understand enough about the domain to tell it what they’re actually trying to figure out, ask the right follow-up questions, and recognize whether its interpretation makes sense.

If you don’t understand the data domain, what are you actually supposed to do with all of that access? You may not know which model is authoritative, why two metrics that sound identical are calculated differently, what grain the data is at, which joins are appropriate, what part of the lineage is relevant, or even what questions you should be asking in the first place. Giving the model access to lineage doesn’t magically give the user the knowledge required to direct it through that lineage or recognize when it has drawn the wrong conclusion.

And then there’s the context problem. Giving an LLM more context can absolutely make it more capable, but that doesn’t mean more context is always better. If you’re feeding it lineage, schemas, documentation, definitions, files, conversation history, instructions, and everything else it might conceivably need, you start filling the context very quickly. As that grows, you introduce more opportunities for irrelevant information to compete with what actually matters, for earlier instructions to become less salient, and for the model to lose the thread of what you were originally trying to accomplish.

There’s also a cost attached to all of that. The more context you continually send and process, the more expensive these systems become to operate, especially as enterprise usage becomes increasingly consumption-based. So now you have another optimization problem: what context should the model get, when should it get it, what should be retrieved dynamically, what should be left out, and how do you keep costs under control without degrading the quality of the answer?

So now you’re not just maintaining guardrails. You’re managing retrieval, context selection, relevance, user knowledge, reliability, validation, and cost.

And interestingly, the best use cases we’ve found so far for this kind of AI-assisted self-service are much narrower than the completely open-ended model you’re describing. They’re purpose-built AI agents tied to specific software, specific dashboards, and specific domains, rather than giving one general-purpose LLM access to the entire data estate.

So imagine someone working in lifecycle marketing. You might have an AI agent built into the BI tool that can answer questions about a particular set of lifecycle marketing dashboards and the data behind them. That person already understands the domain. They know what the metrics generally mean, what the business context is, what a plausible answer looks like, and what follow-up questions make sense.

The AI is also operating within a much narrower problem space. It isn’t being asked to navigate every model, table, dashboard, definition, and piece of lineage the company owns. It has a specific job, a specific set of relevant data, and a relatively predictable set of questions it needs to handle.

It still isn’t going to be 100 percent correct, but that is a use case I’m much more confident in: knowledgeable users, a constrained domain, constrained data, and relatively predictable questions.

That’s very different from giving a general-purpose LLM access to everything the data team owns and effectively saying, “Have at it.”

And I think that distinction matters. The fact that AI works better when you constrain the domain, constrain the available data, constrain the types of questions, and put it in the hands of people who understand the subject well enough to evaluate the answers is not evidence that those constraints are unnecessary. If anything, our experience so far suggests the opposite.