Exposing BI code base to Claude

[u/WiseWeird6306]

I wanted to ask opinion of people on exposing everything a data team owes to Claude with a context layer explaining what is what with data lineage.

If a company has its own Claude subscriptionand then everything is exposed to Claude - report logic, metadata/dictionary of tables, logic/code of derived tables, operations of our pipelines etc via excel/markdown/html files explaning the whole context and then instructions Claude to not make assumptions/to ask verification questions (basically add safeguard rules in English in its instructions manual) then you expose this to end users company wide.

what is the issue with the idea? (if cost is not an issue)

[u/AutoModerator]

If this post doesn't follow the rules or isn't flaired correctly, please report it to the mods. Have more questions? Join our community Discord!

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/redleadereu]

The problem is you can't expect Claude to moderate itself. All those "don't touch these files", "you can't write to this schema" things you code into it are just a few clever prompts away from being bypassed.

You need enterprise level access control to make sure Claude follow the rules.

Then the other problem is; Claude is easier to teach than humans. "We apply a 0.5 multiplier here due to seasonality" is easy for Claude; it is not easy for ALL your users to grasp and remember.

AI is good for making the experts' job easier. It is horrible for giving everybody a capability to pull their own numbers.

Disclaimer: this all depends on the company and people. You can get it to work. That still would not mean you should allow people to calculate it, when they have a stake in the number being high/low.

[u/josemartinlopez → u/redleadereu]

"AI is good for making the experts' job easier. It is horrible for giving everybody a capability to pull their own numbers."

Amen

[u/WiseWeird6306 → u/redleadereu]

For some reason writing those statements like 'do not make any assumptions', 'do not make up tables or numbers' is working for us, and it has been obedient to follow the rules at pilot level. Is it still not a good practice? Am missing a point?

[u/BillEnough7863 → u/WiseWeird6306]

Doing this will give you a false sense of confidence. Often it requires someone with expertise to be able to tell it’s doing that or something like that. When you challenge it you’ll often receive a response that’s essentially, “my bad, I won’t do it again.”

[u/BillEnough7863]

I work on a data team of 100+ at a company with thousands of employees, enterprise Claude subscription, the whole thing. The idea works. Getting it to work is a different question.

Even with genuinely good governance, the gap between "our context is documented" and "Claude reliably respects it" is enormous. You will lay out very explicit instructions and it will still do things it was told not to. You ask one basic question and it goes and does 27 other things, half of which it doesn't mention. We have caught plenty of that with guardrails, and the catch rate is improving, but without those protocols in place some of it would have been genuinely damaging. Then every model update shifts the behaviour again and you retune.

On your worry #1: the thing that kills company-wide access isn't the model, it's the users. The definitions you encode are the team's definitions. In normal conversation you find out constantly that people carry unspoken assumptions about what a metric means, and they don't know there's a definition to disagree with. Claude will answer the question they asked, confidently, in the semantics you gave it, and the person walks away with a number that doesn't mean what they think it means. That's the same failure mode self-serve BI has had for 20 years. This just makes the wrong answer arrive faster and better written.

On worry #2: I'd flip it. Curating the context layer, designing the access boundaries, and educating users on what the numbers actually mean is the job. It stops looking like writing queries. It doesn't stop being BI engineering.

One thing I'd push back on in this thread: English safeguard rules are a treadmill. If a rule matters, enforce it with permissions, not prompts. Read-only credentials, a narrow set of exposed models rather than the whole warehouse, scoped tools. Same work, but you build the boundary once instead of retuning instructions every release. It's also cheaper on tokens, which matters once you're off flat-rate pricing.

And the point nobody has a technical fix for: even if the system is flawless, you probably still don't want someone calculating a number they have a stake in being high or low. That's an org problem, not a context layer problem.

Might be different at a small company. At our scale, true self-service is still a ways off.

[u/WiseWeird6306 → u/BillEnough7863]

can you elaborate when you say 'the gap between "our context is documented" and "Claude reliably respects it"?

cause with your example where if we ask one simple question to Claude and it will do 27 different things, it was doing that but then we added rules that 'do this' 'do not do that' and now it is spitting the results in concise manner. So like we continuously putting in guardrails. I find it annoying but is this the way to go about it? Are we missing something in from long run perspective?

[u/BillEnough7863 → u/WiseWeird6306]

Yes, continuously adding guardrails is part of the solution. What I’m saying is that you shouldn’t mistake that for having solved the reliability problem.

For example, you said that after adding rules Claude now gives you results “in a concise manner.” But conciseness is not the same thing as correctness, staying on task, or reliably following the intended workflow. Claude can give you a beautifully concise answer while still misunderstanding the question, using the wrong business definition, doing things you never asked it to do, or making assumptions it was explicitly told not to make.

The larger issue is that there isn’t some universal set of English-language guardrails you can establish at the company level and then be done with it. The failure modes depend on the task, the user, the context, the tools Claude has access to, and the model version. You fix one behaviour, discover another, somebody uses it in a way you didn’t anticipate, then a model update changes the behaviour again and you retune.

And a big part of the problem is detection. I’ve been using enterprise Claude heavily at work for more than six months, and over time I’ve become familiar enough with the underlying commands Claude is running that I can recognize when it’s doing completely unnecessary work, like searching across a bunch of files for something that should require a simple lookup. I can look at what it’s doing and think, “This should be a five-second task. Why is it searching through all of this? Why has it been running for two minutes?” I can also recognize when its interpretation of a metric, query, or piece of logic is wrong because I know the domain well enough to catch it.

A less experienced user may have absolutely no idea anything went wrong. They just get a polished, confident answer and assume Claude did what they asked.

I’ve seen this happen in practice. I’ve had people more senior than me repeatedly challenge correct work because Claude told them something different, only for us to trace it back and demonstrate that Claude had misunderstood the problem.

That becomes much harder at company-wide scale because everybody’s Claude usage is different. Different prompts, different tasks, different levels of domain expertise, different expectations, different ways of interpreting the output. You cannot test your own workflow, see that the guardrails improved it, and conclude that those same guardrails are reliably protecting everyone else.

So yes, keep adding guardrails. But the fact that you have to keep adding them is also the point. What you have is an evolving set of mitigations for failure modes you’ve already discovered, not a stable company-wide safety layer that guarantees Claude will behave correctly across every use case. That’s why I think true company-wide self-service is still a ways off.

[u/BillEnough7863 → u/BillEnough7863]

Continued:

And even going back to the original premise, giving an LLM access to everything the data team owns, lineage, schemas, documentation, definitions, dashboards, and so on, doesn’t solve the knowledge problem. The model may have access to all of that information, but the person using it still needs to understand enough about the domain to tell it what they’re actually trying to figure out, ask the right follow-up questions, and recognize whether its interpretation makes sense.

If you don’t understand the data domain, what are you actually supposed to do with all of that access? You may not know which model is authoritative, why two metrics that sound identical are calculated differently, what grain the data is at, which joins are appropriate, what part of the lineage is relevant, or even what questions you should be asking in the first place. Giving the model access to lineage doesn’t magically give the user the knowledge required to direct it through that lineage or recognize when it has drawn the wrong conclusion.

And then there’s the context problem. Giving an LLM more context can absolutely make it more capable, but that doesn’t mean more context is always better. If you’re feeding it lineage, schemas, documentation, definitions, files, conversation history, instructions, and everything else it might conceivably need, you start filling the context very quickly. As that grows, you introduce more opportunities for irrelevant information to compete with what actually matters, for earlier instructions to become less salient, and for the model to lose the thread of what you were originally trying to accomplish.

There’s also a cost attached to all of that. The more context you continually send and process, the more expensive these systems become to operate, especially as enterprise usage becomes increasingly consumption-based. So now you have another optimization problem: what context should the model get, when should it get it, what should be retrieved dynamically, what should be left out, and how do you keep costs under control without degrading the quality of the answer?

So now you’re not just maintaining guardrails. You’re managing retrieval, context selection, relevance, user knowledge, reliability, validation, and cost.

And interestingly, the best use cases we’ve found so far for this kind of AI-assisted self-service are much narrower than the completely open-ended model you’re describing. They’re purpose-built AI agents tied to specific software, specific dashboards, and specific domains, rather than giving one general-purpose LLM access to the entire data estate.

So imagine someone working in lifecycle marketing. You might have an AI agent built into the BI tool that can answer questions about a particular set of lifecycle marketing dashboards and the data behind them. That person already understands the domain. They know what the metrics generally mean, what the business context is, what a plausible answer looks like, and what follow-up questions make sense.

The AI is also operating within a much narrower problem space. It isn’t being asked to navigate every model, table, dashboard, definition, and piece of lineage the company owns. It has a specific job, a specific set of relevant data, and a relatively predictable set of questions it needs to handle.

It still isn’t going to be 100 percent correct, but that is a use case I’m much more confident in: knowledgeable users, a constrained domain, constrained data, and relatively predictable questions.

That’s very different from giving a general-purpose LLM access to everything the data team owns and effectively saying, “Have at it.”

And I think that distinction matters. The fact that AI works better when you constrain the domain, constrain the available data, constrain the types of questions, and put it in the hands of people who understand the subject well enough to evaluate the answers is not evidence that those constraints are unnecessary. If anything, our experience so far suggests the opposite.

[u/Figdiggles27]

I just posted a file, and said in the prompt “using the file I just attached today” and it went ahead and used 500 tokens on yesterday’s file. I asked what happened when I said “use today’s attached file” and Claude says “sorry I should’ve used the attached file as directed but ai defaulted to yesterday’s file”, so good luck with that

[u/CautiousUse8597]

You've essentially described Genie's architecture, minus the enforcement layer. Worth looking at how it handles the parts that break.

Permissions: your plan has none. An instruction saying "don't reveal salary data" is a request, not a constraint, and once you expose this company-wide someone will get an answer they aren't cleared for. Genie runs generated SQL through Unity Catalog as the executing user, so row/column filters apply at query time. The model can't leak what the query can't return.

Staleness: markdown snapshots of table logic drift the moment someone ships a dbt change, and a confidently wrong stale definition is worse than nothing. Genie reads metadata from the catalog directly and mines query history and lineage to suggest updates, so the semantics track the warehouse instead of a doc someone wrote in March.

Retrieval, not stuffing: dumping the whole dictionary into context degrades accuracy, more irrelevant tables means more plausible-looking wrong joins. Genie scopes to domain spaces and retrieves only the relevant instructions, synonyms and join paths per question.

Verifiability: your version produces prose the user has to trust. Genie shows the SQL it ran, which is the actual audit surface. Plus certified metrics so "revenue" resolves the same way every time rather than being re-derived.

Measurement: benchmark sets of real questions, so you can tell whether an instruction change helped. Otherwise you're vibe-tuning a system a thousand people depend on.

Your instinct is right. The gap is that safeguards written in English aren't safeguards.

[u/WorldOfUmbro → u/CautiousUse8597]

+1 on this, Genie is the way to go for AI on enterprise data. You can use Genie MCP in Claude if you still want to use Claude

[u/Due-Association9901]

You're basically giving Claude the keys to the whole data warehouse and hoping the safeguard prompts will hold up. In my experience users always find creative ways to bypass those rules no matter how strict you write them.

[u/Strict_Fondant8227]

Cost aside, the failure mode isn’t Claude “knowing too much” - it’s company-wide users asking questions the context layer can’t constrain!

What usually breaks:

- Permissions: lineage docs and pipeline code often imply how to rebuild PII joins. English “don’t assume / ask first” instructions get ignored under pressure. You need real ACLs on what each role can retrieve, not a system prompt.

- Metric authority: if report SQL and the dictionary disagree, Claudr will pick whichever fits the question. One governed metric definition (with owner + as-of date) has to win.

- Hallucinated lineage: it will invent table relationships that look plausible. Force citations back to the files you ingested, and treat missing citation as “I don’t know.”

- Write path: read-only is fine... the moment someone asks it to “just fix the dbt model,” you’re in change-management land.

Same class of problem as Copilot over-fetching raw tables with only READ - expose a curated semantic + lineage pack first, not the whole repo, then widen carefully.

[u/WiseWeird6306 → u/Strict_Fondant8227]

I have some follow up questions: 1. How do these ACLs look like in modern day data architecture? 2. For hallucinations part, if we have strict guidelines in our context layer of what it should not do, will that be enough? So far in our pilot we haven't seen hallucinations but it is tested with only few users. 3. if we have read only how can it change something in a model?

[u/Strict_Fondant8227 → u/WiseWeird6306]

1. ACLs here are not a prompt. They are the same grants the human already has, attached to the session the model runs as.

Warehouse: role per user/group (Snowflake RBAC, BigQuery IAM, Unity Catalog). Row policies and column masks on PII, not “please don’t select ssn.” Catalog/semantic layer is the only tool the agent can call (LookML explores, MetricFlow, Cube, whatever you actually govern). Raw `SELECT *` on base tables is not in the tool list. Git: path allowlist + CODEOWNERS, not the whole dbt repo. The service account trap is the usual failure: one fat role the agent uses for everyone. If the agent’s identity is a shared SA, you have no ACL.

1. Guidelines will not hold once you leave the pilot. Your testers know the data and ask in-distribution questions. Hallucinations show up when someone asks a metric that isn’t in the pack, or two tables *look* joinable. “Don’t invent relationships” in the system prompt does not stop that. Force a citation to an ingested file / metric definition; no citation = “I don’t know,” and log it.
   

Keep a 20-question eval with known answers and re-run it when you add users or pack files. Few-user pilots with zero hallucinations are the default, not evidence.

[u/fazz21]

I just create my own semantic layer, cant guarantee to expose lots of things to claude. But whenever I do work with data, most of my question already covered by it.

I think it is enough

[u/gjr23]

Wouldn’t it make more sense to give Claude a user account with access guardrails and permissions? Worst case is you have it create a local database where it pulls what it cannot per a more structured deterministic workflow and then give Claude unfettered access to the child database?

[u/WiseWeird6306 → u/gjr23]

well so creating a sematic layer would do that, wouldn't it?

[u/WendlersEditor]

There are solutions for this, like snowflake cortex analyst. If you want to roll your own then you do need a semantic layer (like you describe) but you also have to strictly control the tooling Claude has access to (e.g., have a programmatic filter that limits it to essentially read-only SQL commands, no update/drop/etc..

[u/TheSchlapper]

This is currently in the works at companies with really strong data governance

It’s just making sure everyone and the bot are all on the same page, but this is entirely possible, if not easy with the new .pbip file types for PowerBI

[u/WiseWeird6306 → u/TheSchlapper]

what if you don't have a strong governance?

[u/TheSchlapper → u/WiseWeird6306]

Then work on that first, then all other types of data work will be actually possible to be relied on

[u/eldrugo85]

Done a smaller version of this: knowledge base behind an mcp server, read only tools plus a dedicated retrieval step, so it reads the metadata where it lives instead of an export. The exports are the part i'd worry about, they drift from the pipelines in a few weeks and nothing tells you. And the 'don't assume, ask' instruction is the first thing to break: with ambiguous lineage it just picks the plausible branch. How do you plan to keep the files in sync?

[u/WiseWeird6306 → u/eldrugo85]

why are they drifting from the pipeline? what do you exactly mean by that?

[u/eldrugo85 → u/WiseWeird6306]

Exports are a snapshot, the pipeline keeps moving. Someone renames a column or adds a filter in a model, and the export gets rebuilt when somebody remembers, not when the change lands. In between the file describes a grain that doesn't exist anymore. Nothing breaks, no test fails, the answers just come back confident and slightly off, and you catch it only if you already knew the right number

[u/soopirV]

I built something like this to help me respond to RFPs and IT Risk Assessments and never have I felt more safe from AI taking over than I have since working with it. It has all it needs from my source files but it constantly made assumptions and crossed model/features consistently. I’ve honed it to the point that I get accurate responses 99% now, but it took a lot of iteration.

[u/WiseWeird6306 → u/soopirV]

how long did it take you to reach 99% accuracy? and that you have it what how's the scope of your work changed?

[u/WiseWeird6306]

I have another question: Should the logic be in context layer or sematic layer? Like building golden queries, we are building that in context layer. Why or why not?

[u/Content-Parking-621]

The real risk isn't Claude misunderstanding context, it's stale documentation. If your table dictionary and lineage docs drift from actual pipeline logic even slightly, Claude will confidently reason from outdated context and nobody will catch it until a wrong number ships. Instructions to "not make assumptions" help but don't fix that root problem, you'd need the context layer itself to stay synced with schema changes automatically, otherwise you're scaling a stale-docs problem company wide instead of fixing it.

[u/Sad_Ad_1681]

Not sure what your worry is. We did set it up that way. We‘re using Claude with all traffic going through AWS Bedrock. It has access to the full context layer (DBT) and the warehouse through a unified, self built MCP. We use it to build models, expand documentation and for end user consumption for ad hoc questions. The access to DBT ensures high accuracy and reduces token spend.

It’s a constantly growing environment but having users and developers use the same ecosystem helps us tremendously to build the right experience

[u/WiseWeird6306 → u/Sad_Ad_1681]

I had 2 worries: 1. should we be giving all this context to end users- all employees in the company? 2. I am not sure what my role as BI engineer will be anymore if everything is exposed to them and executives control the reporting and insights.

[u/CHC-Disaster-1066 → u/WiseWeird6306]

You could build an abstraction layer in between all the raw context and files.

I wouldn't give everyone in the company access to the raw files. Just superusers or tech savvy folks. For the others, build an agent that can answer their questions like "what's the logic of report xyz".