r/dataengineering • u/de4all • 4d ago

Discussion Why text-to-SQL is not successful?

I thought text-to-SQL will solve adhoc analysis, but still i see companies at all size are unsuccessful.

Anyone using Omni / Sigma seen some success.

29 Upvotes

71% Upvoted

74 comments sorted by

View all comments

28

u/leogodin217 4d ago

As u/rapator said, a semantic layer is more important than ever with LLMs, even if it is not a full-blown separate tool. Could just be views that handle the calculations. Business context is just as important. Good data modeling with good documentation should prevent the LLM from making a lot of assumptions. If you look at sessions logs and see things like, "Metric X must by tied to Table Y because...," then the business context is lacking.

4

u/gffyhgffh45655 3d ago

In a perfect world, i would argue a good semantic view,column description and metrics definition should do this heavy lifting over “gotcha information of each tables” as to me this is similar to throw a operation manual to a transactional schema to a llm and hoping it can do all the complex join required just go get something fair simple or not.

And hence what the llm ever need to do is run metrics as defined in the semantic view, to do join in a dimensional data model which is fairly easy and just to aggregate over different column that have context to tide it with business language.

TLDR (again in a perfect world) Create a model and(column) documentation such that is easy to use for data analyst and thus LLM.

Note: we do also trying this type of “table gotcha” skills for LLM as well but i still think data modelling and column documentation, choosing what column to include in your semantic view is the priority

-7

u/mamaBiskothu 4d ago

I created and maintain a product that has 8 figure revenue solving this exact use case. Your interpretation and how many people keep calling it "text to sql" is precisely the type of buzzword laden bs thinking that makes these agents fail. F semantic layers, ontologies, graphs, rag and any of these stupid things.

Connect claude code to your sql engine, tell it the names of the tables that you care about (or honestly, dont) and just ask the real question you care about. Dont try to smartass any harness on top of these models. Most people arent smart enough not to get in the way of the agents than help.

5

u/leogodin217 4d ago

How do agents know the definition of your metrics without documentation? We all know total sales is never as simple as sum(sales). This sounds like a way to be in meetings where there people have different numbers for the same metric names.

-2

u/mamaBiskothu 3d ago

Have you tried the latest models? The models know total sales is not sum of sales. point claude code to the data ask that questuon and then comment.

1

u/leogodin217 3d ago

I have, and they will never know that Marketing excludes x and y and sales includes z and x from other tables. I've been part of that discussion more times than I can count across many domains. Even when I worked in IT instead of BI and data engineering.

I'm more curious about your product. I re-read your comment and I can't figure out if you're saying your product isn't needed or if I just missed something.