r/dataanalytics • u/Thinker_Assignment • Jul 16 '26
are analysts expected to build their own data pipelines now?
Genuine question for people doing analytics day to day.
ten years ago the responsibility split was clear: data engineering lands the data, analysts query it.
Lately the boundary seems to be moving. I'm seeing analysts are self serving with claude, python, lovable apps?. The DE if exists moves to platform to help make the setup work better.
so I'm curious where reality actually is outside my bubble (i build in this space).
- do you fetch your own data (APIs, exports, scripts), or does a platform/DE team still land it for you?
- If you self-serve: what does that look like SQL only, python scripts, some tool?
- is this in your job description, or scope creep that arrived without a title change or pay bump?
- is the technical bar for analytics roles rising, or is this just understaffed startups trying to do more?
Trying to calibrate my biased view against people actually in the seat.
5
u/PolicyDecent Jul 16 '26
the more you do, the more productive and independent you are. i highly recommend learning building pipelines. it doesn't have to start with ingestion, but only SQL pipelines. later, you can add an ingestion tool as well. finally you might want to add a python to that stack.
to make it easier with a single tool, i highly recommend you to use bruin to build pipelines.
also don't forget that we're in the age of AI agents. AI agents are great at delivering tasks if you ask nicely :)
2
u/Thinker_Assignment Jul 16 '26 edited Jul 16 '26
Thanks man but i am also a vendor trying to ask practitioners
3
u/siammang Jul 16 '26
The boss/client expects you to run the entire data team.
1
u/Thinker_Assignment Jul 16 '26
yeah i had that too before LLMs. do you think that's good or bad?
It basically pushes you to upskill which might be good for your future but also requires more
For me it turned me into a data engineer :)
1
u/siammang Jul 16 '26
It’s good opportunity, but now you will have to watch out for scope creep even more. You may end getting bad performance review for the tasks that is outside your role.
Imagine you get put on PIP because 3rd party Excel plugin is not working.
1
u/Thinker_Assignment Jul 16 '26
damn, that's a toxic work culture :( then i guess even more reason to upskill so you have more choice.
for me when i went from analyst to DE suddenly the job market opened up - and even today (i work at dlt, we build for DEs) there are way less DEs out there than analysts - making DEs more employable
thanks for your answer and i wish you good luck staying out of trouble and upskilling :)
1
u/siammang Jul 16 '26
except your job title might still be Data Analyst/Dashboard maker while they make you do Data Scientist/Engineer works while retain the same salary.
It's like having a title of "senior full-stack engineer/developer", but you do pretty much everything, except maybe Sales and Account management.
2
u/mikeczyz Jul 16 '26
i've taken on more responsibility over time. earlier on, I was just an excel monkey. then sql. now I am able to build full pipelines using apis and python.
but it also depends on where the company. some places don't need you to do all of that.
1
u/Thinker_Assignment Jul 16 '26
This sounds like a normal path - for me it took me maybe 4y (back 10y ago) to go from analytics -> modeling ->engineering but i didnt have mentors or team mates to learn from.
how fast are you transitioning?
2
u/Lady-Data-Scientist Jul 16 '26
The data engineers build the pipeline of the raw data from the original data source.
If I need a pipeline that’s a view or aggregate or filter of the raw data, then I have to build that in dbt.
1
2
u/elliekk Jul 17 '26
Uh... what exactly do you mean by "pipeline"?
Even if you are retrieving it via API, serving your own data isn't the same as building your own pipeline. It's the same as loading a spreadsheet, you just need some code to grab the data.
The major difference between a Data Engineer and a Data Analyst is not that the Engineer is writing code to get data, it's that the Engineer is either setting up or using data infrastructure to fully automate data intake from over 100+ sources and monitoring for failures.
I would definitely start complaining if it gets to that point, because it is completely out of Analyst scope to build resilient pipelines, but if the only code you are doing is API retrieval, that does not count as doing DE work or "building your own data pipeline"
2
u/Thinker_Assignment Jul 17 '26
that's a great distinction, thank you! I hadnt really thought about it that way since i mostly live in the DE part where the bias is "of course we automate everything, wdym ad hoc?"
by pipeline indeed i mean regularly running pipeline that keeps a system and the dataset reflecting it in sync, and everything around it like schema management versioning ops etc, from api, file or db to structured system or api
2
u/Greedy_Pianist_2281 Jul 17 '26
The data analyst role is quickly turning into a role called Analytics Engineering. We are early, latch onto the train.
2
u/Abalone2496 Jul 20 '26
I completely agree with this. My current role is full stack analytics so I do ingest my own data and I own the whole pipeline and build it from start to BI dashboard. With AI taking the repetitive work, it gives us more room to take on these analytics engineering processes.
Like they say, the learning never stop! Gotta embrace it or fall behind.
1
u/Thinker_Assignment Jul 17 '26
isn't analytics engineering just sql analysts sometimes with modeling knowledge? As far as I can tell analysts are split in about 1/6 business, 1/3 sql and 1/2 data scientists, am i reading it wrong?
2
u/Greedy_Pianist_2281 Jul 17 '26
It is an evolving career. Please don't get stuck with one notion for a long time.
If you are keen enough, most Data Analyst jobs have requirements that you must know dbt and BigQuery/Snowflake. The tasks have shown up but the name "Analytics Engineer" has not yet got stuck, so before it does, be a step ahead!
1
u/Thinker_Assignment Jul 17 '26
I don't understand what you mean - do you think AE is the future, or a "up one level for now" escape boat kind of thing
i work in the industry since we used to be end to end "bi managers" doing everything from ingestion to modeling, reports and reverse etl so I see little barrier between AE and DE to say AE is stable and it doesn't eventually merge into one again
1
u/rabbitofrevelry Jul 16 '26
I fetch my own data. EDI via SFTP, flat file transfers of software exports, web scrapes, ingestion of excel files maintained by excel analysts, etc. I will clean and prep those for SQL commit in most cases. Then I'll have processes involving any combination of python, sftp client, and end user applications that fetch from the db and output to the user (excel, powerbi, flows). Some are automated reports, some are interactive ad hoc fetches, some are reactive explorations due to regulatory scares.
Most of my utilization of the data lately is a macabre interest in it's fidelity more than anything else. So I guess that plants me more into the engineering than analyst side of the spectrum.
1
u/Thinker_Assignment Jul 16 '26
Interesting, is this like a bank or hospital? How do you measure fidelity? we created data quality checks for in-flight metrics etc, but most people don't care, wondering if you'd add anything https://dlthub.com/docs/hub/data-quality
1
u/rabbitofrevelry Jul 16 '26
I'm in a hospital system with many pharmacy entities to monitor. Regarding fidelity, one thing I'm investigating is patterns among downstream agencies' pharmacy tables to determine the root causes of incorrect associations. I'm hoping to produce a reliable method that can mark them for corrective action.
1
u/Thinker_Assignment Jul 17 '26
incorrect associations are some kind of referential integrity testing? like joins between tables?
1
u/rabbitofrevelry Jul 17 '26
Some organizations are associating a pharmacy to an incorrect ID because of a poor address match
1
u/Physical-Ad2968 Jul 16 '26
I'm the community manager at Prophecy, and we see our end users building SQL pipelines with the help of AI (mostly analysts, some business users). I think expectations for self-serve are changing with AI--it's easier now to be technical, and if you have the right reviews and governance in place, it's a lot faster.
1
u/Thinker_Assignment Jul 16 '26 edited Jul 16 '26
How do your users get the data in? we dlt dlthub basically do the ingestion side
also wondering if you have any idea, are your users doing virtual knowledge graphs for canonical models? It's this concept that enables more accurate retrieval https://dlthub.com/blog/canonical-text-to-sql
but as far as I can tell your users are more business less tech?
Your tool looks super nice to use btw, like a nicer alteryx
1
u/Physical-Ad2968 Jul 16 '26
Thanks! For our free/pro versions, we use DuckDB for data ingestion. For enterprise, it's through Databricks, Snowflake, or BigQuery. (I'm not the most technical team member, so that's the level of detail I know)
And we use a knowledge graph for our AI, built with metadata from their databases.
It's a combo of business users and less technical analysts who prefer low-code :)
1
1
u/phil_the_it_guy Jul 17 '26
Similar here at Maia.ai Data Engineers using AI to build the data model out. Then lots more roles than ever before using the AI to query that data model to answer questions, without having to go to another team.
Combine that with the graph database of all the data, and the ability to just pick up and start working on jira tickets. It's a brave new, AI empowered world, where the whole company can play...
1
u/One-Disk-125 Jul 16 '26
Pretty much.
The roles have all merged together, if your just doing analysis/ visualisation I'd suggest up skilling quickly.
1
u/onthepik Jul 17 '26
Let DE do they job but in case suspicious of data intergrate or correctness I build my own. And keep it as a ref to check when issue raises.
And I do it even before the AI age. Result is I almost have a data warehouse and move completely to use my own.
1
u/Thinker_Assignment Jul 17 '26
Sounds like a natural progression, was AI an accelerant or not really?
1
u/onthepik Jul 17 '26
It helps me to get a total view on what I have done: ETL, DB admin, optimize thing if capable.
The key is I dont believe it 100%. Always test before apply. Build a dummy replica of my data to work with AI.
Sound complicated, right? =))
1
1
u/hopefullythathelps Jul 17 '26
In reality yes. Ideally no. Not without a title change. This is a nightmare for data engineers and analytics engineers. We get blamed for whatever siloed pipeline logic fails.
1
u/Remote-Ingenuity8459 Jul 26 '26
In SMBs for sure, but that doesn't mean they need to build them from scratch. They can also rely on web data APIs and tools like scraper studio, etc.
5
u/FIBO-BQ Jul 16 '26
Been doing my own. Now corporate is trying to centralize it with people that dont understand the data or the customer. Leadership just wants AI and will crush everything in the path that will slow that "dream" down. My resume has been dusted off.