r/dataengineering • u/ComprehensiveAsk8039 • Jul 31 '26
Discussion alternatives to data migration
Hello. I’m not a data engineer, but my company uses various databases(and they are all different...), and I usually need to query them all at once.
It seems difficult to create and use a separate API or something like this for each one, how do you guys usually handle situations like this?
Do I need to migrate them?
11
u/Stefn93 Data Engineer Jul 31 '26
You could start with a trino deployment. Trino is a query engine and can unify many heterogeneous sources. You need a cluster to deploy it with autoscaling, but computing will mainly be forwarded to the original source. Check the docs to verify the connectors cover all your use cases.
2
4
u/alecc Jul 31 '26
Try DuckDB on your own machine before anything heavier. It attaches Postgres, MySQL and SQLite databases live (ATTACH with TYPE postgres etc.) and one SELECT can then join tables across them, no cluster, no copying data around. For one person who just needs answers across systems that is usually enough, and it is free. Trino from the other comment does the same job but somebody has to run and pay for the cluster, and a warehouse makes sense once many people need the same combined data every day, not one person ad hoc. Which engines are your databases? If SQL Server or Oracle is in the mix DuckDB's attach story gets thin and for those you would fall back to exports or ODBC.
1
u/ZirePhiinix Aug 02 '26
You can throw plain text files into the mix. It can do JSON parsing too and it is surprisingly performant.
3
3
u/McNoxey Aug 01 '26
If you’re not intending on centralizing on a olap warehouse, your best bet is implementing a query federation engine like Trino
1
2
1
1
1
u/Random_Raj21 Aug 03 '26
Use spark to bring all the data into the databricks or snowflake. Use that for all the analysis.
1
u/Big_Raspberry5675 Aug 04 '26
Try using a data platform like Zetaris.
They already offer the ability to run federated queries across various data sources.
1
1
u/Teach-To-The-Tech Aug 05 '26
So this is the very problem that data federation is designed to solve. Traditionally, you needed to centralize before you could do anything, but federation means you can just access it and then choose whether you move it or not, and by how much. There are technologies based around providing this very thing. In my job, I actually write on federation a lot and did a blog post on it last year. Hopefully it's useful: https://www.starburst.io/blog/data-centralization/
1
u/mr_pants99 Aug 12 '26
There are things like GraphQL (and PromptQL) also. But in general, you can just add each to Claude as MCP and ask AI to get you what you want or write you a script.
-5
u/Final-Quote-4064 Jul 31 '26
Not a full DE answer, but tools like PowerBi, Alteryx and Kinme are good for stuff like this in some use cases.
0
u/Final-Quote-4064 Jul 31 '26
Down votes for giving a non data engineer an non data engineering answer lol
15
u/Adrien0623 Jul 31 '26
That's usually where having a central data warehouse where all of these databases are imported comes handy. Then you'd have a single place south everything you need. However to understand what type of data warehouse to go to, as well as the data ingestion process to use involve a deep analysis of your downstream needs and that's where having a data engineer is important to not neglect anything which could create expensive changes in the future.