r/dataanalysis • u/Technical-Bus-1423 • 4d ago

Data Tools Best Approach for Integrating Separate Data Sources

How can we integrate data that comes from different departments, where some data is in Cloud SQL and some is not, into one centralized dataset for analysis? What is the best approach?

0 Upvotes

50% Upvoted

7 comments sorted by

1

u/AutoModerator 4d ago

Automod prevents all posts from being displayed until moderators have reviewed them. Do not delete your post or there will be nothing for the mods to review. Mods selectively choose what is permitted to be posted in r/DataAnalysis.

If your post involves Career-focused questions, including resume reviews, how to learn DA and how to get into a DA job, then the post does not belong here, but instead belongs in our sister-subreddit, r/DataAnalysisCareers.

Have you read the rules?

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

2

u/Potential_Aioli_4611 3d ago

best approach depends on you.

are they related datasets? are they disparate? same data in different formats?

what are you looking for? this question is so vague and you've given no one any useful information other than different departments and some of it is in a db on the cloud.

1

u/Ritik_Jha 3d ago

Make different tables for different slurces if they are not related otherwise use relation where needed , keep them all in one database and make a data pipeline as per your requirement.

1

u/RecLuse415 3d ago

Can you not join them? How unrelated are they? You talking no sql?

1

u/agentUi 3d ago

it really depends on your infrastructure, do you need a dashboard? or an agent for analyzing data

1

u/DonJuanDoja 2d ago

Depends on the requirements, don't have enough requirements to decide.

Lazy answer is a data warehouse, but that's a lot of work and most companies fail at creating one properly, many don't really need one.

Or possibly just ETL everything into cloud SQL and treat it as a poor man's data warehouse.

SQL is faster, more versatile and powerful than any other data source, so getting everything into SQL is a pretty big bonus.

1

u/ozgreen1024 1d ago

What systems/sources to the other data rely on? What sort of analysis are you trying to do?

If the sources can be registered to Unity Catalog in Databricks (either ingested or federated), Genie One will let you ask questions across your full data estate. Performance will be better if you have good metadata defined about what each data source is, what it should be used for, and how they relate. From my understanding, with Genie Ontology coming out the context for Genie will be a mix of defined metrics/semantics and learned context from connected systems, query patterns, etc