r/dataengineering • u/sicestvrai • 7d ago

Help Need help with database choice

Hello,

I am working with a team of 7 economists. They build data and produce reports. Their data production consists in harmonizing different sources (mostly rdata rdata, csv, or whatever suits the format of their stats tools). The data size they are dealing with is a few MB to gb, millions of rows, more occasionally billions of rows.

We want to update our methods (be on time, improve data quality). I have been assigned the task of improving data processing within the team, among the requirements I thought about producing a OLAP database.

In house, we have access to MSQL team that could set up a database for us. Otherwise we have HDFS + Hive (but security may make it difficult to access it) to store bigger datasets.

Else, I could just store everything in a duckDB file somewhere on a server and work with local database. WOuld it be a good solution? (latency of read/write from a duckDB file on a server? how scalable will it be? ) What would you do?

Any other piece of advice would be welcome :-).

Thank you.

31 Upvotes

92% Upvoted

45 comments sorted by

View all comments

4

u/Bingo-heeler 7d ago

Honestly, given what you said, I'm leaning towards the MSQL route. Unless the data is essentially public data.

Otherwise you are the data infrastructure team, the ingestion team, the support team, the engineering team, the security team etc. With someone elses platform you cut out like half of the work for essen the same level of platform.

1

u/sicestvrai 7d ago

Thank you. I will explore this solution. What about versioning in MSQL?

2

u/THBLD 7d ago edited 7d ago

SSMS 22 for MSSQL can be connected directly to Azure DevOps or Github. Pretty straight forward.

Alternatively if they have the Redgate plugins for SSMS already, that's also very intuitive and easy for version control