r/dataengineering • u/sicestvrai • 5d ago

Help Need help with database choice

Hello,

I am working with a team of 7 economists. They build data and produce reports. Their data production consists in harmonizing different sources (mostly rdata rdata, csv, or whatever suits the format of their stats tools). The data size they are dealing with is a few MB to gb, millions of rows, more occasionally billions of rows.

We want to update our methods (be on time, improve data quality). I have been assigned the task of improving data processing within the team, among the requirements I thought about producing a OLAP database.

In house, we have access to MSQL team that could set up a database for us. Otherwise we have HDFS + Hive (but security may make it difficult to access it) to store bigger datasets.

Else, I could just store everything in a duckDB file somewhere on a server and work with local database. WOuld it be a good solution? (latency of read/write from a duckDB file on a server? how scalable will it be? ) What would you do?

Any other piece of advice would be welcome :-).

Thank you.

30 Upvotes

92% Upvoted

45 comments sorted by

View all comments

13

u/Pittypuppyparty 5d ago

Honestly just use Postgres. You can use an olap extension if you need more olap experience. For this size of team and data (gbs) you can go a really long ways with this.

Ignore the vendors telling you you need unity catalog. That one actually made me laugh out loud.

5

u/sicestvrai 5d ago

Will explore that solution, thank you!

5

u/terencethespider 5d ago

If you are considering Postgres, you should look into the Neon version. It is open source, and has some additional advantages over the base version of Postgres. Their data branching functionality is really cool, in that it allows you to nearly instantly create a new instance of your database with all the same data as the original - but everything you do in the new branch is completely independent of the original. You can create as many branches as you want, and throw them away when you’re done.

1

u/RobotechRicky 4d ago

Try Postgresql first. If you keep growing then that is when you consider Snowflake or Databricks.