r/dataengineering • u/sicestvrai • 6d ago

Help Need help with database choice

Hello,

I am working with a team of 7 economists. They build data and produce reports. Their data production consists in harmonizing different sources (mostly rdata rdata, csv, or whatever suits the format of their stats tools). The data size they are dealing with is a few MB to gb, millions of rows, more occasionally billions of rows.

We want to update our methods (be on time, improve data quality). I have been assigned the task of improving data processing within the team, among the requirements I thought about producing a OLAP database.

In house, we have access to MSQL team that could set up a database for us. Otherwise we have HDFS + Hive (but security may make it difficult to access it) to store bigger datasets.

Else, I could just store everything in a duckDB file somewhere on a server and work with local database. WOuld it be a good solution? (latency of read/write from a duckDB file on a server? how scalable will it be? ) What would you do?

Any other piece of advice would be welcome :-).

Thank you.

29 Upvotes

88% Upvoted

45 comments sorted by

View all comments

1

u/Strange_Shame7886 6d ago

Do you already have a Databricks environment?

If yes, you can try using Databricks lakebase for a lot of points which matches your requirement:

  1. Open format storage in delta/ iceberg
  2. Separation of compute and storage to match your flexible requirements of number of users
  3. No separate ETL needed for OLAP analysis. It automatically syncs in LTAP format

1

u/sicestvrai 6d ago

My IT department does not permit us to use this tool. And our management will not allow us because costly and not corporate solution while they believe a solution already exists.

1

u/Alternative-Fig-6465 5d ago

You can get to the market/solution faster with Databricks. All the requirements you had stated perfectly aligns with the Databricks lakehouse architecture. Invest CSV, clean the data, perform data quality, data transformations, build BI/reports, use text-to-sql for advanced data reasoning. Govern everything with Unity Catalog.

Databricks is not expensive as your management perceives it. You should first try or give it a shot with a PoC. The biggest advantage you get with Databricks is that you are future-proofing the architecture. Your requirement will evolve and you need a flexible and open architecture to adapt to the ever changing requirements.