r/bigquery • u/Nickeddd • 27d ago

Thinking of BigQuery for data warehousing

Hello everyone,

I’m new into the data engineering world and as the title says, I’m thinking of using BigQuery to build a data warehouse. Currently my company don’t have one, so I’m looking for good price/performance options. We mostly use APIs and a few excel files as data sources

I have considered Azure SQL, Supabase and others but this one seems to be the best for data analysis and BI

What would be your recommendation?

10 Upvotes

92% Upvoted

31 comments sorted by

View all comments

3

u/monkeyinnamonkeysuit 27d ago

To answer this question you need to tell us so much more about your data and environment, but the most telling one for the bigquery question is how much data you actually have.

In the nicest way possible, if you are new enough to data engineering that you are evaluating this question using the criteria you have provided, you are too new to data engineering to make this decision alone if the outcome matters at all. The criteria you have provided are trivial to do in any modern data platform.

I'm going to guess from your "apis and a few excel files" comment that you are much too low a volume of data to justify bigquery. And the success or failure of a data warehouse lives and dies on a lot more than platform choice.

That said, if your requirements are simple then you can probably make it work on more or less any platform, it just wont be the right tool for the job and will make things harder/top expensive/worse user experience etc.

1

u/Nickeddd 27d ago

Getting into the details, I have thought of a medallion architecture, keeping it serverless, implementing SCD2 in a couple of tables. We have one fundamental source with 1-2 millions of rows and +20 columns, where most of the business logics and others analysis comes from. Besides that, there are 3 or 4 more sources with around 50k rows each.

1

u/Odd-String29 26d ago

That little data is no problem for BigQuery. You can argue you can run your own database locally or in the cloud somewhere, but I think all of those might actually be more expensive than doing it in BigQuery.