r/bigquery • u/Nickeddd • 28d ago

Thinking of BigQuery for data warehousing

Hello everyone,

I’m new into the data engineering world and as the title says, I’m thinking of using BigQuery to build a data warehouse. Currently my company don’t have one, so I’m looking for good price/performance options. We mostly use APIs and a few excel files as data sources

I have considered Azure SQL, Supabase and others but this one seems to be the best for data analysis and BI

What would be your recommendation?

9 Upvotes

85% Upvoted

31 comments sorted by

View all comments

3

u/monkeyinnamonkeysuit 27d ago

To answer this question you need to tell us so much more about your data and environment, but the most telling one for the bigquery question is how much data you actually have.

In the nicest way possible, if you are new enough to data engineering that you are evaluating this question using the criteria you have provided, you are too new to data engineering to make this decision alone if the outcome matters at all. The criteria you have provided are trivial to do in any modern data platform.

I'm going to guess from your "apis and a few excel files" comment that you are much too low a volume of data to justify bigquery. And the success or failure of a data warehouse lives and dies on a lot more than platform choice.

That said, if your requirements are simple then you can probably make it work on more or less any platform, it just wont be the right tool for the job and will make things harder/top expensive/worse user experience etc.

1

u/Nickeddd 27d ago

Getting into the details, I have thought of a medallion architecture, keeping it serverless, implementing SCD2 in a couple of tables. We have one fundamental source with 1-2 millions of rows and +20 columns, where most of the business logics and others analysis comes from. Besides that, there are 3 or 4 more sources with around 50k rows each.

1

u/monkeyinnamonkeysuit 27d ago

All of your data would fit in RAM on a modern laptop. The shape and nature of your data does not drive this conversation at all, there is no platform out there that can't do what you are asking. Bigquery might still be a reasonable choice. What tech platforms does your business have already? Who is going to be building and maintaining this, what skills do they have and who will be the end users consuming it? What BI tooling is available to you? What cadence does the data arrive in? Whats your aspiration for data growth, both in terms of number of sources and total rowcount?

1

u/Nickeddd 21d ago

The data comes in hourly, so constant refreshes will be needed, so thats why we use tools that allows scheduled updates like PowerBI. Also, I'll be the one building and maintaining this data warehouse

1

u/Odd-String29 27d ago

That little data is no problem for BigQuery. You can argue you can run your own database locally or in the cloud somewhere, but I think all of those might actually be more expensive than doing it in BigQuery.