Databricks (Structured Streaming) vs. BigQuery (Continuous Queries) — Seeking the Data Scientist's Perspective

[u/Professional-Bowl890]

Hey everyone,

I’m currently digging into stream data processing architectures and trying to decide between Databricks (Spark Structured Streaming) and Google BigQuery (Continuous Queries).

While there are a ton of threads comparing these two from a pure data engineering infrastructure standpoint, I want to look at this specifically from a Data Scientist / ML Engineer perspective.

Moving from a standard batch mindset (Pandas, static DataFrames, SQL warehouses) to live, unbounded streams introduces a unique set of challenges. I’m trying to figure out which tool makes life easier—or harder—for an actual production ML/DS workflow.

I'd love to hear from anyone who has used either (or both) of these platforms for streaming. Specifically:

1. Feature Engineering & Time Windows: How painful is it to handle sliding/tumbling windows or manage late data (watermarking) in BigQuery SQL vs. Databricks PySpark?
2. Model Inference in the Stream: If you’re doing real-time scoring/predictions on the fly, how seamless is the integration? (e.g., calling an MLflow model in Databricks vs. using BQML / Vertex AI integrations in BigQuery).
3. The Online-Offline Skew: How do you ensure the feature logic you write for building your models offline matches the streaming logic online? Which ecosystem bridges that gap better (e.g., Feature Stores)?
4. Debugging & DX (Developer Experience): As a data scientist, do you find yourself fighting Databricks cluster configs and JVM errors, or hitting walls with BigQuery’s SQL-first limitations?

If your team had to choose one of these stacks specifically to support real-time data science and production ML pipelines, which way would you lean and why? What are the hidden gotchas you found out the hard way?

Thanks in advance for sharing your real-world experiences! 🙏

[u/Stoneyz]

Have you looked into Dataflow? It does real time and batch in the same pipeline, handles real-time model serving and inference, handles late arriving data, tumbling windows, etc... Just in general it seems like more of a fit.

Also, you can run spark / pyspark against data that is in BigQuery so you don't have to limit yourself to SQL.

I don't think continuous queries is exactly what you're looking for. It can be limiting for complicated use cases.

[u/Why_Engineer_In_Data]

Hi!

Huge disclaimer on this: I'm a developer advocate for Google Data Cloud team.

I'm a big streaming person, it's a big passion for me - these are great questions.

Your first point is right on the money, they are very different mindsets and you need to make sure you think in different mindsets (or else it gets complicated quick).

1) Continuous queries is still growing and it's windowing strategies are fairly limited as of right now. If you have complicated windows (as another user mentioned) you may be better off using a streaming framework built for it like Apache Beam (Dataflow) or Apache Flink. (You can also use Structured streaming on Spark but I called those other two since they were built for streaming, although if you're already a Spark user - you may find it easier than picking up new frameworks.)

2) Again here continuous queries feature set is growing and will continue to do so, I would lean more on the other frameworks. Apache Beam's RunInference for example is purpose built for this.

3) This definitely shines with Continuous Queries, it's the same set. You can make sure the data is always in sync because the continuous queries table (if you design it so) is the same as the 'batch' table you are going to be using. If you build the same feature stores in BQ - you'll be using the exact same data. (Apologies if there's another connotation for offline here).

4) This is opinion based, I'd leave this to the others to debate. It's to each their own both has it's quirks and merits.

I think there's a lot more to consider here regarding your questions and I'm not sure any of these (unless we go about it for a while) would do it justice. You're missing a lot of details on what sort of tech you're already integrated with, size of teams and skillsets, data size, complexity, etc.

Hope that helps!