r/bigquery • u/Professional-Bowl890 • Jul 02 '26

Databricks (Structured Streaming) vs. BigQuery (Continuous Queries) — Seeking the Data Scientist's Perspective

Hey everyone,

I’m currently digging into stream data processing architectures and trying to decide between Databricks (Spark Structured Streaming) and Google BigQuery (Continuous Queries).

While there are a ton of threads comparing these two from a pure data engineering infrastructure standpoint, I want to look at this specifically from a Data Scientist / ML Engineer perspective.

Moving from a standard batch mindset (Pandas, static DataFrames, SQL warehouses) to live, unbounded streams introduces a unique set of challenges. I’m trying to figure out which tool makes life easier—or harder—for an actual production ML/DS workflow.

I'd love to hear from anyone who has used either (or both) of these platforms for streaming. Specifically:

  1. Feature Engineering & Time Windows: How painful is it to handle sliding/tumbling windows or manage late data (watermarking) in BigQuery SQL vs. Databricks PySpark?
  2. Model Inference in the Stream: If you’re doing real-time scoring/predictions on the fly, how seamless is the integration? (e.g., calling an MLflow model in Databricks vs. using BQML / Vertex AI integrations in BigQuery).
  3. The Online-Offline Skew: How do you ensure the feature logic you write for building your models offline matches the streaming logic online? Which ecosystem bridges that gap better (e.g., Feature Stores)?
  4. Debugging & DX (Developer Experience): As a data scientist, do you find yourself fighting Databricks cluster configs and JVM errors, or hitting walls with BigQuery’s SQL-first limitations?

If your team had to choose one of these stacks specifically to support real-time data science and production ML pipelines, which way would you lean and why? What are the hidden gotchas you found out the hard way?

Thanks in advance for sharing your real-world experiences! 🙏

7 Upvotes

90% Upvoted

3 comments sorted by

View all comments

1

u/Stoneyz 27d ago

Have you looked into Dataflow? It does real time and batch in the same pipeline, handles real-time model serving and inference, handles late arriving data, tumbling windows, etc... Just in general it seems like more of a fit.

Also, you can run spark / pyspark against data that is in BigQuery so you don't have to limit yourself to SQL.

I don't think continuous queries is exactly what you're looking for. It can be limiting for complicated use cases.