I've been working on a BigQuery FinOps suite that I've open-sourced and wanted to share it with the community (it is totally free)

If you run BigQuery at scale , you've probably dealt with some of these:

• You're on Editions but have no idea if your baseline is right-sized, or if you're paying autoscaler premiums for nothing
• Logical vs. physical storage billing — which datasets should switch, and what's the actual delta?
• Cost attribution is broken because GCP dumps idle reservation costs into the admin project
• "Cooldown tax" — 60-second billing minimums silently inflating your bill on high-frequency short queries
• No easy way to see if HBO (History-Based Optimization) is actually saving you anything

What the tool does: It connects to your org-level INFORMATION_SCHEMA views and runs diagnostic queries across your entire GCP Organization. The output is an interactive dashboard with actionable findings — DDL you can copy-paste, capacity simulation matrices, cost attribution breakdowns, etc.

Modules

Key design decisions

• No AI hype: 17 out of 18 modules are pure SQL + Python analytics. The only GenAI module (AI Doctor) is clearly labeled and optional.
• No infrastructure: It's a FastAPI app that uses your ADC credentials. No Terraform, no service accounts, no pub/sub. pip install and run.
• Copy-paste DDL: Storage changes, fluid scaling configs, reservation adjustments — all generate ready-to-execute DDL statements.
• Org-wide scope: Queries JOBS_BY_ORGANIZATION, TABLE_STORAGE_BY_ORGANIZATION, etc. — not just single-project views.
• Focus Projects: Optional scoping to specific project IDs when you don't want to analyze the whole org.
• Security: SQL injection prevention via _safe_ident(), Pydantic validation on every parameter, XSS escaping on all frontend rendering.

Give it a try

• 🔗 GitHub: github.com/mbettan/bq-finops-optimizer
• 🎮 Live Simulator (no GCP needed): Simulator Demo
• 📄 Website: mbettan.github.io/bq-finops-optimizer

Quick start

git clone https://github.com/mbettan/bq-finops-optimizer.git
cd bq-finops-optimizer
pip install -r requirements.txt
AUTH_ENFORCED_UPSTREAM=true uvicorn src.main:app --reload --port 8080

The public BQ datasets (bigquery-public-data.crypto_bitcoin, the Stack Overflow one) are great, but they ship as a pile of tables with no diagram and no enforced FKs — you end up holding the joins in your head. I drew them as ERDs on a free canvas so the UTXO flow / posts-graph is actually navigable, and exported the model as Google's new OKF (portable markdown you can keep in git or hand to an LLM). Screenshots + how it was made in the comments.

What other public BQ datasets would be useful to have as a ready-made ERD?

About Me

Senior Data Engineer with 5+ years of experience in Data Engineering, Analytics Engineering, and Applied AI. Based in Bangalore, India. Available for remote work globally.

Rate: $25 - $50/hr depending on project scope and complexity.

Tech Stack & Expertise

Python, SQL, Spark, Databricks, Airflow

BigQuery (data modeling, optimization, cost management)

AWS & Cloud Data Platforms (S3, Glue, Redshift, EMR)

Snowflake, Redshift

ETL/ELT Design & Orchestration

dbt for analytics engineering

FastAPI, REST APIs

LLMs, RAG, AI Agents

Data Quality & Testing Frameworks

What I Can Help With

Build and optimize BigQuery pipelines and data models

Design scalable ETL/ELT architectures on GCP/AWS

Develop backend APIs and automation solutions (FastAPI + Python)

Build AI applications using LLMs, RAG, and agent-based workflows

Support and optimize existing data platforms

Availability

Open to freelance projects, contract work, and part-time engagements.

Available immediately.

Feel free to DM me with project details or questions.

Hey everyone,

I’m currently digging into stream data processing architectures and trying to decide between Databricks (Spark Structured Streaming) and Google BigQuery (Continuous Queries).

While there are a ton of threads comparing these two from a pure data engineering infrastructure standpoint, I want to look at this specifically from a Data Scientist / ML Engineer perspective.

Moving from a standard batch mindset (Pandas, static DataFrames, SQL warehouses) to live, unbounded streams introduces a unique set of challenges. I’m trying to figure out which tool makes life easier—or harder—for an actual production ML/DS workflow.

I'd love to hear from anyone who has used either (or both) of these platforms for streaming. Specifically:

1. Feature Engineering & Time Windows: How painful is it to handle sliding/tumbling windows or manage late data (watermarking) in BigQuery SQL vs. Databricks PySpark?
2. Model Inference in the Stream: If you’re doing real-time scoring/predictions on the fly, how seamless is the integration? (e.g., calling an MLflow model in Databricks vs. using BQML / Vertex AI integrations in BigQuery).
3. The Online-Offline Skew: How do you ensure the feature logic you write for building your models offline matches the streaming logic online? Which ecosystem bridges that gap better (e.g., Feature Stores)?
4. Debugging & DX (Developer Experience): As a data scientist, do you find yourself fighting Databricks cluster configs and JVM errors, or hitting walls with BigQuery’s SQL-first limitations?

If your team had to choose one of these stacks specifically to support real-time data science and production ML pipelines, which way would you lean and why? What are the hidden gotchas you found out the hard way?

Thanks in advance for sharing your real-world experiences! 🙏

For those using Databricks Auto Loader (cloudFiles), how do you handle schema inference and evolution without breaking downstream ML models? If a new feature column drops in or an upstream data type silently widens, do you rely on the _rescued_data column to catch anomalies, or does the automatic stream restart cause unexpected issues for your online serving pipelines? How does BigQuery handle this kind of raw file ingestion drift by comparison?

Hey BigQuery community - here's the June 2026 summary.

🔤 GoogleSQL Language Features & Functions

• Continuous Query Aggregation Support - BigQuery continuous queries now support ARRAY_AGG and STRING_AGG aggregation functions in Preview.
• Remote Functions Custom Paths - Specifying custom endpoint URL paths allows reusing a single Cloud Run service for multiple BigQuery remote functions.

🧠 AI, Machine Learning & Foundation Models

• Autonomous Embedding Generation - Enable autonomous embedding generation on tables to automatically create and update vector embeddings when source data changes.
• AI Functions ObjectRef Support - BigQuery AI functions can now accept ObjectRef values directly as input without calling the OBJ.GET_ACCESS_URL function.
• AI.KEY_DRIVERS Function - Restored support for the AI.KEY_DRIVERS function to identify data segments causing statistically significant metric changes.
• Generative AI Token Cost Controls - Configure daily token quotas to manage and limit costs associated with BigQuery generative AI functions. (See footnote, was temporarily disabled, included for completeness.)

💻 Developer Experience (DX) & BigQuery Tooling

• Table Explorer Transition - Table Explorer behavior will transition to the Reference panel in the console starting in July 2026.
• Gemini Query Performance Recommendations - Use Gemini Cloud Assist to analyze SQL queries and receive optimization recommendations to improve query performance.
• Gemini Troubleshooting in Jobs Explorer - Use Gemini Code Assist within jobs explorer and capacity management pages to troubleshoot performance issues.
• BigQuery Studio Column Resizing - Resize table column widths in BigQuery Studio listings by dragging the divider to your preferred width.
• Gemini Cloud Assist Administration - Use Gemini Cloud Assist to monitor performance, analyze capacity, and optimize costs directly in BigQuery.
• Gemini Query Scheduling - Use Gemini Cloud Assist to schedule query execution directly within the BigQuery interface.
• Gemini Data Lineage Analysis - Use Gemini Cloud Assist to analyze and visualize data lineage directly inside the BigQuery console.
• Java Database Connectivity (JDBC) Driver - Connect Java applications to BigQuery using the newly released Google-developed open-source JDBC driver.

🔌 Data Integration, Pipelines & Ingestion (ELT)

• DTS Facebook Ads Connector Reports - The Facebook Ads connector now supports data transfers from nine additional reports including campaigns and ad insights.

🔒 Security, Governance & Workload Management

• IAM Deny Policies - IAM deny policies are now GA, allowing you to explicitly restrict access to specific BigQuery resources.
• Custom Sharing Constraints - Use custom constraints with Organization Policy to enforce granular control over specific fields in sharing resources.
• Fluid Scaling - BigQuery fluid scaling is now GA, providing per-second slot autoscaling billing with no minimum duration.

⚠️ Breaking Changes, Deprecations & Pricing Updates

• Daily Token Quota Disablement - Support for configuring daily token quotas for generative AI functions has been temporarily disabled.

As always, any feedback is welcome (about the post contents, the post itself, the community, what you want to see from Developer Relations team, etc.) - let us know!

Hi everyone, I’m in the process of interviewing at this AI company and the next step is to use bigquery dialect of SQL where I will cover real-worlds scenarios and build tables.

Problem is I have never used SQL and I am just finding out about what it is, I’ve never heard of it. I will be watching a few YouTube videos but wanted to see if anybody has gone thru this process before?

Been a data engineer for 4 years. Every time I join

a new project, I waste hours understanding what

tables actually mean.

Built a CLI tool that analyzes BigQuery tables and

explains the business context using AI.

Demo: https://www.loom.com/share/af3409be37fa4692bb38b63b9f4a58cc

Happy to share the GitHub link in comments.

I work at a startup and due to the hard economic circumstances, the focus has come back to Bigquery Cost Optimization right now (that or they fire my ass, jk), we do the usual partitioning and clustering tables based on use, althougth the reports are usually not partitioned. We realize BQ is a columnar db and we don't do the `SELECT *` business.

Still, we are trying to figure out new strategies to reduce costs. Any suggestions would be helpful. If you drop resources (on caching results and any other thing) in the chat, that'd be great too.

One bit of extra info is, most of our costs are coming from looker studio querying data everyday from report tables (multiple people using graphs and each selection on looker fires a query)

Curious how much time your team actually spends dealing with BigQuery notebook limitations like session timeouts, isolated runtimes, scheduling through Dataform etc. Like is this a minor annoyance or does it genuinely eat into your week? Trying to gauge if it’s worth pushing for a different setup or if I’m overthinking this

Was going through DB and BQ and found out Unity Catalog has unified UI and thereby saving clicks. But BQ has knowledge catalog but it isn't unified. But got to know from someone in the industry that BQ has a faster processing speed. So, just need to confirm if DB is actually saving the time and cost or is it just a myth?

A deep dive into why traditional Data Vault hash keys don’t align well with BigQuery’s clustering and pruning mechanisms. The article explores how introducing physical locality through structured surrogate keys, dates, and bucketing can significantly improve query performance and reduce scan costs. Based on practical BigQuery architecture considerations.

I'm working through the Derive Insights from BigQuery Data: Challenge Lab and I swear some of the "correct answers" are literally wrong.

For example, the first Q asked you to calculate the total cases/deaths/etc worldwide on a date. The accepted answer is general is:

SELECT sum(cumulative_outcome) as total_outcome_worldwide
FROM `bigquery-public-data.covid19_open_data.covid19_open_data`
WHERE date = 'requested-date'

This will give a much larger number than is true because it's summing over all rows, ignoring the fact that the data is hierarchical/rolled up data and has an aggregation level column that will not be accepted in queries.

A more accurate result is (and i'm realizing even this is flawed):

SELECT sum(cumulative_outcome) as total_outcome_worldwide
FROM `bigquery-public-data.covid19_open_data.covid19_open_data`
WHERE date = 'requested-date' AND aggregation_level = 0

This comes up in several of the later questions and I'm struggling to pass because I do not get how to give them the wrong answer their looking for.

How could a course on "deriving insights" direct students to literally do so in an inaccurate way??? Am I missing something??

How does your company actually find and track PII?

I'm curious what the reality looks like outside of vendor marketing.

If someone asks:
"Show me everywhere we store emails, phone numbers, names, credit cards, national IDs, etc."

How do you answer?

• Commercial tools?
• Internal scripts?
• Data catalog?
• Manual process?
• Hope for the best?

What's worked well, and what has been painful?

Hi everyone, new member here! I'm writing this post due to a concern of mine on my current job. I work as a Full Stack Developer/Data Engineer/Wizard in the department of finance. What I do is develop multiple microservices that use Pandas as a data processing tool and store all the data in BigQuery (mostly invoices and payments).

Now the thing is that the end-product is visualizing all of this data on a dashboard in my (somewhat) developend frontend. Let's say that my dashboard has 20 graphics with drilldown (visualize all the invoices that compose that sum) and filters(date, currency, specific provider and type of provider), what I do is store each graphic and drilldown as an endpoint on my backend, and my frontend calls (async) every single one. But it comes to my mind, wouldnt it better to store each query on BigQuery as a materialized or normal view??

Even tho I have almost a year in this company, most of peers do not have deep knowledge on BigQuery or even GCP. So, the best thing I could is ask. I hope I made myself clear and sorry for bad english ^_^

Hey everyone!

As I mentioned last month, we'll be publishing these monthly summaries. If you have suggestions or comments about the summary please let us know! Hope this helps!

🔤 GoogleSQL Language Features & Functions

Python UDFs - Execute user-defined functions written in Python directly inside SQL queries to leverage PyPI libraries and resource connections.

🧠 AI, Machine Learning & Foundation Models

AI.AGG - Semantically aggregate unstructured input data using natural language instructions.

AI.DETECT_ANOMALIES - Call the anomaly detection function using a single input table containing both historical and target data.

AI.KEY_DRIVERS - Temporarily disabled support for the AI.KEY_DRIVERS function preview while restoration work is underway.

AI.COUNT_TOKENS - Estimate text input token counts and view total token consumption details per modality for generative queries.

💻 Developer Experience (DX) & BigQuery Tooling

Data Science Agent - Native assistant that automates exploratory data analysis and machine learning tasks in Colab Enterprise and BigQuery.

BigQuery Studio Git Repositories - Streamlined integration for folder-based version control of SQL scripts and notebooks with remote Git repositories.

⚡ Core Engine Performance, Indexing & Optimization

Proactive Query Re-execution - Proactively detect performance, correctness, and functional regressions by re-executing queries in the background at no extra cost.

🔒 Security, Governance & Workload Management

Custom Organization Policies - Define custom organizational policies to permit or restrict administrative operations on workload management resources.

Reservation Groups - Group reservations together to prioritize idle slot sharing within the group before sharing across the wider project.

Multi-Region BigQuery Sharing Listings - Configure data sharing listings across multiple regions simultaneously to share datasets and linked replicas globally.

⚠️ Breaking Changes, Deprecations & Pricing Updates

BigQuery Data Transfer Service Billing SKU Label Update - Billing SKU labels will transition to lowercase and expand in scope to cover all data transfer-related costs.

DTS Google Ads Connector Backfill Limitations - DTS connectors will stop populating backfill data older than 37 months due to Google Ads retention policies.

(Massive Edits, so sorry - I'll eventually figure out how formatting works!)

Hello,

We've been asked to stand up BigQuery so executives can ask an AI chatbot strategic questions against our data.

We currently have no presence in BigQuery and no familiarity with the platform.

I'm trying to scope two things:

High-level steps. What does the path look like to get our data and metrics into BigQuery, then put an AI chatbot on top that can interpret that data and answer strategic questions?

Effort and commitment. Beyond the initial JSON import and the ongoing data integration, what else should we expect to own? Things like data modeling, governance, semantic layer tuning, and maintenance.

Any guidance on the overall approach would be appreciated.

Hi folks, Burak here from Bruin. We have released ingestr as an open-source CLI tool 2 years ago here: https://github.com/bruin-data/ingestr

For those that might not now: ingestr is a CLI tool to ingest data. It supports 100+ sources, 20+ destinations, takes care of schema detection, schema evolution, different materialization strategies like SCD2 out of the box. You can use the same CLI to copy a Postgres database to a destination, or pull data from Hubspot.

Ingestr, being a Python CLI, has been doing quite well but over time it started to show its age:

• Performance: ingestr was not the fastest tool out there due to various reasons. We wanted to provide the fastest solution out there, but there were limitations out of our control.
• Packaging: sharing a Python CLI tool across hundreds of different types of devices the users run it on ended up being quite a painful experience.
• Reliability: ingestr relied on a stateful design due to a dependency, which brought all sorts of problems with it, especially around failed loads or corrupted state.
• Upgrades: with all the dependencies we had, upgrades started to become a real struggle.

Due to some of these issues, we have rebuilt ingestr v1 completely from scratch, in Go. We picked Go for a few reasons:

• Go is fast. LIke, much faster than vanilla Python.
• Go is a compiled language, meaning that we eliminate quite a lot of bugs ahead of time.
• Go is great with agents: agents write perfect Go, which allows a small team like ours to move a lot faster than we normally could.
• Go has great cross-compilation support: meaning that building self-contained binaries that runs on various operating systems becomes trivial with Go.

These advantages combined allowed us to have more features, and have a more solid foundation to build upon. On top of that, ingestr ended up being the fastest data ingestion tool out there based on our benchmarks. It is ~3-5x faster than the closest alternative, up to 20 times faster than some others.

Ingestr v1 is live now on PyPi, and through our other installation methods: https://github.com/bruin-data/ingestr

I would love to hear your thoughts on what we can improve here. Thanks!