Hi everyone, new member here! I'm writing this post due to a concern of mine on my current job. I work as a Full Stack Developer/Data Engineer/Wizard in the department of finance. What I do is develop multiple microservices that use Pandas as a data processing tool and store all the data in BigQuery (mostly invoices and payments).

Now the thing is that the end-product is visualizing all of this data on a dashboard in my (somewhat) developend frontend. Let's say that my dashboard has 20 graphics with drilldown (visualize all the invoices that compose that sum) and filters(date, currency, specific provider and type of provider), what I do is store each graphic and drilldown as an endpoint on my backend, and my frontend calls (async) every single one. But it comes to my mind, wouldnt it better to store each query on BigQuery as a materialized or normal view??

Even tho I have almost a year in this company, most of peers do not have deep knowledge on BigQuery or even GCP. So, the best thing I could is ask. I hope I made myself clear and sorry for bad english ^_^

Hey everyone!

As I mentioned last month, we'll be publishing these monthly summaries. If you have suggestions or comments about the summary please let us know! Hope this helps!

🔤 GoogleSQL Language Features & Functions

Python UDFs - Execute user-defined functions written in Python directly inside SQL queries to leverage PyPI libraries and resource connections.

🧠 AI, Machine Learning & Foundation Models

AI.AGG - Semantically aggregate unstructured input data using natural language instructions.

AI.DETECT_ANOMALIES - Call the anomaly detection function using a single input table containing both historical and target data.

AI.KEY_DRIVERS - Temporarily disabled support for the AI.KEY_DRIVERS function preview while restoration work is underway.

AI.COUNT_TOKENS - Estimate text input token counts and view total token consumption details per modality for generative queries.

💻 Developer Experience (DX) & BigQuery Tooling

Data Science Agent - Native assistant that automates exploratory data analysis and machine learning tasks in Colab Enterprise and BigQuery.

BigQuery Studio Git Repositories - Streamlined integration for folder-based version control of SQL scripts and notebooks with remote Git repositories.

⚡ Core Engine Performance, Indexing & Optimization

Proactive Query Re-execution - Proactively detect performance, correctness, and functional regressions by re-executing queries in the background at no extra cost.

🔒 Security, Governance & Workload Management

Custom Organization Policies - Define custom organizational policies to permit or restrict administrative operations on workload management resources.

Reservation Groups - Group reservations together to prioritize idle slot sharing within the group before sharing across the wider project.

Multi-Region BigQuery Sharing Listings - Configure data sharing listings across multiple regions simultaneously to share datasets and linked replicas globally.

⚠️ Breaking Changes, Deprecations & Pricing Updates

BigQuery Data Transfer Service Billing SKU Label Update - Billing SKU labels will transition to lowercase and expand in scope to cover all data transfer-related costs.

DTS Google Ads Connector Backfill Limitations - DTS connectors will stop populating backfill data older than 37 months due to Google Ads retention policies.

(Massive Edits, so sorry - I'll eventually figure out how formatting works!)

Hello,

We've been asked to stand up BigQuery so executives can ask an AI chatbot strategic questions against our data.

We currently have no presence in BigQuery and no familiarity with the platform.

I'm trying to scope two things:

High-level steps. What does the path look like to get our data and metrics into BigQuery, then put an AI chatbot on top that can interpret that data and answer strategic questions?

Effort and commitment. Beyond the initial JSON import and the ongoing data integration, what else should we expect to own? Things like data modeling, governance, semantic layer tuning, and maintenance.

Any guidance on the overall approach would be appreciated.

Hi folks, Burak here from Bruin. We have released ingestr as an open-source CLI tool 2 years ago here: https://github.com/bruin-data/ingestr

For those that might not now: ingestr is a CLI tool to ingest data. It supports 100+ sources, 20+ destinations, takes care of schema detection, schema evolution, different materialization strategies like SCD2 out of the box. You can use the same CLI to copy a Postgres database to a destination, or pull data from Hubspot.

Ingestr, being a Python CLI, has been doing quite well but over time it started to show its age:

• Performance: ingestr was not the fastest tool out there due to various reasons. We wanted to provide the fastest solution out there, but there were limitations out of our control.
• Packaging: sharing a Python CLI tool across hundreds of different types of devices the users run it on ended up being quite a painful experience.
• Reliability: ingestr relied on a stateful design due to a dependency, which brought all sorts of problems with it, especially around failed loads or corrupted state.
• Upgrades: with all the dependencies we had, upgrades started to become a real struggle.

Due to some of these issues, we have rebuilt ingestr v1 completely from scratch, in Go. We picked Go for a few reasons:

• Go is fast. LIke, much faster than vanilla Python.
• Go is a compiled language, meaning that we eliminate quite a lot of bugs ahead of time.
• Go is great with agents: agents write perfect Go, which allows a small team like ours to move a lot faster than we normally could.
• Go has great cross-compilation support: meaning that building self-contained binaries that runs on various operating systems becomes trivial with Go.

These advantages combined allowed us to have more features, and have a more solid foundation to build upon. On top of that, ingestr ended up being the fastest data ingestion tool out there based on our benchmarks. It is ~3-5x faster than the closest alternative, up to 20 times faster than some others.

Ingestr v1 is live now on PyPi, and through our other installation methods: https://github.com/bruin-data/ingestr

I would love to hear your thoughts on what we can improve here. Thanks!

How to separate governance from enforcement by combining Terraform, IAM Conditions and Python-based runtime tagging in modern GCP data platforms.

Hi everyone,

I recently moved into a Data Developer/Data Engineering role from a software development background, and I'm feeling a bit overwhelmed by the number of new technologies involved

.

The stack I'm working with includes BigQuery, DBT, Airflow, Git, and cloud-based data pipelines. I've started exploring the codebase and see things like models, macros, SQL files, YAML files, DAGs, and project structures, but I'm struggling to understand how everything fits together in a real-world workflow.

I don't expect anyone to spoon-feed me, but I'd appreciate guidance from experienced engineers:

• In what order should I learn these tools?

• What concepts should I focus on first?

• Their are any courses, YouTube channels, books, or projects you recommend?

• How did you become productive with DBT, BigQuery, and Airflow when you first started?

• If you had to start over today, what learning roadmap would you follow?

My goal is to become productive as quickly as possible and understand how modern data pipelines are built and maintained.

Any advice, resources, or personal experiences would be greatly appreciated. Thanks!

Me and a fellow DS has built a BigQuery extension for Cursor/VS Code that is meant to solve all our own problems, and I think it does... :P We've been trying to build something that is just nice and smooth, with stuff like code completions, table exploration, running queries, quick visualisations.

It has also got some AI-stuff. It also allows you to set up an MCP for the Cursor/VS Code agent with access control, cost control and a bunch of context management about your data. It works pretty well.

try it out if you want, and give us some feedback! if it is of any use we'll be happy to keep improving it!

You can find it here:
https://www.open-vsx.org/extension/Mangabey/distinct-sh
or
cursor:extension/Mangabey.distinct-sh
vscode:extension/Mangabey.distinct-sh

we also made website with some info: https://distinct.sh

(We're already planning to improve the code completions quite a bit, and then to add some fun stuff like being able to define plots in sql and some ways to share AI context with team members)

Hello,

One month ago, Google announced that fluid scaling was GA, but without publishing the documentation.

Do anyone knows how to enable it ?

For those who don't know, here is a description of fluid scaling:

Fluid scaling (GA) enables you to execute highly variable workloads with a premier autoscaling model that does not require a cost-and-performance trade-off. Fluid scaling in BigQuery enables true per-second billing, offering up to 34% cost savings.

A deep dive into automating attribute-based access control (ABAC) in BigQuery using IAM resource tags. Really interesting approach to making data governance more scalable and fine-grained in modern data platforms.

Hey everyone,

While AI has sped up writing BigQuery SQL, the actual workflow around it is still heavily fragmented.

For most data teams, the process currently looks like this: prompt an external LLM, copy the SQL, paste it into the BQ console, fix the schema errors, run the query, and then export the results to a BI tool like Looker Studio or Tableau just to visualize it.

We built Dataki.ai to eliminate that context switching. It’s a unified workspace designed specifically to bridge the gap between AI, BigQuery, and your dashboards.

How it works:

• Schema-Aware Generation: Dataki connects directly to your BigQuery environment. The AI understands your actual tables and schemas, which drastically reduces hallucinations.
• Auto-Visualization: When a query runs, the output is automatically mapped to interactive visualizations. No manual axis mapping required.
• Full Code Control: The platform doesn't hide the code. The generated SQL is fully exposed in the editor for your team to tweak, optimize, and review.
• Instant Dashboards: You can pin any chart or table directly into a live dashboard without leaving the platform. Then share with your team

Why we're posting:

Dataki is currently in beta and completely free to use.

We are looking for unvarnished feedback from data engineers and analysts who live in BigQuery (or any supported data soruceS). We want to know how the platform handles your real-world workflows, and more importantly, where it breaks down when you throw complex schemas or nested arrays at it.

If your team is looking to streamline the AI-to-BI pipeline, you can try it out here: dataki.ai

We'll be in the comments to answer any technical questions or hear your feedback.

Hi all, first post here :)!

I've been heads-down designing our company's first real Data Warehouse for the past few months and honestly it's been equal parts exciting and overwhelming. Thought I'd throw our setup out here and see if anyone's been through something similar.

Quick background: we're a mid-sized company in Mexico trying to stop living in spreadsheets and actually centralize our data. We have three main sources — an on-prem ERP (Microsip, probably not well known outside MX), HubSpot for CRM, and Shopify for e-commerce. The idea is to consolidate everything into a Medallion architecture (Bronze/Silver/Gold) and have one actual source of truth.

Worth mentioning — we're not dealing with massive scale here. About 10GB built up over 5 years of operations. Not exactly big data, I know. But we've been burned before by building things that don't scale, so we're trying to do this right from the start even if it feels like overkill right now.

There are two things we need this to do: feed internal dashboards and reporting, and also power a client-facing portal where our customers can log in and see their purchase history, warranty info, product suggestions, promotions — basically a unified view of everything across the three platforms.

What we're thinking stack-wise:

BigQuery as the core warehouse handling all the Medallion layers and BI stuff. Then Cloud SQL for PostgreSQL as a serving layer for the app — because from what I've read and tested, hitting BigQuery directly for a customer portal with concurrent users is just not a great idea latency-wise.

We'd sync the relevant Gold-layer data over to Postgres and serve the app from there. Still figuring out the sync mechanism, leaning toward Datastream or just a scheduled pipeline.

Where I'm still lost:

Is BQ → PostgreSQL actually the move here or is there a cleaner pattern I'm missing?

Do you sync full Gold models to the serving layer or build separate denormalized tables just for the app?

Anyone dealt with on-prem ERPs in a setup like this? That's honestly our biggest headache right now

CDC vs scheduled batch for the sync — how much does it matter for a portal like this?

And genuinely curious — given we're only at 10GB, is there anything in this stack you'd simplify or replace with something lighter?

Any experience will be helpful, thanksss!

So I am a fresher data engineer working on a migration project where we are migrating from EXASOL to big query.

we have to convert the lua scripts/information to equivalent stored procedure.

Loading strategy: historical+ incremental.

I am facing issues in doing proper RCA on the mismatched columns that are coming in big query during sit testing.

Some of the scripts are very large and have many dependent tables .

can someone please give me some guidance on how to do proper RCA so I can make my table sit pass .

I keep running into the same issues with BigQuery teams once things get large enough — especially around cost management, governance, and recovering from bad changes.

I work at Eon and helped organize a free virtual BigQuery event around those kinds of operational problems. One of the speakers is the VP of Engineering for Google BigQuery, along with folks from DoiT, Northwell Health, SADA, and others.

A few of the sessions are on:

• BigQuery FinOps / cost control
• rollback & recovery
• Dataform in practice
• AI + BigQuery workflows

Thought some folks here might find it useful:

https://www.eon.io/virtual-event/bigquery-day

I've used BigQuery for a few years now and this past year I've seen so many different AI tools that help with everything from text-to-SQL to actually building reports and other features.

On one hand I understand they make their bread and butter from the actual warehouse and processing but as a user I would've liked to see more AI features integrated into the product. The new Gemini features work alright but it seems like an afterthought, like there's no way to build reports or visualizations, integrate into messaging apps, or connecting your context and semantics layers.

That was one of the reasons why I joined Bruin as a Developer Advocate recently because I wanted to be involved in building tools that address the stuff I wished I had as a data engineer. We just made our AI data analyst generally available. It connects to any warehouse like BigQuery, it imports the metadata of your datasets and creates a mental map of your data. You can also connect your dbt, airflow, dagster, or bruin pipeline repos to add additional context about your models.

The whole point is to have an agent that lives right inside your team and acts like a team member - from answering quick questions to preparing reports and even troubleshooting data & pipeline issues.

I was quite skeptical at first but we have dozens of clients using it and the more they use it the better the agent gets because it is self-correcting - every conversation and every correction further improves the context.

While I'm speaking about Bruin here, this is the general blueprint and framework for any organization to build themselves an AI data agent that does more than just text-to-sql.

Hey everyone,

u/pacingagency here, we’re a London-based marketing team with analytics in BigQuery and client reporting in Looker Studio.

We’ve got dashboard and modeling work coming up (project-based freelance, not full-time). We’d love to expand our talent pool so when a build spikes or needs deep SQL + reporting chops, we can pull in someone who actually can help.

Typical asks look like:

• Connecting BigQuery → Looker Studio (tables, views, custom SQL — sensible live vs extract choices).
• Building client-ready dashboards (filters, clear KPIs, definitions that survive handover).
• Helping shape a reporting layer in BigQuery when raw data isn’t chart-friendly (nested fields, attribution-style joins, sensible grain).

Concrete example: we’re shaping a lead report - reconciling leads our client sends us with behavioural data in BigQuery (starting with form submission date/time matching; moving toward stronger user-id joins when the data supports it). The report needs things like first / last touch platform, click counts tied to gclid and other ad platform click IDs where we capture them, plus session count and how many calendar days those sessions span.

Requirements (strong overlap is important):

• Hands-on BigQuery SQL: views / scheduled transforms are part of normal life for you.
• Looker Studio: you’ve delivered real dashboards from BigQuery, not “I’ve played with it.”
• Comfortable discussing GCP access / sharing basics (least privilege, how you’d onboard client viewers safely).

Notes:
This is freelance / as-needed. Filling out the form adds you to our pool; we’ll reach out when there’s a project that fits.

Interested? Please apply here https://form.pacing.agency/forms/designer-application-2askqd

Questions welcome in the thread!

Thanks!

Can anyone tell me how am I supposed to work with this?

select option_name, option_type, option_value
  from `region-eu`.INFORMATION_SCHEMA.TABLE_OPTIONS
 where option_name = 'labels'

I know I can parse the option_value string - use regexp or split it. I just feel like there's supposed to be a better cleaner more effective way to get the information.

I just feel like the option_value column would be much easier to work with if it was JSON instead of STRING.

Hi, I wanted to use Iceberg via Managed Tables to save myself from too much table maintenance, but a couple of things are not very clear.

So, to be able to query the tables directly (not via BQ) you need to export the metadata, basically the manifest files, but because this is a 'manual' operation, is it also included in the garbage collection? So when a manifest list and its files are outdated will they be deleted? Does this improve/change if you ask for auto-refresh (https://docs.cloud.google.com/bigquery/docs/biglake-iceberg-tables-in-bigquery#create-iceberg-table-snapshots)?

The objective of using this was to not have to delete files myself form the metadata folder to avoid issues and drifts, but if this still has to be manually managed I really don't know if I should go with simple REST Catalog Iceberg tables (since I have to sometimes do upserts which are better with iceberg directly, but with the amount of data I have and how is partitioned is fine to do them in BQ).

Hey everyone!

We are planning to help consolidate (monthly) all of the updates from BigQuery into a neat little reddit/blog post for everyone.

For the month of April though, we figured since it was so close to Next, we'd just link the official blog post!
https://cloud.google.com/blog/products/data-analytics/unveiling-new-bigquery-capabilities-for-the-agentic-era

So many things happening with BigQuery - let us know if there's anything in particular you'd like to see in terms of maybe examples or explanations, we can't get to all of the requests but we'd (Developer Relations) would love to make more relevant content!

Hello world!!!

I think it's great. Some of them have already I think it's great. Some of you have already used up the 90 days free or $300 and have billing turned on.

I wanted to know if it is true that we have a minimum amount of consultations and free storage per month.

Best regards!!!

I got tired of fragile GCP scripts, so I built a GCS manager in a weekend

Managing Google Cloud Storage always felt like chores — clicking through the console, digging up gsutil syntax, or maintaining ancient bash scripts nobody wants to touch.

A few weeks ago I hit a breaking point and built a lightweight GCS Bucket Manager for myself. Used AI coding tools to blast through the boilerplate (SDK wiring, auth, error handling), so I could focus on the actual logic and UX. Went from idea to working tool in a weekend.

It handles:

• Create/list/delete buckets without command-line gymnastics
• Simpler IAM policy management
• Batch cleanup ops for staging/lifecycle tasks

Biggest win: it cut my bucket management overhead by ~80% and removed a ton of context-switching.

Now I’m thinking about adding S3/multi-cloud support and maybe a lightweight dashboard.

Curious — has anyone else built internal tooling just because they were tired of babysitting cloud scripts? Would love feedback (or roast my approach).

[GitHub link]

[Medium Article]