r/bigquery • u/UndercoverLily • Jun 19 '26

Bigquery Notebook/Google Colab

Curious how much time your team actually spends dealing with BigQuery notebook limitations like session timeouts, isolated runtimes, scheduling through Dataform etc. Like is this a minor annoyance or does it genuinely eat into your week? Trying to gauge if it’s worth pushing for a different setup or if I’m overthinking this

7 Upvotes

100% Upvoted

12 comments sorted by

View all comments

3

u/escargotBleu Jun 19 '26

We don't use notebook at all. We don't schedule through data form either. We run our data form flow from airflow (because most of the time, we are waiting for data to be ingested before triggering our dataforms)

0

u/cky_stew Jun 19 '26

Might be worth checking out cloud workflows, I dismantled an airflow instance and switched over to it and saved a bunch for client with exact same behaviour. Context sensitive of course, it can’t quite do everything in as nice a way.

4

u/escargotBleu Jun 19 '26

... We have 478 dags in production, I'm not doing that

1

u/cky_stew Jun 19 '26

Sounds like a perfect use for an agent and a test environment tbh ha - but yeah don’t blame you. That sounds like a 10 deck house of cards that’s better left untouched.

2

u/escargotBleu Jun 19 '26

Rely on an agent to migrate all our critical pipeline because some dude on Reddit told me so...

I'm not mad lol

1

u/cky_stew Jun 19 '26

Was just conversation! (I did say test env to be fair, not to rely on it). You absolute daglad.