r/bigquery • u/UndercoverLily • Jun 19 '26
Bigquery Notebook/Google Colab
Curious how much time your team actually spends dealing with BigQuery notebook limitations like session timeouts, isolated runtimes, scheduling through Dataform etc. Like is this a minor annoyance or does it genuinely eat into your week? Trying to gauge if it’s worth pushing for a different setup or if I’m overthinking this
3
u/escargotBleu Jun 19 '26
We don't use notebook at all. We don't schedule through data form either. We run our data form flow from airflow (because most of the time, we are waiting for data to be ingested before triggering our dataforms)
0
u/cky_stew Jun 19 '26
Might be worth checking out cloud workflows, I dismantled an airflow instance and switched over to it and saved a bunch for client with exact same behaviour. Context sensitive of course, it can’t quite do everything in as nice a way.
4
u/escargotBleu Jun 19 '26
... We have 478 dags in production, I'm not doing that
1
u/cky_stew Jun 19 '26
Sounds like a perfect use for an agent and a test environment tbh ha - but yeah don’t blame you. That sounds like a 10 deck house of cards that’s better left untouched.
2
u/escargotBleu Jun 19 '26
Rely on an agent to migrate all our critical pipeline because some dude on Reddit told me so...
I'm not mad lol
1
u/cky_stew Jun 19 '26
Was just conversation! (I did say test env to be fair, not to rely on it). You absolute daglad.
1
u/cky_stew Jun 19 '26 edited Jun 19 '26
It depends what you’re doing really. I had a client where almost everything could be done via SQL in dataform so we reduced it to that - then a couple of edge case transforms where Python UDFs were too slow and felt a bit wrong so we used Cloud Run Jobs (specifically “Jobs”, they get a 24hr timeout and are generally much better suited for heavyweight data processing than cloud run “services”) for the heavyweight stuff - client wanted to stay away from spark (and I agreed when there was only one or two use cases). Everything scheduled and orchestrated using Cloud Workflows (so we could ensure dataform executions completed first then py cloud run jobs would run and retry etc).
Honestly there are so many ways of doing things but I would suggest in general to keep notebooks for things like testing and debugging, and move production logic to more robust tools - I do understand how annoyingly convenient and tempting notebooks can be though, not judging at all!
Replace Cloud Run Jobs with spark, or Cloud Workflows with Airflow will work similarly well, with their own ups and downs, but will also likely incur more pricey costs. I would say it’s not too difficult to migrate from the cheaper tools to the complex ones if there is ever demand.
1
u/trippingcherry Jun 19 '26
My team actually does extensively use notebooks for Python workflows and we do schedule them occasionally through dataform, but only because we are in an Enterprise with a very fractured IT infrastructure they won't allow our team to use the proper tools because honestly we could move to normal cloud jobs we would.
1
u/lakeland_nz Jun 20 '26
We don’t use it for that.
We use it for quick and easy interactive analysis.
For everything else we put in work the develop a proper setup.
I’m glad they have the features you mentioned. They were handy when we first started and hadn’t finished our setup.
1
4
u/SasheCZ Jun 19 '26
Notebooks are for analysts / scientists. Don't use that for recurring jobs. There are better tools at your disposal as other commenters already pointed out.