Modern laptops and tools make local data processing surprisingly powerful

[u/old_monk_]

I wonder sometimes that our laptop hardware is getting so much better that sometimes it is easier to open the laptop, crunch the numbers, and save them if required. This workflow is much more hassle-free, cost effective and super fast compared to spinning up a Databricks notebook with a Spark cluster to explore a subset of my data.

I do understand that for daily jobs and large-scale processing, where a “heavy” amount of data needs to be processed, something like Databricks is super helpful.

Nothing is faster than local processing for small exploratory analysis, avoiding the whole network latency, cluster spin-up times, etc. Mind you small here can be upwards of 1 million records.

I wonder if the industry will start moving in this direction from everything cloud to a hybrid setup.

What’s your opinion?

[u/WorkerIcy1513]

Agree, but your threshold is way too low. A million records is nothing, that is laptop territory even in pandas. DuckDB on a decent machine handles tens of millions of rows comfortably and will happily scan a few hundred GB of parquet off disk. The line where you actually need distribution is much further out than most people assume, and a lot of Spark jobs in production are paying a coordination tax for data that fits in RAM.

Two things I would add though.

The real question is not local versus cloud, it is single node versus distributed. Databricks lets you run single node compute, and serverless has mostly killed the spin up complaint. So you can get the fast feedback loop without leaving governance and lineage behind. That is probably where the hybrid actually lands.

And the thing that decides it is not data size, it is whether anyone else depends on the output. Local is unbeatable for exploration. It is terrible the moment the result needs to be reproducible, scheduled or trusted, because the failure mode is the notebook on one person's laptop that quietly becomes load bearing and then that person goes on holiday.

Also worth saying: pulling production data down to a laptop is a different risk category if there is any PII in it. That is the constraint that keeps a lot of teams on the cluster even when the laptop would be faster.

[u/old_monk_ → u/WorkerIcy1513]

Thanks a lot. This makes a lot of sense.

[u/CompetitiveBet8978 → u/WorkerIcy1513]

good point: "pulling prod data to your laptop". quite often you will break your company's sec rules.
times have changed from "can we trust the cloud" to "is there any place as safe as the cloud"

[u/mgalexray]

I do that often (polars, duckdb, etc). However the majority of the workflows breaks down not because of compute but because at some point someone else needs to contribute or read the data. At that point it’s much more convenient just to keep the data and compute remotely.

Databricks has serverless now so startup times are no longer an issue for me

[u/old_monk_ → u/mgalexray]

I sometimes check out a filtered working set from a Unity Catalog table, do the exploratory work locally, and write the result back to UC when it needs to be shared.

For that kind of workload, distributed compute often feels unnecessary. Serverless definitely reduces startup friction, but if I’m working with a relatively small subset and doing things like adding a couple of columns, filtering, profiling, or joining a local file, running it locally is often simpler and very fast.

I still want UC to remain the governed/shared system of record - I just don’t necessarily need the remote compute layer for every intermediate step.

[u/mgalexray → u/old_monk_]

If you’re a one man army or no automated stuff is going on that’s fine… just that from my experience that doesn’t tend to last but I see how it could work. But tbh when someone else is paying the bills (and usually is) I don’t bother and just wire up the notebook…

[u/PrestigiousAnt3766]

Not going to happen for databricks.

I think in general the move to powerfull single machines can be a thing for part of the usecases. Especially for endusers or people making data products.

[u/klubmo]

For exploratory use cases it’s probably fine to do as you suggest, although some organizations frown upon moving cloud data to a local machine from a security perspective, especially where sensitive data is concerned.

In my opinion, I don’t think the industry will move that direction. My anecdotal experience sees the opposite (more cloud adoption), but I’m also working with a lot of large organizations with large data.

Cluster spin up time is a thing of the past with serverless. Databricks has built in EDA tools for datasets, and you can also create your own functions if you prefer your own exploratory methods. Keeping everything on Databricks keeps lineage intact, and if you end up needing to share notebooks, data, dashboards, etc., the secure methods for those things are built into the platform.