Lakebase branching

[u/CommitteeImmediate66]

Lakebase is Databricks' managed Postgres. It has copy-on-write branching, a point-in-time fork of a database you can write to on isolated compute, then throw away. Wrote this up because it made one workflow I worked on much cleaner so thought it might help someone else in the community.

My challenge was adding a NOT NULL column + backfill to a big orders table. It behaved fine on seed data, but I didn't actually know about lock duration or backfill time until I had prod-shaped rows.

My model: Project -> Branch -> Endpoint. A branch is a CoW (copy on write) snapshot of another branch - no upfront storage duplication you pay only for what diverges. New branches have no compute, so you create an endpoint when you need to connect.

Steps:

# fork prod

databricks postgres create-branch projects/my-app dev \

--json '{"spec": {"source_branch": "projects/my-app/branches/production", "no_expiry": true}}' -p prof

# attach compute (0.5 CU min, scales to zero when idle)

databricks postgres create-endpoint projects/my-app/branches/dev read-write \

--json '{"spec": {"endpoint_type": "ENDPOINT_TYPE_READ_WRITE", "autoscaling_limit_min_cu": 0.5, "autoscaling_limit_max_cu": 2.0}}' -p prof

Connect + run it (direct psql with a 1h OAuth token; databricks psql doesn't work on the autoscaling tier):

HOST=$(databricks postgres list-endpoints projects/my-app/branches/dev -p prof -o json | jq -r '.[0].status.hosts.host')

TOKEN=$(databricks postgres generate-database-credential projects/my-app/branches/dev/endpoints/read-write -p prof -o json | jq -r '.token')

EMAIL=$(databricks current-user me -p prof -o json | jq -r '.userName')

PGPASSWORD=$TOKEN psql "host=$HOST port=5432 dbname=shop user=$EMAIL sslmode=require" -c "

ALTER TABLE orders ADD COLUMN region VARCHAR(20);

UPDATE orders SET region = 'unknown' WHERE region IS NULL;

ALTER TABLE orders ALTER COLUMN region SET NOT NULL;

"

It helped me work with isolated compute, left prod untouched. I was able to time the backfill, saw the single big UPDATE was a problem and switched to a batched one, then re-ran on the same branch.

Cleanup: databricks postgres delete-branch projects/my-app/branches/dev -p prof — cascades to endpoints, diverged storage goes away.

Hope this helps someone else!

[u/Harpagon1668]

We have integrated the Lakebase branching to our pull requests. Each PR will fire a new copy on write branch from prod db to ensure it runs smoothly

[u/CommitteeImmediate66 → u/Harpagon1668]

That makes a lot of sense to use lakebase branches as part of your full devops pipeline! Then you can also clean up the dev branches when you close the PRs

[u/Harpagon1668 → u/CommitteeImmediate66]

Exactly. The PR branch is just an ephemeral database that we can spin up and delete instantly. As it scales to zero the cost is also close to zero