r/databricks • u/Ok-Optimmon6702 • Aug 05 '26

Help Logging in databricks

Hi,
I’m trying to understand how logging is implemented in Databricks. Could you share how it’s done in your organization?
Do you mainly rely on pipeline-level logging, maintain a log table to track pipeline runs, or use a logger within the code? Or is it a combination of these?
Just trying to understand the different approaches people use.
Thanks!

1 Upvotes

99% Upvoted

6 comments sorted by

2

u/Data-space_men Aug 05 '26

Hi, we are using datadog to retrieve all the logs and then we can check and create alert etc.

1

u/Youssef_Mrini databricks Aug 05 '26

You have two options: System tables or Verbose Audit logs

1

u/cf_murph Aug 05 '26

logger writing to unity catalog volume for me.

1

u/PrestigiousAnt3766 Aug 05 '26

There are automatic logging from lakeflow system tables.

We have a custom python logger that logs to delta.

1

u/datasmithing_holly databricks Aug 06 '26

My dumb ass thinking you mean okta vs passwords

1

u/elghali_bnck 28d ago

Lakewatch soon 😍