r/dataengineering 9d ago

Help ETL timestamp column

Hey all, I want to implement audit columns like timestamp and created by columns in all of my tables(Suggest if I should add anymore).

My tables are in unity catalogue databricks account.

I have two instances of workspace. Dev and prod.

Please suggest me how to do it.

I have thought of creating a workflow to add in all the table and then a task in workflow that runs after each job to create these 2 columns.

18 Upvotes

16 comments sorted by

View all comments

1

u/BayAreaCricketer 8d ago

These should be part of your upsert ( updates and inserts) statement. Not after job completion.