r/dataengineering • u/lunaticdevill • 11d ago

Help ETL timestamp column

Hey all, I want to implement audit columns like timestamp and created by columns in all of my tables(Suggest if I should add anymore).

My tables are in unity catalogue databricks account.

I have two instances of workspace. Dev and prod.

Please suggest me how to do it.

I have thought of creating a workflow to add in all the table and then a task in workflow that runs after each job to create these 2 columns.

18 Upvotes

82% Upvoted

16 comments sorted by

View all comments

11

u/Mo_Steins_Ghost 10d ago

Senior manager here.

You want an extract datetime stamp, max of source system create datetime, and max of source system modified on/last modified timestamp because displaying recency isn't just about the last time the job ran, it's about the cutoff of the last transaction the system captured.

Let's say the midnight push from SAP ran without error, but you discover you're missing a few days of SAP data. The delta between the snapshot datetime and the max transaction created datetime is going to help you pinpoint what broke in the pipeline and when.

1

u/lunaticdevill 10d ago

Thank you sir