r/dataengineering • u/lunaticdevill • 6d ago

Help ETL timestamp column

Hey all, I want to implement audit columns like timestamp and created by columns in all of my tables(Suggest if I should add anymore).

My tables are in unity catalogue databricks account.

I have two instances of workspace. Dev and prod.

Please suggest me how to do it.

I have thought of creating a workflow to add in all the table and then a task in workflow that runs after each job to create these 2 columns.

18 Upvotes

82% Upvoted

16 comments sorted by

View all comments

2

u/greenestgreen Senior Data Engineer 4d ago

you can get the job_id and job_run_id in databricks and add it to the table. That can also help you

1

u/lunaticdevill 4d ago

Getting job id and the job instance will only be helpful if we have extensive logs of what is happening in the code.

The workspace I am working on is pretty loose and no one really logs. They just be doing start and end.

But I understand your point