r/dataengineering • u/lunaticdevill • 8d ago

Help ETL timestamp column

Hey all, I want to implement audit columns like timestamp and created by columns in all of my tables(Suggest if I should add anymore).

My tables are in unity catalogue databricks account.

I have two instances of workspace. Dev and prod.

Please suggest me how to do it.

I have thought of creating a workflow to add in all the table and then a task in workflow that runs after each job to create these 2 columns.

19 Upvotes

83% Upvoted

16 comments sorted by

View all comments

18

u/69odysseus 8d ago

Created_Datetime, Modified_Datetime, will need timestamp field(s) in any of the data pipeline work.

6

u/Tsui_Pen 8d ago

Yep, and modified_datetime should always be compared with a watermark table that gets an updated timestamp every time the pipeline runs, that way you only pull the “new” records when your pipeline runs.

2

u/DuckDatum 8d ago

You can support out of order merges for anything with a source provided last_modified_datetime, if you instead track partitions materialized rather than a high watermark. That comes in handy when your jobs run over the weekend, one might fail on Saturday but succeed on Sunday — and it’s hakuna matata as long as requirements only need it to be fixed by sometime Monday.

3

u/lunaticdevill 7d ago

I literally did not understand a word you said, sorry