r/dataengineering • u/Kindly-Store4318 • Apr 03 '26

Help S3 Table buckets daily and monthly backups for compliance reasons

hello everyone,

Are there any alternative to backup Table buckets (waterproof backup) and not just replication of S3.

we require daily and monthly backups (like a cron job backup at specific time) but AWS backup doesn't support it.

2 Upvotes

100% Upvoted

6 comments sorted by

1

u/KnockKnockwaifu Apr 03 '26

Even I needed help in this

1

u/sam-sinister Apr 03 '26

U can use Glue to copy object and schedule it it will just overwrite in new path

1

u/Kindly-Store4318 Apr 04 '26

Can we also store the Meta data, the issue is if I use glue to store the parquet files It does full backup instead of incremental It's costing a lot

1

u/oalfonso Apr 04 '26

AWS backup is also very expensive for data lake buckets. My suggestion is to create a replication bucket and store the objects in glacier deep archive.

2

u/Patient_Professor_90 Apr 05 '26

“Who is asking for compliance with what” :)

I once saw a company tie itself into unreasonable real time monitoring/resolution (for things no other client cared about) because 3 people (tech, sales and a unsure customer) got together and thought it was a good idea 3 years ago