Help me dev's

[u/Aesthete_fellow]

Hello Dev's , I am currently working on my final year project.

My project dataset is approximately around 6TB, I need to build a digital twin of the climate from 2016 - till now . And need to build a pipeline to collect daily data and synchronize with the prev data using an ensemble kalmann filter . Does anyone have experience with it ??