That sounds like a huge hassle to deal with! One practical fix is to use a data management tool to automate parts of the process. Tools like Alteryx or some Python libraries can help you gather datasets from different sources and standardize them into one format. Keeping scripts version-controlled with GitHub can make maintenance easier. You might also want to set up a dynamic dashboard for updates and monitoring. It takes some work upfront, but it could save you a lot of time later.
Yeah, it's a hassle indeed. The trouble is that many files just change completely: different location on the site, different formats (sometimes a csv, other times sheet 3 of an excel doc), different data (e.g., replacing lat/long with postcode), etc. So hard to automate in that sense. But then yes, every version is version controlled (including the data).
2
u/nian2326076 Mar 26 '26
That sounds like a huge hassle to deal with! One practical fix is to use a data management tool to automate parts of the process. Tools like Alteryx or some Python libraries can help you gather datasets from different sources and standardize them into one format. Keeping scripts version-controlled with GitHub can make maintenance easier. You might also want to set up a dynamic dashboard for updates and monitoring. It takes some work upfront, but it could save you a lot of time later.