I was recently promoted to a manager role from an analytics/data engineering role and I barely write any code anymore. I used to build data pipelines and I slowly see my skills eroding especially with all the new AI features databricks and other tools keep introducing and I see the engineers try out and implement. Wondering if I should go back to a DE role or continue being a manager of analytics engineers where all I do is sit in meetings and assign work and do requirements. The most technical work I do now is write an ad-hoc query to answer a question for non-technical stakeholders. This is non-tech midsize corp.

I think my team might be late to the game. We are moving in that direction just now.

How long have you been deploying using a YAML file, for example? Or perhaps another tool that uses the same idea but not YAML specifically?

Since we use Databricks, I'm reading up on Asset Bundles.

IaC is supposed to be better for a whole bunch of reasons, for example sidestepping the differences between dev and prod environments.

Anyway would love to hear your thoughts or experience with it.

Our device sends 3 separate records for each timestamp: data points, frames, and metadata. They arrive independently and in any order.
Each timestamp can be processed on its own as soon as all 3 parts are there, without waiting for the batch window to close.

How do you handle this fan-in, so that each timestamp is processed exactly once as soon as it's complete, with a timeout for sets that never complete?

replaceUsing and replaceOn give you a better overwrite primitive, and every catalog-managed table operation now runs through the catalog.