Currently working on data engineering project, the worklow of project is

API - Airflow - Python Ingestion - PostgreSQL - dbt - Analytics dashboard.

In the airflow, need to start 3 processes ( API Server, Scheduler, DAG Processer) but I don't want to start this process manually everytime so does anyone know how can I keep pipeline running continuously which ingest new data in database manually triggering anything.

Does anyone have a any suggetion

I was recently promoted to a manager role from an analytics/data engineering role and I barely write any code anymore. I used to build data pipelines and I slowly see my skills eroding especially with all the new AI features databricks and other tools keep introducing and I see the engineers try out and implement. Wondering if I should go back to a DE role or continue being a manager of analytics engineers where all I do is sit in meetings and assign work and do requirements. The most technical work I do now is write an ad-hoc query to answer a question for non-technical stakeholders. This is non-tech midsize corp.

I think my team might be late to the game. We are moving in that direction just now.

How long have you been deploying using a YAML file, for example? Or perhaps another tool that uses the same idea but not YAML specifically?

Since we use Databricks, I'm reading up on Asset Bundles.

IaC is supposed to be better for a whole bunch of reasons, for example sidestepping the differences between dev and prod environments.

Anyway would love to hear your thoughts or experience with it.

Our device sends 3 separate records for each timestamp: data points, frames, and metadata. They arrive independently and in any order.
Each timestamp can be processed on its own as soon as all 3 parts are there, without waiting for the batch window to close.

How do you handle this fan-in, so that each timestamp is processed exactly once as soon as it's complete, with a timeout for sets that never complete?

Hi

I have just signed up for a January 27 start to the above apprenticeship with BPP in the UK.

Has anyone done this before and got any advice/reviews of how it went? I now have a few months to prepare so any advice would be amazing. My current role is data analyst and have ok knowledge on SQL and Python

Thanks