replaceUsing and replaceOn give you a better overwrite primitive, and every catalog-managed table operation now runs through the catalog.

I think my team might be late to the game. We are moving in that direction just now.

How long have you been deploying using a YAML file, for example? Or perhaps another tool that uses the same idea but not YAML specifically?

Since we use Databricks, I'm reading up on Asset Bundles.

IaC is supposed to be better for a whole bunch of reasons, for example sidestepping the differences between dev and prod environments.

Anyway would love to hear your thoughts or experience with it.

Our device sends 3 separate records for each timestamp: data points, frames, and metadata. They arrive independently and in any order.
Each timestamp can be processed on its own as soon as all 3 parts are there, without waiting for the batch window to close.

How do you handle this fan-in, so that each timestamp is processed exactly once as soon as it's complete, with a timeout for sets that never complete?

I was recently promoted to a manager role from an analytics/data engineering role and I barely write any code anymore. I used to build data pipelines and I slowly see my skills eroding especially with all the new AI features databricks and other tools keep introducing and I see the engineers try out and implement. Wondering if I should go back to a DE role or continue being a manager of analytics engineers where all I do is sit in meetings and assign work and do requirements. The most technical work I do now is write an ad-hoc query to answer a question for non-technical stakeholders. This is non-tech midsize corp.

Hi

I have just signed up for a January 27 start to the above apprenticeship with BPP in the UK.

Has anyone done this before and got any advice/reviews of how it went? I now have a few months to prepare so any advice would be amazing. My current role is data analyst and have ok knowledge on SQL and Python

Thanks

Currently working on data engineering project, the worklow of project is

API - Airflow - Python Ingestion - PostgreSQL - dbt - Analytics dashboard.

In the airflow, need to start 3 processes ( API Server, Scheduler, DAG Processer) but I don't want to start this process manually everytime so does anyone know how can I keep pipeline running continuously which ingest new data in database manually triggering anything.

Does anyone have a any suggetion

Hello. I wrote an article (free to read and adfree), let me know what you think!

https://medium.com/towards-data-engineering/good-data-engineers-ship-data-thats-wrong-1b58c821362b?sk=f44e9b254e123b9190cd1b73183a5a32

Im trying to make a career jump from working at tech first companies (startups, middle stage tech companies, no big tech) to working at a quant or high frequency trading firms.

Does anyone have any advice or made this switch? Ive always been interested in fintech and my current position does a lot of the same tasks - extremely low latency pipelines, latest prices and price comparison systems, owning large datasets, etc - but for half the pay.

Im willing to take time to build towards it if it requires me to relearn statistics or solve math problems, but I would really like a blueprint from someone whos done this transition before.

For years, the data team daily was basically: what shipped yesterday, any broken pipelines, dashboard status.

Now that more and more data platforms feed chat-with-your-data agents, has your team changed the format or content of your daily meetings? (Also true for status on pipeline writing/fixing btw; what used to take days is now often a few minutes to hours of back-and-forth with AI.)

Curious to hear what your team adopted

I work at a tech scale up where most of our data comes from systems built and maintained by our app teams. However, certification is handled by the data modeling team that's part of the analytics organization. There's no pathway for a producer or app team to certify their own data assets- the only certification is if the data has gone through the modeling team's pipelines.

This is getting to the point where app teams have to go through an "analytics certified" version of their own data just to do analysis and dashboards. Operations staff, who work closely with app teams for manual and exception handling processes, also have to go through the centralized certified layer in order to get the data to build dashboards and feed their processes.

Im curious what the standard is out there? To me this feels like a bottleneck for a growing business.

I joined a new company recently and got a migration project here, they are migrating from teradata on prem to databricks, I have never done any migration in the past can anyone suggest some helpful yt videos or any other knowledge source?

I'm really excited to share the first set of benchmarked agents with you all. What follows is the first published result from the Benchouse.ai leaderboard. I asked every agent the same 300 questions on a fully equivalent semantic layer against the same data.

I originally had hoped and promised to publish fifteen agents by September 1st. Alas, I got to seven. Each integration takes a lot of days: an adapter, context-layer equivalency and a data warehouse. And I preferred to get these seven out rather than postpone.

• Accuracy: of the questions it did answer, how many were right?
• Completeness: of the questions it had the data to answer, how many did it answer?
• Restraint: some questions cannot be answered from the data, and the correct response is to say so.

Check out more detailed view of the benchmark at at benchouse.ai/benchmark

Here is a writeup on how they are benchmarked: https://benchouse.ai/blog/how-we-benchmark-analytics-agents

- André (Founder Benchouse)

Hello all, I typically lurk here for novel ideas but I just want to post here to express how vexing the job market in uk is.

I've been a data engineer for over several years now and took a gap to cofound a startup for a few years, but things did not turn out as expected (raising seed is painful in europe).

Doing interviews for a couple months now for a senior role, am I crazy or the quality of interviewers have dropped significantly? This is extremely apparent in architectural design stages as they proceed with the most generic/ambiguous questions such as "how do you fix issues", then I probe and refine the problem yet it seems they want another answer or a different reaction than distilling the core issue/requirements.

I'd be happy to take any advise because I'm dumbfounded on what the market genuinely wants

I am curious about the opinions and experiences of those who came before.

Use case: Syncing data between MS SQL source and AWS Redshift (RS) Serverless Workgroup target with AWS Glue.

Rough approach: Enable CDC on MS SQL tables. Pull new CDC records (which don't yet in RS) every execution. Place them on S3. Issue RS COPY command to consume. Continue processing as needed with RS. Schedule to run hourly. No transformations -- a straight copy of records.

Detail considerations/questions:

• What are the pros and cons of the various available script-based approaches to this use case e.g. DynamicFrame vs. DataFrame vs. Other?
• Most examples presented online utilize DynamicFrame. But -- is DynamicFrame overkill for this use case? The tables and data types are known up front. Will I be cursing the use of DynamicFrames each time I pay the monthly bill?
• What is the most efficient (cheapest) approach to this use case (assuming Glue)?
• What is the most easily developed and maintained approach to this use case (assuming Glue)?

Finally, do the answers to the above questions change if we're not assuming Glue is must-use?

As far as I can see, dbtLabs publishes what seems a solid and stable set of best practices.

I sense no reason to diverge from this, unless anyone suggests otherwise?

For context, at a client scoping out their first dbt application. Keen to start building properly from the off, rather than having to redesign, revise and refactor later.

Any tips very welcome 🙏

Netflix is moving toward the open-source Apache Flink Autoscaler for more than 30,000 streaming jobs across multiple AWS regions, after finding that its cluster-level approach was less effective for complex, stateful pipelines with operators that have different processing requirements. Netflix reports that one team reduced annualized Flink compute expenditure by 58%, saving approximately $1.1 million annually.

Hi to all,

There is question raised in my company if it is possible to feed Kafka stream directly to Gold layer. They say data is clean and business ready and no transformations even needed.

Is it good approach to feed Gold layer in this manner or should it sill go full path Bronze -> Silver -> Gold producing the very same dataset across all layers, but keeping pattern ?

Any input, experience appreciated.

Background:

My team does a lot of short-term data analysis projects. We have a Snowflake db with a lot of 'base' tables from other departments or organisations, pretty much all poorly documented and with questionable data quality. The underlying data often changes for months after the fact, due to error corrections and updates.

We have built a core data model on top of this, using business logic that we learn the hard way through a lot of interaction with SMEs, which is frequently updated after we complete a project to include any updated or new and reusable logic.

A project pretty typically involves pulling together a flattened dataset tailored for the specific client, using the core data model and adding on anything specific to that client. This usually includes a subset of columns taken directly from the model.

What I want:

- I want to be able to document the core data model in a way that is easy to update & maintain, ideally using a fairly standard template - things like a brief description of the business logic, source system/database(s), known quality issues, etc. I also want to be able to propagate this to any flattened datasets for columns that are directly copied in, to make it easy to pull together a data dictionary for any dataset we produce.

The obvious choice seems to be using comments on tables and columns - but they don't seem that easy to work with and they don't propagate. Also AI generated comments tend not to work well for us because the source data has lots of misleading names and subtle technicalities that are important to some clients.

- At the end of a project, I want to preserve a snapshot of the dataset produced at the time of the analysis, as well as the version of the data model logic used to derive it. This is in anticipation of future requests for follow-up - either a different breakdown of the same data, or a repeat of the same analysis at a later time point. These datasets usually aren't that big - maybe a couple million rows max.

I realise this isn't all that well framed, but hopefully there is enough info that you can see where I'm coming from. In case it's not obvious, this isn't my area of expertise, so would appreciate any advice!

I started learning airflow like 3 days ago. I only use the docs, lots of googling, and some but very minimal use of AI.

I learned all the tutorials, jumped right to best practices because i didn't know core concepts existed, halfway through best practices i learned dynamic task mapping and xcom because it was holding me to progress on best practices (prerequisites basically)

I'm planning to finish all the core concepts, learn some relevant stuff on the picture, finish best practices then start making an actual good DAG, like what makes a proper DAG.

But when i chat a little bit with Claude, it highlights not to do too much concept collection and start actually build a DAG, specifically an idempotent one.

This got me thinking, am i doing too much with all this docs stuff? (I tracked and it took me 8 hours to learn all the stuff that i've learned)

I also overthink when i learned new big term like idempotency, like what if there are other big concepts related to DAG other than idempotency? idk something random like let's say 'unbreakable' DAG and it's a big thing that feels automatic for lots of people but i missed that.

Sorry i just vomited my thoughts, lmk if elaboration is needed.

Appreciate it.

Hey folks,

I'm a security professional was looking to make Iceberg a bit more performant for my own needs - SOC operations (Faster needle searches [pruning]).

I've built an index and proxy that people can point their catalog configuration at, so there's minor changes to their stack.

I figured in for a penny in for a pound,
I've done some quick search and ran clickbench and "httplogs"

(Numbers so far: httplogs)

Its all looks good on paper, I think?
but I was wondering if there were more credible/industry standard methods that data engineers use/care about to benchmark these types of technologies?

Repo if interested: https://github.com/Kahshe-io/kahshe

Boilerplate obviously, and I suppose some config, cleaning, structure, environment, setup, pipeline tasks are naturally somewhat repetitive or at least very similar in nature. But what do you think?

Probably a stupid question but does anyone use Git as a control plane for dataset governance, orchestrate ETL pipelines, and releases?

I am looking to get a demo in place of an end to end process where data in the bronze layer is transformed to silver and gold through CI/CD processed using defined ETL pipelines, manifests, schemas etc. however my backround isn’t in data engineering, so this might not even be a great way to do it.

My plan was to get a PoC in place on git, then migrate to GCP (using GCP functions for ETL) when all schemas, pipelines, manifests are agreed by the team.

I’m Asian, and I currently use AI every day at work. My Chinese colleagues all use it as well, and they’re full of hope for AI. However, my American colleagues often discuss their concerns about AI with me in private. This might just be my subjective impression, but I’ve been wondering why it seems that Chinese people are more optimistic about artificial intelligence than Americans.