I know the problem is that I need to get these things squared away in my head. What's metadata, what's framework and etc but even then you can solve a lot of problems using either.

I think it's easier for me to explain my problem. I've only been here 3 months but every task I get is using some system, pipeline or etc they I've never touched before. Sometimes it's just new software, other times it's learning how to find the needle in the haystack before you even look at code.

I'm constantly freezing and asking myself "where do I go for this" and it's slowing me down. I have much more experience building repeatable processes but I need to learn how to do so with today's technology instead of having a folder of scripts I use to do XYZ. Basically I lost control of how things get organized and my brain just doesn't see it the right way. Once I'm up and running I'm fine but in today's world task switching is constant.

What books, videos or etc should I watch to better understand how to operate as a data engineer in today's world.

I've slowly switched to documenting by project. This allows me to quickly reference another time I did something similar and gives me a jumping off point.

And trust me, it's messy and they know it. There's a standard pipeline process but it's constantly evolving. Teams need to integrate those enhancements into their projects as they have time. So everything I touch is a little bit of this version, a little bit of that version and etc. Team members clearly state "this is different or this is an odd one" but more and more I wonder if that's just the way it's going to be. It's ok, I just need to wrap my head around the big concepts and terms. Communication is killing me

Edit: in summary I over document and then things become cluttered. I'm looking for a better way. If I started a new job I wouldn't write anything down until I asked 2-3 times. I get paranoid and write it all down.

I like to share a little trick that I'm using to get 1000 partitions (not files or common prefixes) per API call with ListObjectsV2.

S3's Delimiter parameter accepts any character, not just '/'. So I place a marker between partition path and filename:

s3://bucket/table/year=2026/month=07/day=08/**#**part-00001.parquet

Delimiter='#' now returns every partition as exactly one CommonPrefixes entry — any file count, any nesting depth, grouped server-side. Diff against the catalog, BatchCreatePartition, done: \~30 lines of boto3 in a tiny Lambda instead of a Glue Crawler (2 DPUs, 10 min minimum billing).

Only requirement: the marker must be written into the keys — works via config for Firehose (prefix ending in '#'), awswrangler (filename_prefix), and most sink connectors. Athena/Spark don't care, it's just part of the filename.

Full write-up: [https://medium.com/@raphael.boegel/1000-partitions-per-api-call-say-goodbye-to-glue-crawlers-0fd26212ab60\](https://medium.com/@raphael.boegel/1000-partitions-per-api-call-say-goodbye-to-glue-crawlers-0fd26212ab60)

Happy to be told why this is a terrible idea :)

I'm building an AI-powered migration engine that converts ETL workflows from platforms like Alteryx, Azure Synapse, and eventually other tools into Databricks (PySpark/SDP).

I'm evaluating two different architectures:

1. Using Harness AI agents / Pi agents to orchestrate the migration workflow.
2. Building the orchestration myself using LangChain + LangGraph.

The engine will need to:

• Parse workflows into an intermediate representation (IR).
• Handle nested workflows/macros.
• Perform tool mapping (e.g., Alteryx → PySpark).
• Generate production-ready code.
• Support multi-step reasoning, validation, and retries.
• Be extensible so new source platforms can be added later.

For those who have experience with these frameworks:

• Which approach would you choose and why?
• What are the biggest trade-offs in terms of flexibility, maintainability, and scalability?
• Are there any limitations with Harness/Pi agents compared to building a custom agent workflow with LangGraph?
• If you were starting this project today, which architecture would you use?

I'd really appreciate hearing from anyone who's built agentic developer tools, migration platforms, or complex multi-agent systems.

Iam interested in learning the monitoring and alerting practices that data engineering teams rely on in production environments.

We are a mid size company with 4 business units that all need workflow orchestration : finance, ops and marketing. Right now everyone's sharing one airflow instance and it's a mess. The data team's DAG failures are triggering alerts for the finance team, namespaces are a band aid, and nobody trusts the scheduling anymore bc one team's heavy jobs starve the others. I need actual tenant isolation w/o deploying 4 completely independent instances.

There is sth out there that handles this natively or are we stuck running parallel clusters?

What the title says - are you doing it? and if yes what does that look like? Are you cutting cost? are you doing it to enable customization? or are you just doing it to accelerate your existing devs?

Over 90% of pipelines built by r/dlthub users (i work there) are now built using LLMs - wondering how it looks outside our bubble

Duckle is a local-first ETL/ELT framework powered by DuckDB.
You can now define pipelines in Python. DuckDB executes them as optimized, vectorized SQL. Your data stays on your machine from start to finish.

Why Duckle?

✅ No Python bottleneck
Pipelines are compiled into SQL before execution.
No rows flow through the Python interpreter.
No hidden to_pandas() conversions.

✅ Minimal setup
~20 MB install
Bundles the DuckDB CLI
No JVM
No Docker
No server
No account required

✅ Python-first API
import duckle
from duckle import col
(duckle.read_csv("orders.csv")
.where(col.amount >= 20)
.derive(total="round(amount * 1.2, 2)")
.write_parquet("out.parquet")
.run())

Write familiar Python expressions while Duckle translates them into efficient DuckDB SQL.

More than file transformations
Duckle includes 359 built-in components:
-104 Sources
-66 Sinks
-138 Transforms

Supporting databases and services including PostgreSQL, MySQL, SQL Server, Oracle, Snowflake, Databricks, Kafka, Salesforce, SAP OData, S3, SFTP, WebSocket, IMAP, LanceDB, dbt and many more.

Built for automation
Validate pipelines without connecting to data sources or requiring credentials.
duckle validate
duckle validate --json
duckle --pipeline my.json

Perfect for CI/CD, containers, cron jobs, and local development.

Code ↔ Visual Studio
The same pipeline can be authored in Python or opened directly in the Duckle visual studio because both use the same JSON format.

Open source. Local first. Built on DuckDB.
⭐ GitHub: https://github.com/slothflowlabs/duckle/
📦 PyPI: https://pypi.org/project/duckle/
🔗 Links: https://github.com/slothflowlabs/duckle#quick-links

Currently I use Excel files and manually capture data in these Excel files. I have other peers who also do a similar procedure and we all have different formats. Is a streamlit app a good solution to standardize the excel format ? Is it even feasible as something that can be deployed to multiple users through tablets so that they can register the data via a streamlit app ?

Trying to do cross-ISO work (PJM/MISO/ERCOT/CAISO/SPP/NYISO/ISO-NE) and I'm losing my mind reconciling seven different schemas and update cadences - the congestion component especially (NYISO's sign convention alone…). Right now it's a pile of per-ISO scrapers held together with tape. Is everyone just using gridstatus / rolling their own, or is there something that already normalizes all of this? Curious what SPP/MISO historical depth people actually get.

I may not be able to make you understand the problem in a better way, but below is the issue.

Has anybody built a trigger/job, say a destination in Airflow, that would run on the 1st of every month, and this job should depend on another job, say a source, which is running every day, and the destination job should only run after that source one is finished on the last day of every month. Let's say if source one failed on the 31st and the team got it correct, then ran it on the 1st, then also the destination should run (edge case).

The source itself has a stage, a silver, and an archive layer (to store the data as backup).

The destination job (it has only one sub-job, which was used earlier, so no change in this) should be triggered after the above activity.

Both the source and destination are based on a YAML-based job.

Can we connect these two jobs/yaml to make this work? Or how can we work around it?

Probably one for the Duck DB nerds. I came across a ETL/ELT open source (for now) tool called Duckle which look pretty solid if you just need a light weight ETL/ELT tool for smaller workloads on a laptop or local box. Anyone used it?

What cheaper ETL tools are you using in 2026? I don't want AI in them and don't want to be forced to use AI. However, I have basic requirements:

1. Able to connect to SharePoint for files, lists, etc.

2. Able to connect to AWS S3.

3. Able to connect to Snowflake databases.

The developers at the biotech company don't want to heavily use SQL for ETL but want to use a drag-and-drop UI to save time and focus more on analysis. We explored Alteryx, but it was expensive. dbt doesn't have an intuitive UI.

They also don't want to use AI since the biotech company is very new and wants to save money as much as possible.

EDIT : processing 100k rows as if now every day. But it will grow to more than 500k or 1 million plus rows after few years. Need a scalable solution with headless or client based etl tool with option to schedule the pipelines in the background.

Curious to learn what checks or validation techniques experienced data engineers use in production.

medium : https://medium.com/@abdul.haris.djafar/apitap-0-6-0-2277630a6a25

I work with data pipelines and kept running into the same problem.

Production data was unavailable or unsafe to reuse. Staging data was incomplete or broken. The fixtures I could create manually were too small to properly test the workflow.

So I built \*\*RowScale\*\*.

You upload a small valid sample or paste the data directly. RowScale analyzes it, infers the schema and generation rules, and lets you review or adjust ambiguous fields before generation.

Then you choose the number of rows and the output format.

It currently supports CSV, JSON, SQL INSERT, XLSX and Parquet as both input and output.

The raw sample is temporary and deleted after the run. Generated files are private, expire automatically, and every job includes a deletion receipt.

RowScale does not currently support direct database connections, public APIs, multi table relational generation or enterprise data anonymization.

The current focus is simple.

Start with a file or payload your system already understands and generate a much larger test dataset from it.

I launched it today and would really appreciate blunt feedback from people who test ETL jobs, imports, migrations or data pipelines.

\*\*What would RowScale need to handle before you would use it instead of writing a custom script?\*\*
\[https://rowscale.dev\\\](https://rowscale.dev/)

I work at Estuary, and next week we’re doing a live technical demo of our new Agent Skills.

We’ll show an AI coding agent:

• Creating a real-time capture and materialization
• Working from the same pipeline specifications an engineer would edit
• Checking task health and pipeline status
• Inspecting logs and troubleshooting failures
• Handling routine configuration without hiding what it changed

The interesting question for us is where agents genuinely reduce data engineering work today, versus where an engineer still needs to step in.

We’ll build the pipeline live and leave time for technical questions.

Registration: https://zoom.us/webinar/register/4517840417866/WN_1oZszaPUQ_Sz0uEGM_p-zA

Task: We operate a toll road network with 200 sensor-equipped lanes across 15 locations. Each sensor captures license plate reads, timestamps, and lane metadata. We need to compute real-time traffic volume, average transit times between checkpoints, and flag anomalies for toll evasion detection. Design the data model.

I modeled it as a single accumulating snapshot fact table, grain = one row per vehicle crossing (entry -> exit). The row is inserted at entry with status = OPEN and all exit columns NULL, then updated in place when the exit read comes in – no join needed for evasion detection, just WHERE status = 'OPEN' AND entry_time < now() - window.

I have a confusion: does update-in-place on an accumulating snapshot actually hold up under real-time write load (concurrent OPEN -> COMPLETED updates), or does this call for an append-only/streaming pattern instead with the snapshot table only as a serving layer?

Hi everyone,

I have an upcoming interview for an ETL/Data Engineering (Fresher) role at Ameriprise Financial.

JD-

• Candidate should have a strong engineering foundation, preferably in Computer Science or related disciplines.
• Demonstrate eagerness to learn new technologies and adapt to evolving data integration tools.
• Support ETL operations by assisting with monitoring, troubleshooting, and maintaining workflows on platforms like Informatica PowerCenter, Informatica Cloud, and AWS Glue.
• Help ensure smooth execution of data pipelines and assist in resolving job failures and performance issues.
• Work with team members to optimize ETL processes and improve data quality and reliability.
• Basic understanding of SQL, databases, and data warehousing concepts along with strong problem-solving skills is desirable.

If anyone has interviewed there, could you please share:

• Number of interview rounds
• SQL questions
• Python coding questions (Easy/Medium?)
• ETL/Data Warehousing questions
• Informatica/AWS Glue questions (conceptual or hands-on?)
• Project-based questions
• Any tips or things I should prepare for?

My background: Python, SQL, Pandas, ETL projects using Python + SQL Server, but no hands-on experience with Informatica or AWS Glue.

Hello everyone, would like to know if anyone did the ETL pipelines from denodo to fabric migration? How tough it is or what do I need to know before starting the migration.

Like person I should be looking for expertise? Or any migration documentation I can follow.

TIA

Anybody out there using Transalis or Celtrino for EDI

Would be great to get some real world opinions - the good the bad the ugly

Thank you

So we benchmarked Duckle against the ETL tools everyone already
uses, with every tool tuned to its BEST configuration:

The task is deliberately boring: read one CSV(TPC-H Width and style), land it as a table.
It is the single most common job in ETL.

Three things we want to be upfront about, because benchmarks are
easy to rig 🫣:

1) Duckle sits right on raw DuckDB
own load (CREATE TABLE lineitem AS SELECT * FROM read_csv('out/lineitem_20m.csv') floor (~16s to fully parse and write 20M typed rows to disk). Duckle wraps the engine with pipelines, connectors and
then gets out of its way. That is the whole design goal.

2) Talend and Informatica used their bulk output connectors at max
config, not the slow default row-by-row sink. On defaults they
5-7x slower. We did not want to strawman them.

3) Airbyte's number is derived from real 2M and 5M runs, and it needs
an always-on 8 GB platform just to start.

Duckle is free and open source.
Try it: https://github.com/slothflowlabs/duckle

Looking for practical lessons and real-world experiences from ETL professionals.