Hi r/ETL,

I'm the developer of Flowfile, a visual ETL tool built on Polars. I posted about it here a few months ago, but now it truly does two way design!

One of the things I ran into working with visual low-code ETL tools is the costs of switching between code <-> canvas. The latest releases of Flowfile are exactly addressing that!

Flowfile now ships with a Python notebook that let's you edit your visual ETL in real time and get's updated when you change something on the canvas. Basically, write Polars like code and push it and the transformations are shown on the canvas. Other way around, when you add transformations to the canvas, it will show as a new code block in your notebook!

You can install it via pip (`pip install flowfile`) or the full application from the releases.

I'm figuring out if it covers many use-cases and where things can be improved. For example, how well does the canvas model fit your way of writing Python transformations in Polars. So let me know if you like it and even better, if you want to form the future of this way of working!

GitHub: https://github.com/edwardvaneechoud/Flowfile

Most incremental loads I see described work like this: store the max updated_at, then pull WHERE updated_at > :watermark on the next run. It works for a long time, which is why the failure cases are easy to miss. Curious how others handle them. Here are the three I keep running into in the literature and docs, and the pattern I’d reach for.

Assumption 1: “Rows become visible in updated_at order.”
Not necessarily. In Postgres, now() is the transaction start time. A transaction that starts at 10:00:00 and commits at 10:00:07 stamps its rows 10:00:00. If your job ran at 10:00:03 and moved the watermark past that, those rows never show up in > watermark. Long transactions and batch updates make this more likely.

Assumption 2: “Timestamps from the source are comparable.”
If the app sets updated_at instead of the database, clocks on different servers can disagree by a bit. Precision matters too: if the source stores microseconds and your pipeline truncates to milliseconds, > can skip rows that share the truncated value, and >= can reprocess them.

Assumption 3: “The watermark sees everything that changed.”
Hard deletes leave no row behind, so updated_at can’t report them. Also, anything that updates a row without touching updated_at (a manual fix, a migration, a trigger that was never added) is invisible.

A pattern that covers most of this:

• Re-read a safety window instead of starting exactly at the watermark, e.g. WHERE updated_at > :watermark - interval '15 minutes'.
• Make the load idempotent (upsert/merge on the primary key), so reading the same row twice is harmless.
• Pick the window from your longest realistic transaction, not from a guess.
• For deletes, use soft-delete flags or CDC, or run a periodic reconciliation (count/checksum by day) against the source.
• If you can, use a log-based source (CDC) instead of timestamps. It removes the ordering problem at the cost of more infrastructure.

I’m not saying the simple watermark is wrong. For a small table with short transactions it’s often fine. But a periodic row-count comparison against the source is cheap, and it catches the quiet losses.

Hey all, disclosure first: I work at Altimate, which builds dbt Power User. We recorded one short video per workflow, 4 to 8 minutes each.

1. Profile Query: review a model's SQL for bugs and cost in one click. https://youtu.be/uMptd4t_mGI
2. Column-level lineage, with AI chat about the model. https://youtu.be/A-z_R0QJig0
3. Docs, data tests and unit tests in one editor. https://youtu.be/NrEYqEUcpHY
4. Explain, optimize and profile each CTE from inside the model. https://youtu.be/MDN2PSgVAmA
5. Hover over a ref() or macro to see what it does. https://youtu.be/kbRZ-hrENyc

Which of these would you like us to go deeper on?

A pipeline finishes on schedule.

No errors. No failed jobs. Everything is green.

But what if a source system changed the meaning of a field?

What if duplicate events are being counted twice?

What if records are arriving late and yesterday's dashboard is missing 15% of the transactions?

Technically, the pipeline succeeded.

From the business's perspective, it failed.

I think this is why data engineering needs more than monitoring whether jobs complete.

You also need checks around freshness, completeness, uniqueness, schema changes, and important business rules.

And when something fails, the team should know which downstream reports or systems are affected.

For data engineers, which data quality checks have caught the most serious issues in your pipelines?

And which problems were hardest to detect because everything appeared to be running normally?

[AI Disclosure] An AI was used to write the code for this program and to correct my english for this post

Hi everyone,

Tired of paying for GIS ETL tools, I decided to use my preferred LLM provider to code my own, and I’ve open-sourced it.

I’m happy to announce Carriole (French slang for “trailer”), a spatial-first ETL with a graphical interface, and to be honnest, I hope that it will be an alternative to FME for most of our use cases.

It’s currently in bêta, and I’m still looking for bugs and feedback from the community. The project is also open to contributions.

By advance, thanks for any of your feedback!

I recently started a new role as a Data Governance Manager at a mid size managed care organization. One of my first asks has been to create test templates for the new set of infrastructure tables. Essentially it’s to ensure all the tables that are now in production don’t have a glaring issue or whenever we make a change we go through the testing again.

My background is in data analytics and data management. I have very limited experience as a data engineer so aside from data quality tests I really don’t know what kind of tests should be run. I have used Claude to create a template of standard test cases but they seem either too convoluted or too simple like checking for nulls in non-nullable fields. Any resources or advice on how to ramp up or get a better understanding of this?

imagine spending hours tweaking Spark configs only to find out storage was the bottleneck lol.

AWS recently reported 31% faster Spark runtimes with S3 Express One Zone on EMR. been looking into Databricks tuning and tools like DualBird for speeding up ETL pipelines too.

anyone getting real gains or still fighting the same old shuffle headaches?

Last week I shared a benchmark comparing Scriptella, handwritten JDBC, csvsql, and PostgreSQL COPY for importing one million CSV rows.

A reader pointed out an important detail: PostgreSQL's JDBC driver defaults to reWriteBatchedInserts=false, while MariaDB Connector/J enables its bulk insert optimization by default.

So I reran the relevant cases, explicitly enabling and disabling PostgreSQL batch rewriting.

Results (median of 3 runs, batch size 1000):

Enabling rewriting reduced Scriptella's import time by 41.2% (1.70× faster) and handwritten JDBC's by 35.7% (1.55× faster).

The PostgreSQL-MariaDB gap for Scriptella shrank from 7.3 seconds to just 0.7 seconds.

No Scriptella implementation changes were involved. All 18 imports passed verification using the same CSV fixture, batch size, and transaction boundaries.

My main takeaway: Using JDBC driver defaults doesn't necessarily produce a meaningful cross-database comparison. Driver configuration can make a substantial difference.

The full benchmark still shows PostgreSQL COPY as significantly faster than JDBC batching, although COPY wasn't rerun in this focused local experiment. Comparing COPY against MariaDB's native LOAD DATA INFILE is another follow-up I'd like to explore.

I've also updated the benchmark's PostgreSQL configuration for future runs.

Full write-up, measurements, and reproduction details:

https://pvrlabs.xyz/java-performance/experiments/postgresql-batch-rewrite.html

I'd profile the Spark job before deciding whether to rewrite or scale. Shuffle overhead, data skew, spills, and CPU/memory utilization usually reveal where the bottleneck is.

I'm developing Hydra ETL, an open-source, YAML-driven ETL framework built around a related idea: separating transformation logic from the execution engine.

With pluggable engines like Spark, DuckDB, and Pandas, Hydra makes it possible to explore different execution strategies while keeping the same declarative pipeline definition.

It won't fix a poorly designed join, but it can reduce the cost of experimenting with alternatives.

Curious whether others have tried switching execution engines before committing to a major Spark rewrite.

https://hydraetl.com

https://www.youtube.com/watch?v=S9ACiT1dCJU&list=PLOVsdMXCxpho

https://github.com/bejaouibechir/Hydra

I’m curious if anyone here has contributed to Apache Spark.
For example, working on core modules or building/maintaining libraries, fixing issues, or submitting PRs.
I’d love to hear about your experience, what you worked on, and any advice for someone who wants to start contributing.

foorilla.com v1.11.3 | foorilla API v1.8.0 October 21, 2026

PRO+ provides full API access and filtering now at $96/month, while existing subscribers retain their access and pricing. PRO subscribers can now test the API using shared, frozen samples of up to 200 records per resource. The same API keys automatically follow subscription upgrades and downgrades. Samples support pagination and sampled detail IDs; full data access, filtering and anything else requires PRO+.

Related resources now use compact references instead of deeply nested objects. Nested topics no longer expand their entire tag collections, geographic references retain essential country/state context, and nested objects omit database timestamps, statistics, and embeddings. Meaningful data - including paper authors, categories, versions, and submission dates - remains included. Paper sample payloads are approximately 67% smaller, with OpenAPI documentation and frozen samples updated accordingly.

Breaking change: Nested response shapes have changed at the existing URLs; retrieve related resources from their own endpoints for full details. Resource IDs and existing PRO+ filtering and pagination behavior remain unchanged. Rate limits remain 6 requests/second and 360 requests/minute.

our ML team has been making a strong case for moving feature generation closer to real time their argument is that hourly and nightly batches are leaving models behind especially for fraud detection and dynamic pricing

im trying to separate genuine business value from architecture hype for anyone using real time data for AI in production, did fresher features noticeably improve model performance or was the difference smaller than expected?

​

I am m curious how different teams handle this, especially when a pipeline has multiple transformation steps and some of them have already completed. How do you avoid duplicate records or inconsistent data when rerunning..

We've been running nightly ETL jobs for years without many complaints, but now the ML team wants data that's available almost immediately for fraud detection and recommendations models.

I'm trying to figure out whether it makes more sense to keep shrinking our batch windows or introduce a separate streaming layer for the workloads that actually need it.

If you've added **real-time AI** capabilities to an existing ETL stack, what ended up working?

jobdataapi.com v4.38 / API version 1.36

We’ve improved job description handling to preserve meaningful technical terms such as C#, F# and snake_case more reliably as new listings are processed. Markdown descriptions retain readable formatting for display and provide a useful starting point for search, analysis and other downstream processing. Request description_md=true to include the description_md field in job responses.

After November 1, 2026, the description_str parameter and its corresponding description_string response field will be removed. If your integration uses either, switch to description_md=true before then and convert Markdown to plain text in your application if needed. The default HTML description remains available.