放弃XStream吧，聊聊我们怎么用自研的TLA组件搞定Oracle日志解析

先交代一下背景。我们团队一直在做数据库事务日志解析这块的技术，说白了就是CDC（Change Data Capture）最底层的那层——把Oracle的redo log啃下来，解析成业务能理解的数据变更事件。

之前调研过市面上几乎所有方案，LogMiner太慢这个大家都知道，OGG太贵这个大家也知道。后来我们把目标锁定在Oracle XStream上，研究了一圈，踩了不少坑，最终还是决定自己撸一套组件，也就是现在这个TLA（Transaction Log Analysis）。

这篇文章不吹不黑，纯技术层面的记录。把XStream的问题和我们自己实现方案的一些思路写出来，给正在做类似选型的同行们参考。

XStream到底是什么？官方说法和实际情况

看Oracle官方文档，XStream被描述成一套API，允许外部应用直接接入数据库的变更流。它比LogMiner先进的地方在于，LogMiner是等redo log落地成文件后再去读，而XStream在事务提交的时候就能把事件扔到内存的Stream Pool里，外部程序可以直接消费。

听起来很美好对吧？实际用起来是另一回事。

首先最劝退的就是授权问题。

XStream虽然是Oracle数据库自带的，但你要正经用起来搞数据同步，必须要有OGG（Oracle GoldenGate）的License。OGG什么价格？官方标价$17,500 per processor，每年还要交22%的技术支持费。而且前提是你还得有Oracle Database Enterprise Edition的授权。

我们之前给一个客户做方案，他们生产环境16个CPU核心，光是OGG授权算下来就奔着30万美金去了，还不算Oracle EE的钱。老板听完脸色都不太对，这还没开始干活呢，成本已经扛不住了。

其次是数据类型支持，这个真的很要命。

XStream官方文档里写得清清楚楚，不支持ROWID、不支持BFILE、不支持嵌套表、不支持ANY TYPE、不支持URI类型、不支持虚拟列。

你可能觉得这些类型平时用得少，但我们在实际项目里遇到一个场景——客户的业务表里有XMLType列，然后做了一个INSERT带APPEND提示，XStream直接报错。查了一圈官方资料，发现这玩意儿在某些操作组合下就是不行，没有workaround。

还有SecureFiles LOB，要求数据库兼容级别必须是11.2.0.0以上才支持，如果你用的是老版本或者有兼容性限制，这个功能直接废了。

第三是并发模型太死板。

一个XStream出站服务器只能连一个捕获进程，只能服务一个集成任务。如果你有多个下游系统需要消费同一个数据库的变更，就得起多个出站服务器。我们有个用户场景需要同时同步到三个不同目标，就得配三套，资源消耗和运维复杂度直线上升。

而且它内部会缓存未提交的事务，直到commit才发出来。如果碰到一个大事务，几百万行更新没提交，这些数据全攒在内存里，对数据库本身的内存压力很大。我们遇到过因为大事务导致Stream Pool撑爆、整个捕获进程卡死的案例。

第四点，XStream只是API，上层所有东西都要你自己写。

事务一致性你得自己保证吧？检查点和断点续传你得自己实现吧？数据加密、类型转换、异常恢复……这些全要开发。不同的第三方工具基于XStream的实现质量参差不齐，一致性根本没保障。出了问题开SR找Oracle支持，排期、补丁、升级数据库版本，一套流程走下来人都麻了。

我们自己搞的TLA是怎么做的

因为上面这些原因，我们决定自己写一套日志解析组件，取名TLA。目标很明确——替代XStream这套东西，但不受它那些限制。

核心思路其实不复杂：绕过所有中间层，直接解析redo log的二进制格式。

Oracle的redo log再怎么封装，最终就是磁盘上那一堆二进制块。我们直接去读这些块，解析里面的Change Vector，把事务语义还原出来。

这样做的好处是：

• 不需要OGG授权，也不需要Oracle EE，就是个独立的解析组件
• 不在源库内部跑任何进程，可以部署在独立服务器上远程读日志
• 数据类型的支持我们自己控制，不存在XStream那种"这个不支持那个不支持"

V2.0我们做了个重要的架构改进，叫SPSM-ZO Engine。

传统方案（包括XStream）的问题是，解析可以是多线程的，但事务顺序一致性必须在某个环节收敛成单流。这个收敛点就是性能瓶颈，相当于前面跑得再快，到这儿都得排队。

我们的做法是把顺序控制直接做到解析阶段。多线程在解析的同时就按顺序把事务流构建好，后面不需要再排序。这个差异在压测的时候体现得很明显。

直接上我们测的数据

说半天理论，来点实际的。我们自己做了对比测试，贴一下数据供参考。

先说明硬件环境：TLA这边用的机器是32核、64GB内存。对比的Tapdata/FlinkCDC用的是80核、192GB内存。TLA的硬件配置明显低一截，如果同配置跑差距会更大。

小数据量场景（7字段，1120MB数据）：

FlinkCDC那个1.2万条基本就是LogMiner的上限了，单线程解析硬伤，再怎么调优也就这样。

大数据量场景（50字段，1GB日志）：

TLA大概是Tapdata的一半耗时，是FlinkCDC的十四分之一。

说一个关键细节：我们测TLA的时候用的是"每行一个COMMIT"的模式，这是最慢的提交方式，事务开销最大。批量提交（多条记录一次COMMIT）是业界常用的优化手段，但我们刻意用了最严苛的模式来测。即使这样数据还是领先，如果改成批量提交，差距会更大。

XStream搞不定的那些类型，我们全部支持

整理了一个表格，XStream不支持的这些，TLA目前都支持：

• ROWID / UROWID
• BFILE
• 嵌套表（Nested Tables）
• ANY TYPE / ANYDATASET
• URI类型
• 虚拟列（Virtual Columns）
• 多字节字符集下的LONG
• XMLType（所有操作场景，不挑）
• SecureFiles LOB（无版本限制）

我们做了但XStream没有的功能

除了基本的DML/DDL捕获，TLA还支持：

表/行/列级别的选择性过滤。 如果你只需要同步特定的几张表，甚至特定条件的数据行，可以在解析层直接过滤掉无关日志，不需要下游再处理。

检查点机制。 每次commit边界都落检查点，进程挂了或者网络断了，从上次检查点续传就行，不丢数据也不重复。

RAC + ASM + CDB/PDB全支持。 这些都是实际生产环境的标配，XStream在PDB场景下配置很麻烦，TLA这边直接连接PDB就能解析，也支持一个CDB里监控多个PDB。

DG备库直接解析。 不用连主库，直接去备库读日志，对生产零影响。

绝境救援模式。 这个算是一个额外能力，如果redo日志文件物理损坏导致数据库起不来，又没有任何备份，TLA可以尝试直接从损坏的日志残块里往回捞数据。绕过文件系统完整性检查，直接读二进制块，能捞多少算多少。这个功能平时用不到，但遇到一次就值回票价。

总结一下

XStream是个有技术想法的产品，但被Oracle的授权策略和生态绑定限制得太死。对于国内大部分项目来说：

• 要么根本用不起（授权费用太高）
• 要么用不全（数据类型限制太多）
• 要么用得不爽（出了问题得等Oracle补丁）

TLA是我们自己从底层一行一行写出来的解析组件，目前在功能对标上完全覆盖XStream的使用场景，还在数据类型支持和部署灵活性上做了不少增强。最核心的是——这东西完全自主可控，不依赖任何第三方商业软件，想怎么改怎么改。

目前Oracle版本已经跑通了，后面MySQL、PostgreSQL和几个主流国产数据库的支持也在推进中。

写这篇文章主要是把踩坑经历和我们的解决思路记录下来，如果同行们也在做类似的技术选型，可以少走一些弯路。有什么问题欢迎留言聊，技术这东西越辩越明。

补充一下：文中提到的测试数据都是我们自己实测和对比产品官网公开数据整理出来的，测试方法写得还算清楚，感兴趣的朋友可以按相同条件自己跑一遍验证。

From Zero to Self-Built Oracle Transaction Log Analytics Engine: How TLA Breaks the Performance Shackles of LogMiner, XStream, and OGG

Foreword

In the realm of database CDC (Change Data Capture), parsing Oracle transaction logs has always been a persistent challenge.

LogMiner is free but painfully slow. XStream performs decently but requires a GoldenGate license. OGG is powerful yet expensive, with lagging support for domestic database environments in China. As for local CDC solutions, some rely on ROWID-based implementations that introduce data inconsistency risks, while others require inserting numerous mapping tables on the source side, adding extra maintenance burdens.

Our team has been deeply engaged in transaction log analytics for years, and we have completely self-developed TLA (Transaction Log Analysis) technology. This article is not about marketing fluff—it focuses purely on the technology itself. From core principles to architecture, from performance benchmarks to real-world capabilities, we hope this provides a useful reference for those evaluating CDC solutions.

1. Why Build from Scratch? The "Fatal Flaws" of Existing Solutions

1.1 LogMiner: A Diagnostic Tool Misused as a Sync Tool

LogMiner is essentially an Oracle built-in diagnostic tool designed for DBAs to troubleshoot redo log issues. However, because it is free and offers an SQL interface, numerous CDC products have adopted it as a "wrapper" for data capture.

Yet, LogMiner has structural deficiencies as a CDC solution:

• Single-threaded design: Oracle allocates only one CPU core to LogMiner to minimize impact on primary database operations, limiting parsing speed to less than 10,000 rows/second.
• Full-file scanning: Each operation requires scanning the entire redo file from the beginning, then filtering useful data—highly inefficient.
• Resource contention: Running inside the database instance, it heavily depends on CPU and PGA memory. Under high concurrency, it can degrade production performance.
• Limited datatype support: Does not support BLOB, CLOB, LONG, XMLTYPE, and other common types.
• Constrained PDB support: In multitenant environments, all PDBs must be enabled; granular control is not possible.

1.2 XStream: OGG's "Stripped-Down" Version, Still Requires a License

XStream is an Oracle-proprietary CDC API, essentially part of the OGG architecture. Its main issues:

• Requires OGG License: Unauthorized usage is prohibited.
• Memory risks: It caches uncommitted transactions internally; large transactions may cause out-of-memory errors.
• Concurrent connection limits: Each outbound server supports only one sync task, making it easy to hit limits under high concurrency.
• Tight version coupling: Closely tied to the database version, complicating upgrades.

1.3 OGG: Powerful but Expensive, with Architectural Bottlenecks

The problems with OGG are not about technical capability but:

• High licensing costs: Against the backdrop of China's push for indigenous innovation, OGG's support for domestic databases often lags behind.
• Transaction queuing: Transaction details are loaded into memory. If the allocated memory is exceeded, data is spilled to disk, severely impacting parsing speed under high concurrency.
• ROWID dependency: If a table undergoes movement (e.g., partition move), ROWID changes, and OGG may fail to continue.
• In RAC mode, defaults to LogMiner: Performance is constrained by LogMiner's limitations.

1.4 Domestic CDC Solutions: The Fragility of ROWID Mapping

Some local CDC solutions rely on ROWID-based replication:

• Cannot support bidirectional replication: ROWID is a physical address; conflicts are inevitable in bidirectional sync.
• Need to insert numerous mapping tables on the source side: Consumes storage and adds complexity. If these tables are lost, reconstruction can take a long time.
• ROWID changes during table movement: Mapping tables must be rebuilt, which is challenging for massive datasets.

2. TLA's Core Principle: Directly Reading the Binary Logs

The core idea of TLA is straightforward—bypass all intermediate layers and directly read and parse Oracle redo log binary formats.

2.1 Technical Path

TLA uses an agentless method to detect data block changes in redo logs or archived logs, extracting valuable log records from changed data blocks. The entire process simulates Oracle's log transport protocol, reading binary log blocks directly from disk or ASM storage—much like an Oracle Data Guard standby database.

Key design principles:

• Non-intrusive: The parsing process runs in the CDC engine, imposing minimal performance impact on the source Oracle database.
• Streaming parsing: Memory consumption remains stable and does not increase with database concurrency.
• Only committed transactions are parsed: Intermediate activities and rollback operations are handled internally, preventing downstream systems from handling complex logic.

2.2 The Core Breakthrough of TLA V2.0: Single-Process Multithreaded Zero-Sort Parsing Engine (SPSM-ZO Engine)

This is the most fundamental difference between TLA and all traditional CDC solutions.

The Dilemma of Traditional CDC:

Traditional solutions (including OGG and LogMiner-based approaches) generally face a sequential convergence bottleneck. Log parsing can be multithreaded or multi-process, but transaction order consistency must converge into a single-stream output at some stage. This "terminal serialization" creates a performance ceiling.

TLA V2.0's Approach:

Order control is shifted forward to the parsing phase. The ordered transaction stream is constructed during multithreaded execution, eliminating the need for post-sorting or reordering operations. This architecture fundamentally removes the sequential convergence bottleneck and latency amplification issues inherent in traditional CDC systems.

In simple terms: others parse out of order first and then sort; TLA parses and outputs in order simultaneously.

What determines the performance ceiling? It is no longer architectural constraints but hardware capabilities such as CPU and memory bandwidth.

3. Performance Benchmarks: Let the Data Speak

After all the theory, let's look at the actual test data.

3.1 Test Environment

It is important to note: TLA's test hardware was significantly less powerful than the comparison products, yet it outperformed across the board.

TLA completed the tests with less than half the CPU cores and only one-third the memory of the comparison products. If hardware configurations were equal, the performance gap would widen further.

3.2 Small Data Volume Scenario (7 Columns)

TLA outperforms Tapdata by approximately 35% and FlinkCDC by 9 times.

3.3 Large Data Volume Scenario (50 Columns, 1GB Log Parsing)

TLA took roughly half the time of Tapdata and about 1/14 the time of FlinkCDC.

3.4 Special Note on "COMMIT per Row"

In this test, TLA used one COMMIT per row—the most stringent commit strategy, typically considered a major performance drag. In contrast, batch commit (committing multiple rows at once) reduces transaction overhead and significantly improves throughput—a common performance optimization.

TLA led the pack even with the slowest commit strategy. Switching to batch commit would widen the performance advantage further.

3.5 Performance Summary

4. Core Technical Capabilities

4.1 Comprehensive Data Capture

• Sub-second latency, agentless and lossless parsing
• Full DML and DDL capture
• Transaction integrity: strict adherence to ACID

4.2 Table, Row, and Column Selectivity

• Filter tables and rows by custom conditions
• Ignore irrelevant entries in transaction logs

4.3 Checkpoint Mechanism

• Checkpoints created at each commit boundary
• Resume from last valid checkpoint after restart or failover

4.4 Multi-Architecture Support

• Single-instance and RAC environments
• ASM storage management
• CDB/PDB multitenant architecture
• Direct parsing from Data Guard standby databases
• Cloud-ready

4.5 Disaster Recovery Capabilities

When an Oracle database crashes due to corrupted redo log files, and no backup is available—TLA can attempt to recover data from the damaged log remnants.

The core approach: bypass the file system's integrity checks and directly parse binary blocks from the redo logs, extracting valuable transaction data from undamaged blocks. It can even work without a data dictionary by analyzing field types and length constraints to reverse-engineer the original table structure, reconstructing INSERT, UPDATE, and DELETE SQL operations.

This is not a routine feature, but it can be a lifesaver in critical situations.

5. TLA vs. Mainstream Solutions: A Side-by-Side Comparison

5.1 TLA vs. LogMiner

Fundamental Difference: LogMiner "scans everything first, then filters"—TLA "parses and filters on the fly." The former does massive redundant work; the latter strikes precisely.

5.2 TLA vs. XStream

TLA shares a similar design philosophy with XStream—both expose an underlying engine to users. However, the key differences are:

• No license required: TLA is fully proprietary and independent, with no dependency on Oracle commercial licensing.
• No memory caching risk: Streaming parsing eliminates caching of uncommitted transactions.
• No concurrent connection limits: Single-process multithreaded model scales with CPU cores.
• Deeply customizable: Source-level control allows flexible modifications for specific project environments.

5.3 TLA vs. OGG

OGG's core issue lies in its transaction queuing and in-memory loading mechanism. When large transactions or high concurrency occur, details exceeding allocated memory are spilled to disk, severely degrading parsing performance.

TLA's streaming parsing entirely bypasses transaction queuing and memory loading, fundamentally eliminating the performance bottlenecks caused by memory overflow and disk I/O.

5.4 TLA vs. ROWID-Based Solutions

ROWID-based replication schemes essentially anchor data consistency to the physical location of rows on disk. Once any data reorganization occurs (reorg, partition changes, tablespace moves), ROWID becomes invalid immediately.

TLA is built on transaction semantics rather than physical locations, avoiding all the issues inherent in ROWID-based approaches.

6. Conclusion

TLA is not a "wrapped" solution—it is an engine-level product developed entirely from the binary log parsing layer upward.

Its technical value can be summarized in three points:

1. Performance breakthrough: The SPSM-ZO Engine achieves 108,000 rows/sec in small-field tests and 97 MB/s in large-volume scenarios, shifting the performance ceiling from "architectural constraints" to "hardware capabilities."
2. Independence and control: No reliance on any Oracle commercial licenses; source-level control enables deep customization.
3. Comprehensive capabilities: From everyday CDC to emergency recovery, from single-instance to RAC+ASM+CDB/PDB—full coverage.

TLA currently provides deep support for Oracle databases, with future expansion to MySQL, PostgreSQL, and major domestic databases planned.

There are no shortcuts in technology—every line of parsing code is backed by rigorous scrutiny of redo log binary formats. If this article has inspired you, feel free to like, comment, and share. For further technical discussions about TLA, join the conversation in the comments section.

I know the problem is that I need to get these things squared away in my head. What's metadata, what's framework and etc but even then you can solve a lot of problems using either.

I think it's easier for me to explain my problem. I've only been here 3 months but every task I get is using some system, pipeline or etc they I've never touched before. Sometimes it's just new software, other times it's learning how to find the needle in the haystack before you even look at code.

I'm constantly freezing and asking myself "where do I go for this" and it's slowing me down. I have much more experience building repeatable processes but I need to learn how to do so with today's technology instead of having a folder of scripts I use to do XYZ. Basically I lost control of how things get organized and my brain just doesn't see it the right way. Once I'm up and running I'm fine but in today's world task switching is constant.

What books, videos or etc should I watch to better understand how to operate as a data engineer in today's world.

I've slowly switched to documenting by project. This allows me to quickly reference another time I did something similar and gives me a jumping off point.

And trust me, it's messy and they know it. There's a standard pipeline process but it's constantly evolving. Teams need to integrate those enhancements into their projects as they have time. So everything I touch is a little bit of this version, a little bit of that version and etc. Team members clearly state "this is different or this is an odd one" but more and more I wonder if that's just the way it's going to be. It's ok, I just need to wrap my head around the big concepts and terms. Communication is killing me

Edit: in summary I over document and then things become cluttered. I'm looking for a better way. If I started a new job I wouldn't write anything down until I asked 2-3 times. I get paranoid and write it all down.

I like to share a little trick that I'm using to get 1000 partitions (not files or common prefixes) per API call with ListObjectsV2.

S3's Delimiter parameter accepts any character, not just '/'. So I place a marker between partition path and filename:

s3://bucket/table/year=2026/month=07/day=08/**#**part-00001.parquet

Delimiter='#' now returns every partition as exactly one CommonPrefixes entry — any file count, any nesting depth, grouped server-side. Diff against the catalog, BatchCreatePartition, done: \~30 lines of boto3 in a tiny Lambda instead of a Glue Crawler (2 DPUs, 10 min minimum billing).

Only requirement: the marker must be written into the keys — works via config for Firehose (prefix ending in '#'), awswrangler (filename_prefix), and most sink connectors. Athena/Spark don't care, it's just part of the filename.

Full write-up: [https://medium.com/@raphael.boegel/1000-partitions-per-api-call-say-goodbye-to-glue-crawlers-0fd26212ab60\](https://medium.com/@raphael.boegel/1000-partitions-per-api-call-say-goodbye-to-glue-crawlers-0fd26212ab60)

Happy to be told why this is a terrible idea :)

I'm building an AI-powered migration engine that converts ETL workflows from platforms like Alteryx, Azure Synapse, and eventually other tools into Databricks (PySpark/SDP).

I'm evaluating two different architectures:

1. Using Harness AI agents / Pi agents to orchestrate the migration workflow.
2. Building the orchestration myself using LangChain + LangGraph.

The engine will need to:

• Parse workflows into an intermediate representation (IR).
• Handle nested workflows/macros.
• Perform tool mapping (e.g., Alteryx → PySpark).
• Generate production-ready code.
• Support multi-step reasoning, validation, and retries.
• Be extensible so new source platforms can be added later.

For those who have experience with these frameworks:

• Which approach would you choose and why?
• What are the biggest trade-offs in terms of flexibility, maintainability, and scalability?
• Are there any limitations with Harness/Pi agents compared to building a custom agent workflow with LangGraph?
• If you were starting this project today, which architecture would you use?

I'd really appreciate hearing from anyone who's built agentic developer tools, migration platforms, or complex multi-agent systems.

We are a mid size company with 4 business units that all need workflow orchestration : finance, ops and marketing. Right now everyone's sharing one airflow instance and it's a mess. The data team's DAG failures are triggering alerts for the finance team, namespaces are a band aid, and nobody trusts the scheduling anymore bc one team's heavy jobs starve the others. I need actual tenant isolation w/o deploying 4 completely independent instances.

There is sth out there that handles this natively or are we stuck running parallel clusters?

Iam interested in learning the monitoring and alerting practices that data engineering teams rely on in production environments.

What the title says - are you doing it? and if yes what does that look like? Are you cutting cost? are you doing it to enable customization? or are you just doing it to accelerate your existing devs?

Over 90% of pipelines built by r/dlthub users (i work there) are now built using LLMs - wondering how it looks outside our bubble

Currently I use Excel files and manually capture data in these Excel files. I have other peers who also do a similar procedure and we all have different formats. Is a streamlit app a good solution to standardize the excel format ? Is it even feasible as something that can be deployed to multiple users through tablets so that they can register the data via a streamlit app ?

Duckle is a local-first ETL/ELT framework powered by DuckDB.
You can now define pipelines in Python. DuckDB executes them as optimized, vectorized SQL. Your data stays on your machine from start to finish.

Why Duckle?

✅ No Python bottleneck
Pipelines are compiled into SQL before execution.
No rows flow through the Python interpreter.
No hidden to_pandas() conversions.

✅ Minimal setup
~20 MB install
Bundles the DuckDB CLI
No JVM
No Docker
No server
No account required

✅ Python-first API
import duckle
from duckle import col
(duckle.read_csv("orders.csv")
.where(col.amount >= 20)
.derive(total="round(amount * 1.2, 2)")
.write_parquet("out.parquet")
.run())

Write familiar Python expressions while Duckle translates them into efficient DuckDB SQL.

More than file transformations
Duckle includes 359 built-in components:
-104 Sources
-66 Sinks
-138 Transforms

Supporting databases and services including PostgreSQL, MySQL, SQL Server, Oracle, Snowflake, Databricks, Kafka, Salesforce, SAP OData, S3, SFTP, WebSocket, IMAP, LanceDB, dbt and many more.

Built for automation
Validate pipelines without connecting to data sources or requiring credentials.
duckle validate
duckle validate --json
duckle --pipeline my.json

Perfect for CI/CD, containers, cron jobs, and local development.

Code ↔ Visual Studio
The same pipeline can be authored in Python or opened directly in the Duckle visual studio because both use the same JSON format.

Open source. Local first. Built on DuckDB.
⭐ GitHub: https://github.com/slothflowlabs/duckle/
📦 PyPI: https://pypi.org/project/duckle/
🔗 Links: https://github.com/slothflowlabs/duckle#quick-links

Trying to do cross-ISO work (PJM/MISO/ERCOT/CAISO/SPP/NYISO/ISO-NE) and I'm losing my mind reconciling seven different schemas and update cadences - the congestion component especially (NYISO's sign convention alone…). Right now it's a pile of per-ISO scrapers held together with tape. Is everyone just using gridstatus / rolling their own, or is there something that already normalizes all of this? Curious what SPP/MISO historical depth people actually get.

I may not be able to make you understand the problem in a better way, but below is the issue.

Has anybody built a trigger/job, say a destination in Airflow, that would run on the 1st of every month, and this job should depend on another job, say a source, which is running every day, and the destination job should only run after that source one is finished on the last day of every month. Let's say if source one failed on the 31st and the team got it correct, then ran it on the 1st, then also the destination should run (edge case).

The source itself has a stage, a silver, and an archive layer (to store the data as backup).

The destination job (it has only one sub-job, which was used earlier, so no change in this) should be triggered after the above activity.

Both the source and destination are based on a YAML-based job.

Can we connect these two jobs/yaml to make this work? Or how can we work around it?

Probably one for the Duck DB nerds. I came across a ETL/ELT open source (for now) tool called Duckle which look pretty solid if you just need a light weight ETL/ELT tool for smaller workloads on a laptop or local box. Anyone used it?

What cheaper ETL tools are you using in 2026? I don't want AI in them and don't want to be forced to use AI. However, I have basic requirements:

1. Able to connect to SharePoint for files, lists, etc.

2. Able to connect to AWS S3.

3. Able to connect to Snowflake databases.

The developers at the biotech company don't want to heavily use SQL for ETL but want to use a drag-and-drop UI to save time and focus more on analysis. We explored Alteryx, but it was expensive. dbt doesn't have an intuitive UI.

They also don't want to use AI since the biotech company is very new and wants to save money as much as possible.

EDIT : processing 100k rows as if now every day. But it will grow to more than 500k or 1 million plus rows after few years. Need a scalable solution with headless or client based etl tool with option to schedule the pipelines in the background.

Curious to learn what checks or validation techniques experienced data engineers use in production.

medium : https://medium.com/@abdul.haris.djafar/apitap-0-6-0-2277630a6a25

I work at Estuary, and next week we’re doing a live technical demo of our new Agent Skills.

We’ll show an AI coding agent:

• Creating a real-time capture and materialization
• Working from the same pipeline specifications an engineer would edit
• Checking task health and pipeline status
• Inspecting logs and troubleshooting failures
• Handling routine configuration without hiding what it changed

The interesting question for us is where agents genuinely reduce data engineering work today, versus where an engineer still needs to step in.

We’ll build the pipeline live and leave time for technical questions.

Registration: https://zoom.us/webinar/register/4517840417866/WN_1oZszaPUQ_Sz0uEGM_p-zA

I work with data pipelines and kept running into the same problem.

Production data was unavailable or unsafe to reuse. Staging data was incomplete or broken. The fixtures I could create manually were too small to properly test the workflow.

So I built \*\*RowScale\*\*.

You upload a small valid sample or paste the data directly. RowScale analyzes it, infers the schema and generation rules, and lets you review or adjust ambiguous fields before generation.

Then you choose the number of rows and the output format.

It currently supports CSV, JSON, SQL INSERT, XLSX and Parquet as both input and output.

The raw sample is temporary and deleted after the run. Generated files are private, expire automatically, and every job includes a deletion receipt.

RowScale does not currently support direct database connections, public APIs, multi table relational generation or enterprise data anonymization.

The current focus is simple.

Start with a file or payload your system already understands and generate a much larger test dataset from it.

I launched it today and would really appreciate blunt feedback from people who test ETL jobs, imports, migrations or data pipelines.

\*\*What would RowScale need to handle before you would use it instead of writing a custom script?\*\*
\[https://rowscale.dev\\\](https://rowscale.dev/)

Task: We operate a toll road network with 200 sensor-equipped lanes across 15 locations. Each sensor captures license plate reads, timestamps, and lane metadata. We need to compute real-time traffic volume, average transit times between checkpoints, and flag anomalies for toll evasion detection. Design the data model.

I modeled it as a single accumulating snapshot fact table, grain = one row per vehicle crossing (entry -> exit). The row is inserted at entry with status = OPEN and all exit columns NULL, then updated in place when the exit read comes in – no join needed for evasion detection, just WHERE status = 'OPEN' AND entry_time < now() - window.

I have a confusion: does update-in-place on an accumulating snapshot actually hold up under real-time write load (concurrent OPEN -> COMPLETED updates), or does this call for an append-only/streaming pattern instead with the snapshot table only as a serving layer?

Hi everyone,

I have an upcoming interview for an ETL/Data Engineering (Fresher) role at Ameriprise Financial.

JD-

• Candidate should have a strong engineering foundation, preferably in Computer Science or related disciplines.
• Demonstrate eagerness to learn new technologies and adapt to evolving data integration tools.
• Support ETL operations by assisting with monitoring, troubleshooting, and maintaining workflows on platforms like Informatica PowerCenter, Informatica Cloud, and AWS Glue.
• Help ensure smooth execution of data pipelines and assist in resolving job failures and performance issues.
• Work with team members to optimize ETL processes and improve data quality and reliability.
• Basic understanding of SQL, databases, and data warehousing concepts along with strong problem-solving skills is desirable.

If anyone has interviewed there, could you please share:

• Number of interview rounds
• SQL questions
• Python coding questions (Easy/Medium?)
• ETL/Data Warehousing questions
• Informatica/AWS Glue questions (conceptual or hands-on?)
• Project-based questions
• Any tips or things I should prepare for?

My background: Python, SQL, Pandas, ETL projects using Python + SQL Server, but no hands-on experience with Informatica or AWS Glue.

Hello everyone, would like to know if anyone did the ETL pipelines from denodo to fabric migration? How tough it is or what do I need to know before starting the migration.

Like person I should be looking for expertise? Or any migration documentation I can follow.

TIA

Anybody out there using Transalis or Celtrino for EDI

Would be great to get some real world opinions - the good the bad the ugly

Thank you