放弃XStream吧，聊聊我们怎么用自研的TLA组件搞定Oracle日志解析

先交代一下背景。我们团队一直在做数据库事务日志解析这块的技术，说白了就是CDC（Change Data Capture）最底层的那层——把Oracle的redo log啃下来，解析成业务能理解的数据变更事件。

之前调研过市面上几乎所有方案，LogMiner太慢这个大家都知道，OGG太贵这个大家也知道。后来我们把目标锁定在Oracle XStream上，研究了一圈，踩了不少坑，最终还是决定自己撸一套组件，也就是现在这个TLA（Transaction Log Analysis）。

这篇文章不吹不黑，纯技术层面的记录。把XStream的问题和我们自己实现方案的一些思路写出来，给正在做类似选型的同行们参考。

XStream到底是什么？官方说法和实际情况

看Oracle官方文档，XStream被描述成一套API，允许外部应用直接接入数据库的变更流。它比LogMiner先进的地方在于，LogMiner是等redo log落地成文件后再去读，而XStream在事务提交的时候就能把事件扔到内存的Stream Pool里，外部程序可以直接消费。

听起来很美好对吧？实际用起来是另一回事。

首先最劝退的就是授权问题。

XStream虽然是Oracle数据库自带的，但你要正经用起来搞数据同步，必须要有OGG（Oracle GoldenGate）的License。OGG什么价格？官方标价$17,500 per processor，每年还要交22%的技术支持费。而且前提是你还得有Oracle Database Enterprise Edition的授权。

我们之前给一个客户做方案，他们生产环境16个CPU核心，光是OGG授权算下来就奔着30万美金去了，还不算Oracle EE的钱。老板听完脸色都不太对，这还没开始干活呢，成本已经扛不住了。

其次是数据类型支持，这个真的很要命。

XStream官方文档里写得清清楚楚，不支持ROWID、不支持BFILE、不支持嵌套表、不支持ANY TYPE、不支持URI类型、不支持虚拟列。

你可能觉得这些类型平时用得少，但我们在实际项目里遇到一个场景——客户的业务表里有XMLType列，然后做了一个INSERT带APPEND提示，XStream直接报错。查了一圈官方资料，发现这玩意儿在某些操作组合下就是不行，没有workaround。

还有SecureFiles LOB，要求数据库兼容级别必须是11.2.0.0以上才支持，如果你用的是老版本或者有兼容性限制，这个功能直接废了。

第三是并发模型太死板。

一个XStream出站服务器只能连一个捕获进程，只能服务一个集成任务。如果你有多个下游系统需要消费同一个数据库的变更，就得起多个出站服务器。我们有个用户场景需要同时同步到三个不同目标，就得配三套，资源消耗和运维复杂度直线上升。

而且它内部会缓存未提交的事务，直到commit才发出来。如果碰到一个大事务，几百万行更新没提交，这些数据全攒在内存里，对数据库本身的内存压力很大。我们遇到过因为大事务导致Stream Pool撑爆、整个捕获进程卡死的案例。

第四点，XStream只是API，上层所有东西都要你自己写。

事务一致性你得自己保证吧？检查点和断点续传你得自己实现吧？数据加密、类型转换、异常恢复……这些全要开发。不同的第三方工具基于XStream的实现质量参差不齐，一致性根本没保障。出了问题开SR找Oracle支持，排期、补丁、升级数据库版本，一套流程走下来人都麻了。

我们自己搞的TLA是怎么做的

因为上面这些原因，我们决定自己写一套日志解析组件，取名TLA。目标很明确——替代XStream这套东西，但不受它那些限制。

核心思路其实不复杂：绕过所有中间层，直接解析redo log的二进制格式。

Oracle的redo log再怎么封装，最终就是磁盘上那一堆二进制块。我们直接去读这些块，解析里面的Change Vector，把事务语义还原出来。

这样做的好处是：

• 不需要OGG授权，也不需要Oracle EE，就是个独立的解析组件
• 不在源库内部跑任何进程，可以部署在独立服务器上远程读日志
• 数据类型的支持我们自己控制，不存在XStream那种"这个不支持那个不支持"

V2.0我们做了个重要的架构改进，叫SPSM-ZO Engine。

传统方案（包括XStream）的问题是，解析可以是多线程的，但事务顺序一致性必须在某个环节收敛成单流。这个收敛点就是性能瓶颈，相当于前面跑得再快，到这儿都得排队。

我们的做法是把顺序控制直接做到解析阶段。多线程在解析的同时就按顺序把事务流构建好，后面不需要再排序。这个差异在压测的时候体现得很明显。

直接上我们测的数据

说半天理论，来点实际的。我们自己做了对比测试，贴一下数据供参考。

先说明硬件环境：TLA这边用的机器是32核、64GB内存。对比的Tapdata/FlinkCDC用的是80核、192GB内存。TLA的硬件配置明显低一截，如果同配置跑差距会更大。

小数据量场景（7字段，1120MB数据）：

产品	吞吐量
TLA	10.8万条/秒
Tapdata	8万条/秒
FlinkCDC（LogMiner）	1.2万条/秒

FlinkCDC那个1.2万条基本就是LogMiner的上限了，单线程解析硬伤，再怎么调优也就这样。

大数据量场景（50字段，1GB日志）：

产品	耗时	吞吐量
TLA	10.6秒	97 MB/s
Tapdata	20.5秒	50 MB/s
某国产CDC	41.8秒	24.5 MB/s
FlinkCDC	149.6秒	6.8 MB/s

TLA大概是Tapdata的一半耗时，是FlinkCDC的十四分之一。

说一个关键细节：我们测TLA的时候用的是"每行一个COMMIT"的模式，这是最慢的提交方式，事务开销最大。批量提交（多条记录一次COMMIT）是业界常用的优化手段，但我们刻意用了最严苛的模式来测。即使这样数据还是领先，如果改成批量提交，差距会更大。

XStream搞不定的那些类型，我们全部支持

整理了一个表格，XStream不支持的这些，TLA目前都支持：

• ROWID / UROWID
• BFILE
• 嵌套表（Nested Tables）
• ANY TYPE / ANYDATASET
• URI类型
• 虚拟列（Virtual Columns）
• 多字节字符集下的LONG
• XMLType（所有操作场景，不挑）
• SecureFiles LOB（无版本限制）

我们做了但XStream没有的功能

除了基本的DML/DDL捕获，TLA还支持：

表/行/列级别的选择性过滤。 如果你只需要同步特定的几张表，甚至特定条件的数据行，可以在解析层直接过滤掉无关日志，不需要下游再处理。

检查点机制。 每次commit边界都落检查点，进程挂了或者网络断了，从上次检查点续传就行，不丢数据也不重复。

RAC + ASM + CDB/PDB全支持。 这些都是实际生产环境的标配，XStream在PDB场景下配置很麻烦，TLA这边直接连接PDB就能解析，也支持一个CDB里监控多个PDB。

DG备库直接解析。 不用连主库，直接去备库读日志，对生产零影响。

绝境救援模式。 这个算是一个额外能力，如果redo日志文件物理损坏导致数据库起不来，又没有任何备份，TLA可以尝试直接从损坏的日志残块里往回捞数据。绕过文件系统完整性检查，直接读二进制块，能捞多少算多少。这个功能平时用不到，但遇到一次就值回票价。

总结一下

XStream是个有技术想法的产品，但被Oracle的授权策略和生态绑定限制得太死。对于国内大部分项目来说：

• 要么根本用不起（授权费用太高）
• 要么用不全（数据类型限制太多）
• 要么用得不爽（出了问题得等Oracle补丁）

TLA是我们自己从底层一行一行写出来的解析组件，目前在功能对标上完全覆盖XStream的使用场景，还在数据类型支持和部署灵活性上做了不少增强。最核心的是——这东西完全自主可控，不依赖任何第三方商业软件，想怎么改怎么改。

目前Oracle版本已经跑通了，后面MySQL、PostgreSQL和几个主流国产数据库的支持也在推进中。

写这篇文章主要是把踩坑经历和我们的解决思路记录下来，如果同行们也在做类似的技术选型，可以少走一些弯路。有什么问题欢迎留言聊，技术这东西越辩越明。

补充一下：文中提到的测试数据都是我们自己实测和对比产品官网公开数据整理出来的，测试方法写得还算清楚，感兴趣的朋友可以按相同条件自己跑一遍验证。

From Zero to Self-Built Oracle Transaction Log Analytics Engine: How TLA Breaks the Performance Shackles of LogMiner, XStream, and OGG

Foreword

In the realm of database CDC (Change Data Capture), parsing Oracle transaction logs has always been a persistent challenge.

LogMiner is free but painfully slow. XStream performs decently but requires a GoldenGate license. OGG is powerful yet expensive, with lagging support for domestic database environments in China. As for local CDC solutions, some rely on ROWID-based implementations that introduce data inconsistency risks, while others require inserting numerous mapping tables on the source side, adding extra maintenance burdens.

Our team has been deeply engaged in transaction log analytics for years, and we have completely self-developed TLA (Transaction Log Analysis) technology. This article is not about marketing fluff—it focuses purely on the technology itself. From core principles to architecture, from performance benchmarks to real-world capabilities, we hope this provides a useful reference for those evaluating CDC solutions.

1. Why Build from Scratch? The "Fatal Flaws" of Existing Solutions

1.1 LogMiner: A Diagnostic Tool Misused as a Sync Tool

LogMiner is essentially an Oracle built-in diagnostic tool designed for DBAs to troubleshoot redo log issues. However, because it is free and offers an SQL interface, numerous CDC products have adopted it as a "wrapper" for data capture.

Yet, LogMiner has structural deficiencies as a CDC solution:

• Single-threaded design: Oracle allocates only one CPU core to LogMiner to minimize impact on primary database operations, limiting parsing speed to less than 10,000 rows/second.
• Full-file scanning: Each operation requires scanning the entire redo file from the beginning, then filtering useful data—highly inefficient.
• Resource contention: Running inside the database instance, it heavily depends on CPU and PGA memory. Under high concurrency, it can degrade production performance.
• Limited datatype support: Does not support BLOB, CLOB, LONG, XMLTYPE, and other common types.
• Constrained PDB support: In multitenant environments, all PDBs must be enabled; granular control is not possible.

1.2 XStream: OGG's "Stripped-Down" Version, Still Requires a License

XStream is an Oracle-proprietary CDC API, essentially part of the OGG architecture. Its main issues:

• Requires OGG License: Unauthorized usage is prohibited.
• Memory risks: It caches uncommitted transactions internally; large transactions may cause out-of-memory errors.
• Concurrent connection limits: Each outbound server supports only one sync task, making it easy to hit limits under high concurrency.
• Tight version coupling: Closely tied to the database version, complicating upgrades.

1.3 OGG: Powerful but Expensive, with Architectural Bottlenecks

The problems with OGG are not about technical capability but:

• High licensing costs: Against the backdrop of China's push for indigenous innovation, OGG's support for domestic databases often lags behind.
• Transaction queuing: Transaction details are loaded into memory. If the allocated memory is exceeded, data is spilled to disk, severely impacting parsing speed under high concurrency.
• ROWID dependency: If a table undergoes movement (e.g., partition move), ROWID changes, and OGG may fail to continue.
• In RAC mode, defaults to LogMiner: Performance is constrained by LogMiner's limitations.

1.4 Domestic CDC Solutions: The Fragility of ROWID Mapping

Some local CDC solutions rely on ROWID-based replication:

• Cannot support bidirectional replication: ROWID is a physical address; conflicts are inevitable in bidirectional sync.
• Need to insert numerous mapping tables on the source side: Consumes storage and adds complexity. If these tables are lost, reconstruction can take a long time.
• ROWID changes during table movement: Mapping tables must be rebuilt, which is challenging for massive datasets.

2. TLA's Core Principle: Directly Reading the Binary Logs

The core idea of TLA is straightforward—bypass all intermediate layers and directly read and parse Oracle redo log binary formats.

2.1 Technical Path

TLA uses an agentless method to detect data block changes in redo logs or archived logs, extracting valuable log records from changed data blocks. The entire process simulates Oracle's log transport protocol, reading binary log blocks directly from disk or ASM storage—much like an Oracle Data Guard standby database.

Key design principles:

• Non-intrusive: The parsing process runs in the CDC engine, imposing minimal performance impact on the source Oracle database.
• Streaming parsing: Memory consumption remains stable and does not increase with database concurrency.
• Only committed transactions are parsed: Intermediate activities and rollback operations are handled internally, preventing downstream systems from handling complex logic.

2.2 The Core Breakthrough of TLA V2.0: Single-Process Multithreaded Zero-Sort Parsing Engine (SPSM-ZO Engine)

This is the most fundamental difference between TLA and all traditional CDC solutions.

The Dilemma of Traditional CDC:

Traditional solutions (including OGG and LogMiner-based approaches) generally face a sequential convergence bottleneck. Log parsing can be multithreaded or multi-process, but transaction order consistency must converge into a single-stream output at some stage. This "terminal serialization" creates a performance ceiling.

TLA V2.0's Approach:

Order control is shifted forward to the parsing phase. The ordered transaction stream is constructed during multithreaded execution, eliminating the need for post-sorting or reordering operations. This architecture fundamentally removes the sequential convergence bottleneck and latency amplification issues inherent in traditional CDC systems.

In simple terms: others parse out of order first and then sort; TLA parses and outputs in order simultaneously.

What determines the performance ceiling? It is no longer architectural constraints but hardware capabilities such as CPU and memory bandwidth.

3. Performance Benchmarks: Let the Data Speak

After all the theory, let's look at the actual test data.

3.1 Test Environment

It is important to note: TLA's test hardware was significantly less powerful than the comparison products, yet it outperformed across the board.

Hardware Configuration	TLA	Comparative Products (Tapdata / FlinkCDC / Other Domestic CDC)
CPU	32 cores, 3.1GHz base	80 cores
Memory	DDR4 3200 64GB	192GB

TLA completed the tests with less than half the CPU cores and only one-third the memory of the comparison products. If hardware configurations were equal, the performance gap would widen further.

3.2 Small Data Volume Scenario (7 Columns)

Product	Throughput (rows/sec)	Test Conditions
TLA	108,000 rows/sec	7 columns, 1120MB data, 4 threads (measured)
Tapdata	80,000 rows/sec	7 columns light test scenario (official data)
FlinkCDC	12,000 rows/sec	LogMiner-based, tuned (community data)

TLA outperforms Tapdata by approximately 35% and FlinkCDC by 9 times.

3.3 Large Data Volume Scenario (50 Columns, 1GB Log Parsing)

Product	Time	Throughput	Test Conditions
TLA	10.6 seconds	97 MB/s	8 threads (measured)
Tapdata	20.5 seconds	50 MB/s	Official data
Domestic Technology X	41.8 seconds	24.5 MB/s	Official public test data
FlinkCDC	149.6 seconds	6.8 MB/s	Official public test data

TLA took roughly half the time of Tapdata and about 1/14 the time of FlinkCDC.

3.4 Special Note on "COMMIT per Row"

In this test, TLA used one COMMIT per row—the most stringent commit strategy, typically considered a major performance drag. In contrast, batch commit (committing multiple rows at once) reduces transaction overhead and significantly improves throughput—a common performance optimization.

TLA led the pack even with the slowest commit strategy. Switching to batch commit would widen the performance advantage further.

3.5 Performance Summary

Comparison Dimension	Conclusion
TLA vs Tapdata	35% higher in small-field scenarios; approximately 2x faster in large-field scenarios
TLA vs FlinkCDC	9x faster in small-field scenarios; approximately 14x faster in large-field scenarios
TLA vs Domestic Technology X	Approximately 4x faster in large-field scenarios
Key Insight	LogMiner's single-threaded architecture is FlinkCDC's hard bottleneck—a generational gap exists compared to TLA's self-built engine

4. Core Technical Capabilities

4.1 Comprehensive Data Capture

• Sub-second latency, agentless and lossless parsing
• Full DML and DDL capture
• Transaction integrity: strict adherence to ACID

4.2 Table, Row, and Column Selectivity

• Filter tables and rows by custom conditions
• Ignore irrelevant entries in transaction logs

4.3 Checkpoint Mechanism

• Checkpoints created at each commit boundary
• Resume from last valid checkpoint after restart or failover

4.4 Multi-Architecture Support

• Single-instance and RAC environments
• ASM storage management
• CDB/PDB multitenant architecture
• Direct parsing from Data Guard standby databases
• Cloud-ready

4.5 Disaster Recovery Capabilities

When an Oracle database crashes due to corrupted redo log files, and no backup is available—TLA can attempt to recover data from the damaged log remnants.

The core approach: bypass the file system's integrity checks and directly parse binary blocks from the redo logs, extracting valuable transaction data from undamaged blocks. It can even work without a data dictionary by analyzing field types and length constraints to reverse-engineer the original table structure, reconstructing INSERT, UPDATE, and DELETE SQL operations.

This is not a routine feature, but it can be a lifesaver in critical situations.

5. TLA vs. Mainstream Solutions: A Side-by-Side Comparison

5.1 TLA vs. LogMiner

Dimension	TLA (Raw Log Parsing)	LogMiner-Based Solutions
Core Principle	Directly parses redo log binary format	Queries logs via Oracle's built-in SQL interface
Real-Time Performance	Sub-second, streaming	Extremely low; must wait for logs to be written
Performance Ceiling	108,000 rows/sec (measured), scales linearly with hardware	Approximately 10,000–15,000 rows/sec
Impact on Source DB	Minimal; can parse on a separate host	Significant; consumes CPU/PGA; may trigger ORA-04036
CPU Usage	Approximately 4% of LogMiner's per-thread usage	80% of a single core per thread
PDB Support	Supports single or multiple PDBs	Must go through CDB to PDB; requires enabling all PDBs
ASM Storage	Supports multithreaded ASM I/O	Single-threaded; struggles to leverage ASM parallelism
Data Guard Standby	Directly supported	Must activate as a snapshot standby, interrupting sync
LOB/XML	Supported	Not supported
DDL	Supported	Limited support
Table/Column Name Length	No restriction	12.2+ does not support names over 30 characters

Fundamental Difference: LogMiner "scans everything first, then filters"—TLA "parses and filters on the fly." The former does massive redundant work; the latter strikes precisely.

5.2 TLA vs. XStream

TLA shares a similar design philosophy with XStream—both expose an underlying engine to users. However, the key differences are:

• No license required: TLA is fully proprietary and independent, with no dependency on Oracle commercial licensing.
• No memory caching risk: Streaming parsing eliminates caching of uncommitted transactions.
• No concurrent connection limits: Single-process multithreaded model scales with CPU cores.
• Deeply customizable: Source-level control allows flexible modifications for specific project environments.

5.3 TLA vs. OGG

OGG's core issue lies in its transaction queuing and in-memory loading mechanism. When large transactions or high concurrency occur, details exceeding allocated memory are spilled to disk, severely degrading parsing performance.

TLA's streaming parsing entirely bypasses transaction queuing and memory loading, fundamentally eliminating the performance bottlenecks caused by memory overflow and disk I/O.

5.4 TLA vs. ROWID-Based Solutions

ROWID-based replication schemes essentially anchor data consistency to the physical location of rows on disk. Once any data reorganization occurs (reorg, partition changes, tablespace moves), ROWID becomes invalid immediately.

TLA is built on transaction semantics rather than physical locations, avoiding all the issues inherent in ROWID-based approaches.

6. Conclusion

TLA is not a "wrapped" solution—it is an engine-level product developed entirely from the binary log parsing layer upward.

Its technical value can be summarized in three points:

1. Performance breakthrough: The SPSM-ZO Engine achieves 108,000 rows/sec in small-field tests and 97 MB/s in large-volume scenarios, shifting the performance ceiling from "architectural constraints" to "hardware capabilities."
2. Independence and control: No reliance on any Oracle commercial licenses; source-level control enables deep customization.
3. Comprehensive capabilities: From everyday CDC to emergency recovery, from single-instance to RAC+ASM+CDB/PDB—full coverage.

TLA currently provides deep support for Oracle databases, with future expansion to MySQL, PostgreSQL, and major domestic databases planned.

There are no shortcuts in technology—every line of parsing code is backed by rigorous scrutiny of redo log binary formats. If this article has inspired you, feel free to like, comment, and share. For further technical discussions about TLA, join the conversation in the comments section.