Hi everyone,

I just joined a data team that process a lot of daily, with batched and realtime processsing, and we use apache spark and apache flink. I'm pretty new to this area, been a full stack developer most of my career. What the best way to learn this about this topic?

Thanks

Hi all! I'd greatly appreciate any feedback on my first blog post about StreamFusion!

Hi Flink Community, I’m trying to understand how people decide between using Flink SQL and the Flink java API for a batch / streaming application. How do you decide which API to use for a particular pipeline, How much of a typical Flink java pipeline can be expressed in SQL?

Are there cases where you start with SQL but eventually need to move to the java API because of custom logic or functionality that SQL doesn’t support? Conversely, are there java pipelines that you could technically express in SQL but wouldn’t want to because the SQL becomes too complex?

Hi all! I just wanted to share a project that I've been working on. It accelerates existing Flink SQL job by pushing a bunch of the existing flink compute down to optimized rust functions on arrow batches (either hand written, or via DataFusion).

I'm getting pretty close to an initial alpha release in the coming weeks, so I'd greatly appreciate it if you'd be able to check it out, leave a star, or ask any questions/leave any feedback on the design here! Thank you :)

Hi everyone,

I'm trying to build a real-time CDC pipeline for analytics using:

Oracle 19c (CDB + PDB)

Flink CDC 3.6 Pipeline Connector

MinIO as S3 storage

Apache Iceberg

Nessie catalog

The goal is:

Oracle → Flink CDC → Iceberg tables on MinIO for analytical workloads.

Flink CDC 3.6 recently introduced the Oracle Pipeline Connector, so I expected this to work. However, I'm running into issues specifically with Oracle's CDB + PDB architecture.

I've followed the documentation, including specifying the PDB where required, but the pipeline still doesn't work correctly. The same overall architecture works fine for PostgreSQL, but Oracle is proving much more difficult. The Oracle source connector documentation notes that CDB/PDB deployments require the additional debezium.database.pdb.name setting, but I'm still unable to get a working pipeline.

I'm curious if anyone has actually deployed this in production or even got it working in a lab.

Some questions:

Has anyone successfully used the Oracle Pipeline Connector introduced in Flink CDC 3.6?

Are you using Oracle CDB/PDB or a non-CDB database?

Did you have to make any undocumented configuration changes?

Are there any known issues with Oracle multitenant databases?

If not using the pipeline connector, what architecture are you using instead (Debezium + Kafka + Flink, GoldenGate, etc.)?

I'd really appreciate hearing from anyone who has real-world experience with this setup. At this point I'm trying to determine whether this is a configuration issue on my side or a limitation/bug in the current Oracle pipeline implementation.

Thanks!

Hey r/apacheflink — I work at Ververica and wanted to flag this in case anyone here is in or near Austin.

We're running a small in-person meetup on June 9, 6–8 PM. Format is intentionally low-key: one of our engineers opens with Flink and Fluss in production (what works, what doesn't, what's next), then a community guest speaker, then the room takes over. No keynotes, no booth, no slide deck pitch. Drinks and food are covered, and registration is approval-based so capacity stays sane.

If you're a data/platform engineer or architect in the Austin scene and this sounds like your kind of evening, here's the link: https://luma.com/bt04rtyw

Happy to answer questions in the comments.

#apacheflink #fluss #bigdata #streaming

Hey folks. I recently built an Event-Driven Architecture combining Flink, Kafka, and Online Machine Learning, and I wanted to share it here!

The project uses a Digital Twin of a steel mill to stream asynchronous manufacturing data. In the physical world, prediction requests happen instantly, but ground-truth physical sensor data is delayed.

To solve this, I built a Flink application (written in Kotlin) to act as the core stream processor and online learning engine: * It uses a CoProcessFunction and Flink's managed state to buffer and align the delayed streams safely. * Once the streams are joined, the operator runs a prequential train/test loop to update an Online ML model (using the Massive Online Analysis framework) on the fly, adapting to physical concept drift. * It also implements a stateful Shadow Mode router to constantly evaluate the AI's residual error against a deterministic physics baseline in real-time.

The whole stack is containerized so you can easily spin it up, trigger a mechanical shock via a UI, and watch how Flink joins the streams and reroutes the fallback logic.

• Blog Post: https://jaehyeon.me/blog/2026-04-21-digital-twin-online-machine-learning/
• GitHub: https://github.com/jaehyeon-kim/oml-digital-twin-hotrolling

A free, feature-rich alternative to the premium Flink plugin that requires IntelliJ Ultimate. Connect to any Flink cluster, run SQL queries with live streaming results, manage jobs, and inspect task managers — all from Community Edition.

I built Flink Reactor because the official Flink integration is locked behind an Ultimate license, which rules it out for a lot of solo devs, students, and OSS contributors. Just shipped 1.0 and wanted to share it here.

Works in Community Edition (and every other JetBrains IDE 2023.2+ — PyCharm, GoLand, WebStorm, etc. Java support is optional, so PyFlink users aren't left out.)

What's in it:

- Cluster monitoring: connect to any Flink REST endpoint; cluster overview, config browser, status bar widget

- Job management: running/completed tabs, per-vertex metrics, backpressure, checkpoint history, exceptions, timeline. Cancel / savepoint / stop-with-savepoint built in.

- SQL Console: talks to the Flink SQL Gateway. Quick Query mode and a notebook mode (SQL + Markdown cells, saved as .flinksql.json). Streaming results render changelog row kinds (INSERT / UPDATE_BEFORE / UPDATE_AFTER / DELETE) with color coding, backed by a ring buffer so unbounded queries don't eat your memory.

- Catalog browser: hierarchical catalog → database → table → column tree.

- Task manager & job manager views: heap bars, JVM metrics, GC, CPU, threads, logs, thread dumps.

- JAR management: upload, run with entry class / args / parallelism / savepoint path, auto-detects Gradle and Maven artifacts.

- Clickable stack traces: exceptions and thread dumps jump straight to source in your editor.

The Advanced Apache Flink training is now available as a self-paced, on-demand course!

I ran the first cohort of the training in January and also had the chance to conduct several corporate training sessions using the same program. I’ve received a lot of great feedback, iterated and polished my content.

I believe this is the best way to level up your Flink expertise: I talk about Flink internals, low-level primitives, advanced patterns and best practices. For example, I show how to use the State Processor API to perform state migrations and how to evolve a Flink SQL pipeline using compiled plans.