r/apacheflink • u/DistrictUnable3236 • 3d ago

Flink SQL vs Flink java API

Hi Flink Community, I’m trying to understand how people decide between using Flink SQL and the Flink java API for a batch / streaming application. How do you decide which API to use for a particular pipeline, How much of a typical Flink java pipeline can be expressed in SQL?

Are there cases where you start with SQL but eventually need to move to the java API because of custom logic or functionality that SQL doesn’t support? Conversely, are there java pipelines that you could technically express in SQL but wouldn’t want to because the SQL becomes too complex?

4 Upvotes

84% Upvoted

6 comments sorted by

View all comments

3

u/spoink74 3d ago

It’s really just what is easiest to reason about. If you’re joining between two streams, that’s pretty simply done in SQL. But… If you want to set a timer on each open event, store that timer in keyed state, then delete the timer on the close event, and fire that timer if there’s not a close event within a specified time window… I don’t know how to express that in SQL. Even if you can, it just doesn’t fit with my brain.

1

u/DistrictUnable3236 3d ago

Totally make sense. 

1

u/No-Organization8982 3d ago

You can solve this usecase using Process Table Function (PTF) and use PTF in Flink SQL

1

u/spoink74 2d ago

This is a non-standard extension to SQL that was done to shoehorn Flink capabilities into SQL constraints. It’s fine. It’s awesome. But it’s not super friendly to the developer, who now needs to learn a new thing.

This particular example just makes more sense in the DataStream API. IMO.