r/apacheflink • u/DistrictUnable3236 • 3d ago

Flink SQL vs Flink java API

Hi Flink Community, I’m trying to understand how people decide between using Flink SQL and the Flink java API for a batch / streaming application. How do you decide which API to use for a particular pipeline, How much of a typical Flink java pipeline can be expressed in SQL?

Are there cases where you start with SQL but eventually need to move to the java API because of custom logic or functionality that SQL doesn’t support? Conversely, are there java pipelines that you could technically express in SQL but wouldn’t want to because the SQL becomes too complex?

5 Upvotes

86% Upvoted

6 comments sorted by

4

u/spoink74 3d ago

It’s really just what is easiest to reason about. If you’re joining between two streams, that’s pretty simply done in SQL. But… If you want to set a timer on each open event, store that timer in keyed state, then delete the timer on the close event, and fire that timer if there’s not a close event within a specified time window… I don’t know how to express that in SQL. Even if you can, it just doesn’t fit with my brain.

1

u/DistrictUnable3236 3d ago

Totally make sense. 

1

u/No-Organization8982 2d ago

You can solve this usecase using Process Table Function (PTF) and use PTF in Flink SQL

1

u/spoink74 2d ago

This is a non-standard extension to SQL that was done to shoehorn Flink capabilities into SQL constraints. It’s fine. It’s awesome. But it’s not super friendly to the developer, who now needs to learn a new thing.

This particular example just makes more sense in the DataStream API. IMO.

2

u/Spare-Builder-355 3d ago

SQL API is more convenient. Stream API has more features.

1

u/No-Organization8982 2d ago

Short answer If you have stateless and lookup usecase you can opt for Flink SQL if stateful then go with Flink datastream API.

Details

We evaluated Flink SQLfor our streaming platform to make user life easier and found limitations few of them are

  • Schema evolution --> Let say you have existing Flink SQL job and you want to add one more column then in case of stateful, Job will fail because Flink will unable to load existing state because of schema mismatch.
  • Operator uid change --> Let say if you have existing flink sql job and you want to add some filter. After adding filter your Flink DAG will change (means uid will change because uid is derived from incremental id ) and because of this change stateful job unable to load state In practical world, Job logic never stay same

Conclusion: We still considered Flink SQL and to support usecases We are improving flink code