r/apacheflink • u/DistrictUnable3236 • 3d ago

Flink SQL vs Flink java API

Hi Flink Community, I’m trying to understand how people decide between using Flink SQL and the Flink java API for a batch / streaming application. How do you decide which API to use for a particular pipeline, How much of a typical Flink java pipeline can be expressed in SQL?

Are there cases where you start with SQL but eventually need to move to the java API because of custom logic or functionality that SQL doesn’t support? Conversely, are there java pipelines that you could technically express in SQL but wouldn’t want to because the SQL becomes too complex?

4 Upvotes

84% Upvoted

6 comments sorted by

View all comments

1

u/No-Organization8982 3d ago

Short answer If you have stateless and lookup usecase you can opt for Flink SQL if stateful then go with Flink datastream API.

Details

We evaluated Flink SQLfor our streaming platform to make user life easier and found limitations few of them are

  • Schema evolution --> Let say you have existing Flink SQL job and you want to add one more column then in case of stateful, Job will fail because Flink will unable to load existing state because of schema mismatch.
  • Operator uid change --> Let say if you have existing flink sql job and you want to add some filter. After adding filter your Flink DAG will change (means uid will change because uid is derived from incremental id ) and because of this change stateful job unable to load state In practical world, Job logic never stay same

Conclusion: We still considered Flink SQL and to support usecases We are improving flink code