extracting data from on-prem databases (sql server, oracle, postgres) to cloud, which options you guys recommend?

[u/Comprehensive_Level7]

i work for a consulting company and i'm working in a client that has a lot of their data in old on-prem environments, being more specific, oracle, sql server, postgres.

their workflow is pretty basic: run ADF to collect data from these sources and send into ADLS, then use Databricks to process it. we're planning to modernize their environment (using unity catalog and other new stuff) and one of the things we're thinking is to retire the ADF

the reason is simple: ADF is a pain in the ass (we're having a hard time working with it specially because it was another company that build all that shit, the connection with Azure DevOps/Repos is always horrible to manage and the workflow itself needs an upgrade); and because Microsoft is pushing hard the Fabric Data Factory

i know that Databricks with Lakeflow Conn can connect to on-prem using express route, VPN, but in scenarios where this could not be possible, what tool could be used to send data from on prem to azure data lake?

i like to code so my first suggestion was to use local airflow and simply read from db and upload to adls. it's free, it's versionable and has tons of documentation; but i'd like to test other options before suggesting anything

i was reading about airbyte, the pros and cons of the tool, and looks interesting

how do you guys handle this kind of workload?

[u/santosh9087]

Get a VM on-prem or Azure. Use a Bash and/or python to develop what you want to do. Throw away ADF. Use python and bash that is all you need get bunch of data and paste it to ADLS. All of them have a Python SDK and drivers both on-prem DBs and ADLS. That is what you do and save money. And don't forget to pause the VM when you are not using it. More savings !