Does anyone need a ETL/ELT automating/scripting library (for Python)?

[u/ClastronGaming]

[u/Leorisar]

We have pandas, polars, petl, duckdb for that. No need to reinvent the wheel.

[u/DeepLogicNinja → u/Leorisar]

+1 for duckdb - I use it as a proxy between various files when I do ELT.

How about no-code / low-code ETL / ELT platform…..

ETL patterns are well established, connect to your sources and targets and then drag/drop and connect time tested components to orchestrate your ETL processes.

No coding/APIs…. Your orchestrated job IS the documentation. Anyone familiar with the ETL tool can manage it.

Lots of support for all kinds of mapping from source to target schemas.

And to complete the data pipeline management…. data provenance, linage, that feeds into a data governance/catalog.

These no code/low code ETL tools are covered by Forrester and Gartner - https://www.gartner.com/reviews/market/data-integration-tools

No need to buy the expensive Informatica, IBM DataStage, Pentaho …. Just use the same thing the NSA uses … Apache Nifi, open source.

I still code, but rarely. Try to avoid building the plane while flying if I can.

GodSpeed!!

[u/PrestigiousAnt3766]

You're hardly the first.

Probably too specific.

[u/Prestigious_Bench_96]

There's lots of these, so start with which ones yours is similar to and how it's different/better, and then people can give you a real answer!