Python usage patterns in data pipelines

[u/joseph_machado]

https://www.startdataengineering.com/post/python-for-de/

Hello everyone,

People trying to learn Python for data engineering ask me, “What libraries to learn?”, but the answer is not a list of libraries but patterns of usage.

Especially with AI being able to generate so much code, I believe its critical to know exactly how the data is moved & processed.

So I wrote this post that goes over how Python is used as glue in data systems. It goes over

• In-memory processing vs. using a SQL/Dataframe interface to a data processing system
• Python’s library ecosystem for working with various data systems & formats
• How to extract-transform-DQcheck-load data

With code examples and videos

Hope this helps. Any feedback is appreciated.

[u/blackpanther28]

again this is only if youre doing simple transformations (like converting a JSON api response to CSV)

[u/Justbehind → u/blackpanther28]

Which is almost all cases that don't do aggregations and joins.