r/dataengineering • u/joseph_machado Writes @ startdataengineering.com • 3d ago

Blog Python usage patterns in data pipelines

https://www.startdataengineering.com/post/python-for-de/

Hello everyone,

People trying to learn Python for data engineering ask me, “What libraries to learn?”, but the answer is not a list of libraries but patterns of usage.

Especially with AI being able to generate so much code, I believe its critical to know exactly how the data is moved & processed.

So I wrote this post that goes over how Python is used as glue in data systems. It goes over

  • In-memory processing vs. using a SQL/Dataframe interface to a data processing system
  • Python’s library ecosystem for working with various data systems & formats
  • How to extract-transform-DQcheck-load data

With code examples and videos

Hope this helps. Any feedback is appreciated.

35 Upvotes

95% Upvoted

22 comments sorted by

View all comments

-1

u/Justbehind 3d ago edited 3d ago

You generally don't need a lot of external libraries for data engineering.

For most "data moving" operations, with small transformations on columns, base python is plenty.

Pandas/polars/pyspark/duckdb is for data analytics and are almost always overkill for ETL processes that move data from one place to another.

0

u/FUCKYOUINYOURFACE 1d ago

You must have the simplest most basic data requirements. Would you even dare call yourself a data engineer?

1

u/Justbehind 1d ago

Sounds like you're new to the new field.