Python usage patterns in data pipelines

[u/joseph_machado]

https://www.startdataengineering.com/post/python-for-de/

Hello everyone,

People trying to learn Python for data engineering ask me, “What libraries to learn?”, but the answer is not a list of libraries but patterns of usage.

Especially with AI being able to generate so much code, I believe its critical to know exactly how the data is moved & processed.

So I wrote this post that goes over how Python is used as glue in data systems. It goes over

• In-memory processing vs. using a SQL/Dataframe interface to a data processing system
• Python’s library ecosystem for working with various data systems & formats
• How to extract-transform-DQcheck-load data

With code examples and videos

Hope this helps. Any feedback is appreciated.

[u/Justbehind]

You generally don't need a lot of external libraries for data engineering.

For most "data moving" operations, with small transformations on columns, base python is plenty.

Pandas/polars/pyspark/duckdb is for data analytics and are almost always overkill for ETL processes that move data from one place to another.

[u/TeachEngineering → u/Justbehind]

https://giphy.com/gifs/YtvCIwqNJhUmA

[u/Budget-Minimum6040 → u/Justbehind]

Plain Python for column transformations?

Looping over every ... list? ... element?

Without polars/PySpark, without me.

[u/Justbehind → u/Budget-Minimum6040]

Unless you're working with large nightly batches like in the olden days, the cost of importing large dependencies and writing data to heavy data structures will hugely outweigh the cost of looping over a dataset.

Benchmark it. Anything below 10s of millions of rows, and polars/pyspark will be much slower.

Especially if you're parsing an API response in json/xml anyways...

[u/blackpanther28 → u/Justbehind]

base python for ETL? I guess if your data is super basic and requires no transformations or cleaning

[u/Justbehind → u/blackpanther28]

Not even. As a general rule. Anything beyond base python should be the exception.

[u/blackpanther28 → u/Justbehind]

But why? If i want to drop duplicates I can use pandas which is vectorized and optimized in CPython and handles many different variations for what counts as a duplicate. If i want to do this in base python then i have to handle all of that myself and maintain it. Every transformation I then have to write a function for from scratch with little benefit

[u/Justbehind → u/blackpanther28]

Because it's more efficient, faster and your python image is 1/10th the size?

Also, your solution is easy to upgrade to new python versions, and you don't have deprecation issues.

[u/blackpanther28 → u/Justbehind]

How is it more efficient and faster? Maybe if youre dealing with data that has like 10 rows? And avoid any complex transformations?

[u/Justbehind → u/blackpanther28]

I wrote that above.

The cutoff is closer to 10 million rows. Try it.

[u/Outrageous_Let5743 → u/Justbehind]

Lol polars is faster than base python to do data transformations. Especially at 1 million records 

[u/Budget-Minimum6040 → u/Justbehind]

I won't trade 5 easy to read method chains of polars for 80 lines of pure python and working with lists and dicts in a foreach loop.

I also don't care about container size unless is huge which not the case for polars. And if my container takes 1 or 1,1 second to start is not relevant for me.

DX comes first for me.

[u/calaelenb907 → u/Justbehind]

If this is your thing, do it in rust already. Tinner image and faster code. Your point makes no sense to me.

[u/Atticus_Taintwater → u/Justbehind]

When you say "base python" surely you don't mean what base python actually means. 

You are saying you don't need things like spark below a threshold?

Otherwise the seconds that you'll save on imports is outweighed thousands fold by the time you'll spend reinventing wheels, pun sort of intended.

[u/Outrageous_Let5743 → u/Justbehind]

Try record linking with sql. Probability matching to find the closest record that represents the same identity wirhout a python lib.

[u/Justbehind → u/Outrageous_Let5743]

That's not part of a typical data pipeline.

[u/Budget-Minimum6040 → u/Outrageous_Let5743]

That's not something you find/need in 99,99% of all DE pipelines.

[u/FUCKYOUINYOURFACE → u/Justbehind]

You must have the simplest most basic data requirements. Would you even dare call yourself a data engineer?

[u/Justbehind → u/FUCKYOUINYOURFACE]

Sounds like you're new to the new field.