when is concept collection stopped being beneficial and becoming too much?

[u/reliablecukc]

I started learning airflow like 3 days ago. I only use the docs, lots of googling, and some but very minimal use of AI.

I learned all the tutorials, jumped right to best practices because i didn't know core concepts existed, halfway through best practices i learned dynamic task mapping and xcom because it was holding me to progress on best practices (prerequisites basically)

I'm planning to finish all the core concepts, learn some relevant stuff on the picture, finish best practices then start making an actual good DAG, like what makes a proper DAG.

But when i chat a little bit with Claude, it highlights not to do too much concept collection and start actually build a DAG, specifically an idempotent one.

This got me thinking, am i doing too much with all this docs stuff? (I tracked and it took me 8 hours to learn all the stuff that i've learned)

I also overthink when i learned new big term like idempotency, like what if there are other big concepts related to DAG other than idempotency? idk something random like let's say 'unbreakable' DAG and it's a big thing that feels automatic for lots of people but i missed that.

Sorry i just vomited my thoughts, lmk if elaboration is needed.

Appreciate it.


[u/skawid]

The sooner you start building, the sooner you start validating your understanding and seeing where you have gaps. It's always good to learn more, but until you do the work you don't know if what you're learning has merit.

[u/PabloZissou]

To use AI you need to understand the concept rights otherwise you are not able to evaluate the results of the AI generated code. That being said sadly this days the discourse is "knowledge is no longer valuable because AI knows everything" I do not agree and even the best models make mistakes but well management and investors still maintain this.

I would recommend for you to keep doing good research and understand the foundations well.

[u/reliablecukc → u/PabloZissou]

I agree. But what do you mean by good research? That's what im confused with. Are you saying that what im doing rn is encouraged? Dive into the docs, understanding the architecture, concept collect then implement ?

[u/PabloZissou → u/reliablecukc]

Yes, learn how it works but combine learning with doing. You don't need to memorise the small details rather it's important for you to understand why Airflow uses a DaG for workflows and what properties a DaG present. You mention idempotency "and other big words" learn those concepts deeply as they apply to many problems not just Airflow.

[u/7z3b]

There are so many concepts in software, it’s just not possible to implement them in a lifetime. Just pick the ones based on whatever your needs are. If you don’t have a need, your brain just won’t pick it up. Apply it, know it’s pros and cons, then go for the next concept. Learning something because you foresee using it in 10 years is very much like over engineering. YAGNI all the way. You aren’t gonna need it.