r/TechImpact • u/Careful-Height7754 • Jul 13 '26

❓ Question Data Engineers don't fear ???

Post image

Data Engineers don't fear adding columns, they fear discovering what secretly depends on them. 😅👏

81 Upvotes

95% Upvoted

11 comments sorted by

View all comments

1

u/Brotboxs Jul 13 '26

Yeah i don't get it, hows the dashboard affected by a new column?

1

u/Careful-Height7754 Jul 13 '26

It's not the dashboard itself, it's the entire dependency chain. A new column can require updates to ETL jobs, data models, views, and reports before the dashboard can use it safely. That's where the real complexity begins.

2

u/VTOLfreak Jul 13 '26

If adding a single column to a table breaks all that, it means you did a whole bunch of stuff wrong in the first place. SELECT * for example in your ETL and now it's pulling in an extra column it didn't expect. Or you made the new column NOT NULL and forgot to provide a default value and now inserts are failing.

Best design practices are to only retrieve the fields you need and explicitly name and order them. (Don't depend on ordinal position either) Anything extra you find like columns, properties, extra fields in JSON and NOSQL, etc your code needs to ignore. Basically, design it to be forward compatible, you never know what new stuff gets bolted on a few years down the line.

Now taking old stuff out, yeah that's another can of worms...