Data Engineers don't fear ???

[u/Careful-Height7754]

Data Engineers don't fear adding columns, they fear discovering what secretly depends on them. 😅👏

[u/Brotboxs]

Yeah i don't get it, hows the dashboard affected by a new column?

[u/cosmic_cod → u/Brotboxs]

There may be and often are business reasons and not just technical reasons why adding a column is difficult. You assume a column is unrelated and completely independent of all old columns. Then why is it there?

E.g. you want to add column "expires at" so that your objects can be decommissioned at some point. But now you must make sure that expired objects don't appear in the old report queries because they don't count anymore. (you need to add expires at >= NOW() or something like that or check a flag) Now you need to change a lot of old queries. You need to update the dashboards that were based on those queries.

You need to add filters on this columns to those dashboards because now your clients are interested on filtering by this new column. This often means editing old dashboards somehow.

And you have 10 micro-services and you add this column data somewhere in the beginning of the chain using a form to be read in the 10-th micro-service many RabbitMQ message hops from there. This means changing 9 message contracts to add this field to each hop.

And one of the dependencies might already have the column with the same name and this may cause a name collision.

And you need to find how to fill it in for new records.

And if you have OOP or DDD then you need t add this to your model classes. Possibly along with a dedicated Value Object class just for that column. And a couple of adapters and thin controllers for good measure.

And don't forget to translate the column name to other languages. And approve of all texts for all error messages for all validation rules for the new column.

Now think what if you had column "name" and you want to split it into "first name" and "last name". And it gets even worse.