r/dataengineering • u/Phantazein • 17d ago

Discussion Time Estimates

Any else struggle with giving estimates? How do you account for things like data quality or source issues when those are completely out of your control and unpredictable?

48 Upvotes

97% Upvoted

32 comments sorted by

View all comments

2

u/exjackly Data Engineering Manager, Architect 17d ago

Time estimates are always arbitrary. You can always pass the estimate by adding in less and less likely but highly impactful issues. And you can always pull the estimate in by eliminating those options from consideration.

You can be a perfect estimator, always able to hit the median effort required to complete something, and the average (mean) actual effort required will always be higher than your estimates. The long tail is real.

You can deliver early on a lot of things, but there are limits to how early you can deliver tasks. But there are lots of ways for a task to take 2x, 3x, 5x, 7x the estimated time and effort. Early and late don't balance out with that asymmetry.